Channels
announcements
ask-the-community
auth
conference-talks
contribute
databricks-integration
datahub-flyte
deployment
ecosystem-unionml
engineeringlabs
events
feature-discussions
flyte-bazel
flyte-build
flyte-console
flyte-deployment
flyte-documentation
flyte-github
flyte-on-gcp
flyte-ui-ux
flytekit
flytekit-java
flytelab
great-content
hacktoberfest-2022
helsing-flyte
in-flyte-conversations
integrations
introductions
jobs
konan-integration
linkedin-flyte
random
ray-integration
ray-on-flyte
release
scipy-2022-sprint
show-and-tell
sig-large-models
torch-elastic
workflow-building-ui-proj
writing-w-sfloris
Title
b
Ben Rosand
03/21/2023, 11:45 AMHey everyone! Has anyone used flyte for model serving? I'm new to flyte but trying to set up some workflows with ray serve. Has anyone experimented with this before?
s
Samhita Alla
03/21/2023, 1:11 PMI recently wrote a blog post on serving models with Banana: https://flyte.org/blog/how-to-serve-ml-models-with-banana. Haven't tried Ray serve, though.
Join our community sync today where I'd be demoing the Banana integration, in case you're interested!
k
Ketan (kumare3)
03/21/2023, 2:57 PMNo one has tried ray serve with Flyte
Please let us know how it goes
Currently we do not have a type for infinite running tasks, so it may be hard
But this is an idea we have on the roadmap
You could proxy that by running with lots of retires and no timeout
b
Ben Rosand
03/28/2023, 5:39 PMSeems like the banana workflow is more for separating serving from flyte, and just serving models trained with Flyte. I guess more broadly people aren’t using flyte with any kind of serving tools? But I’ll take a look at the proxy idea
If infinite running tasks aren’t specced in I guess serving would be specced in either
k
Ketan (kumare3)
03/28/2023, 5:51 PMYa