Channels
#ask-the-community
Title
# ask-the-community
b
Ben Rosand
03/21/2023, 11:45 AMHey everyone! Has anyone used flyte for model serving? I'm new to flyte but trying to set up some workflows with ray serve. Has anyone experimented with this before?
s
Samhita Alla
03/21/2023, 1:11 PM
I recently wrote a blog post on serving models with Banana: https://flyte.org/blog/how-to-serve-ml-models-with-banana. Haven't tried Ray serve, though.
Join our community sync today where I'd be demoing the Banana integration, in case you're interested!
k
Ketan (kumare3)
03/21/2023, 2:57 PM
No one has tried ray serve with Flyte
Please let us know how it goes
Currently we do not have a type for infinite running tasks, so it may be hard
But this is an idea we have on the roadmap
You could proxy that by running with lots of retires and no timeout
b
Ben Rosand
03/28/2023, 5:39 PMSeems like the banana workflow is more for separating serving from flyte, and just serving models trained with Flyte. I guess more broadly people aren’t using flyte with any kind of serving tools? But I’ll take a look at the proxy idea
If infinite running tasks aren’t specced in I guess serving would be specced in either
k
Ketan (kumare3)
03/28/2023, 5:51 PM
Ya