Hey! Has anyone gotten apache-beam to run on Flyte...
# announcementsa
abundant-hamburger-66584
09/06/2022, 7:02 PMHey! Has anyone gotten apache-beam to run on Flyte natively? I was going to explore this a bit, but I wanted to see if anyone here has thought through this.
abundant-hamburger-66584
09/06/2022, 7:02 PMIt could be possible to use the spark components to run apache-beam instead of writing a custom backend.
f
freezing-airport-6809
09/07/2022, 12:38 AM
ya
freezing-airport-6809
09/07/2022, 12:39 AM
but, we have not explored. Would love to understand the usecase
a
abundant-hamburger-66584
09/07/2022, 1:45 PMApache beam is really good for model inference if you are using python ml packages. The sdk is in python so like no annoying spark Java errors or needing to use special python UDFs or building spark transformers.
I want to use it to batch score models in parallel using this https://beam.apache.org/documentation/sdks/python-machine-learning/.
Also using GCPs dataflow is cool because auto scaling features are incredible. Vertical and horizontal scaling. Plus you can use GPUs.
abundant-hamburger-66584
09/07/2022, 1:45 PMI will see if I can run it using the spark backend…would be so cool if it just kind of worked
f
freezing-airport-6809
09/07/2022, 2:41 PM
With spark backend it should
Just work.
Lmk.
But I want to understand wdym by flyte backend for beam
a
abundant-hamburger-66584
09/07/2022, 2:45 PMI just meant like a way to spin up and run Apache beam on Flyte.
abundant-hamburger-66584
09/07/2022, 2:45 PMSimilar to how spark works
abundant-hamburger-66584
09/07/2022, 2:46 PMBut beam. If spark doesn't work then I know people use Flink, which might support more features around streaming, but I don't really need them…maybe one day!
f
freezing-airport-6809
09/07/2022, 5:11 PM
Actually flink is already supported. Cc @melodic-king-871 / @swift-animal-75798 how can we upstream https://github.com/spotify/flyte-flink-plugin
176 Views