Hey Has anyone gotten apache beam to run on Flyte natively I

Question

Hey Has anyone gotten apache beam to run on Flyte natively I was going to explore this a bit but I wanted to see if anyone here has thought through this

Evan Sadler · Answer

It could be possible to use the spark components to run apache beam instead of writing a custom backend

Ketan (kumare3) · Answer

ya

Ketan (kumare3) · Answer

but we have not explored Would love to understand the usecase

Evan Sadler · Answer

Apache beam is really good for model inference if you are using python ml packages The sdk is in python so like no annoying spark Java errors or needing to use special python UDFs or building spark transformers I want to use it to batch score models in parallel using this <https beam apache org documentation sdks python machine learning |https beam apache org documentation sdks python machine learning > Also using GCPs dataflow is cool because auto scaling features are incredible Vertical and horizontal scaling Plus you can use GPUs

Evan Sadler · Answer

I will see if I can run it using the spark backend would be so cool if it just kind of worked

Ketan (kumare3) · Answer

With spark backend it should Just work Lmk But I want to understand wdym by flyte backend for beam

Evan Sadler · Answer

I just meant like a way to spin up and run Apache beam on Flyte

Evan Sadler · Answer

Similar to how spark works

Evan Sadler · Answer

But beam If spark doesn t work then I know people use Flink which might support more features around streaming but I don t really need them maybe one day

Ketan (kumare3) · Answer

Actually flink is already supported Cc < Filipe Regadas> < Babis Kiosidis> how can we upstream