Has anyone tried running Dask together with Flyte

Question

Ketan (kumare3) · Answer

< Stefan Avesand> great timing infact folks from Pachama do this and they are kind enough to upstream the plugin

Stefan Avesand · Answer

Amazing

Bernhard Stadlbauer · Answer

< Stefan Avesand> Hi from over here at Pachama slightly smiling face Happy to share the approach we re using at the moment with a `flytekit` pure python plugin until the work on the backend plugin is finished Also feel free to send me a DM if you have any further questions

Stefan Avesand · Answer

That would be great

Bernhard Stadlbauer · Answer

I ll try to get a PR out sometime this week for the flytekit plugin < Ketan kumare3 > I might do this in a separate repository as the backend plugin will soon also have a `flytekit` PR

Rahul Mehta · Answer

< Bernhard Stadlbauer> also very interested in what a pure python plug in might look like

Ketan (kumare3) · Answer

< Rahul Mehta> you should be able to do everything in pure python right with the right service account

Rahul Mehta · Answer

Ahh ofc < Ketan kumare3 > makes sense

Bernhard Stadlbauer · Answer

< Stefan Avesand> < Rahul Mehta> < Ketan kumare3 > Sorry that this took a little longer but I ve packaged up the initial `flytekit` `dask` code now into I ve also released it to PyPi so you should be able to install it with ```pip install bstadlbauer flytekitplugins dask``` Let me know if there are any issues feature requests etc slightly smiling face

Stefan Avesand · Answer

That s great Can t wait to try it out