Has anyone tried running Dask together with Flyte?

@Stefan Avesand great timing, infact folks from Pachama do this and they are kind enough to upstream the plugin https://github.com/flyteorg/flyteplugins/pull/275

Amazing!

@Stefan Avesand Hi from over here at Pachama 🙂 Happy to share the approach we're using at the moment (with a

flytekit

"pure python" plugin), until the work on the backend plugin is finished. Also feel free to send me a DM if you have any further questions

That would be great!

I'll try to get a PR out sometime this week for the flytekit plugin, @Ketan (kumare3) I might do this in a separate repository, as the backend plugin will soon also have a

flytekit

PR?

@Bernhard Stadlbauer also very interested in what a pure python plug-in might look like!

@Rahul Mehta you should be able to do everything in pure python right - with the right service account?

Ahh ofc @Ketan (kumare3), makes sense

@Stefan Avesand @Rahul Mehta @Ketan (kumare3) Sorry that this took a little longer, but I've packaged up the initial

flytekit

dask

code now into this repository. I've also released it to PyPi, so you should be able to install it with

pip install bstadlbauer.flytekitplugins-dask

Let me know if there are any issues, feature requests, etc. 🙂

That's great! Can't wait to try it out.