https://flyte.org
Join Slack
Has anyone tried running Dask together with Flyte?
# flyte-support
l
Has anyone tried running Dask together with Flyte?
f
@limited-train-92032 great timing, infact folks from Pachama do this and they are kind enough to upstream the plugin https://github.com/flyteorg/flyteplugins/pull/275
🙏 1
l
Amazing!
a
@limited-train-92032 Hi from over here at Pachama 🙂 Happy to share the approach we're using at the moment (with a 
flytekit
"pure python" plugin), until the work on the backend plugin is finished. Also feel free to send me a DM if you have any further questions
❤️ 1
l
That would be great!
a
I'll try to get a PR out sometime this week for the flytekit plugin, @freezing-airport-6809 I might do this in a separate repository, as the backend plugin will soon also have a 
flytekit
PR?
👍 1
e
@agreeable-kitchen-44189 also very interested in what a pure python plug-in might look like!
f
@elegant-australia-91422 you should be able to do everything in pure python right - with the right service account?
👍 1
e
Ahh ofc @freezing-airport-6809, makes sense
a
@limited-train-92032 @elegant-australia-91422 @freezing-airport-6809 Sorry that this took a little longer, but I've packaged up the initial 
flytekit
dask
code now into this repository. I've also released it to PyPi, so you should be able to install it with
Copy code
pip install bstadlbauer.flytekitplugins-dask
Let me know if there are any issues, feature requests, etc. 🙂
❤️ 2
l
That's great! Can't wait to try it out.
168 Views
Open in Slack
PreviousNext
Powered by

Flyte

Flyte enables production-grade orchestration for machine learning workflows and data processing created to accelerate local workflows to production.

Powered by