#427 [Backend][Plugin]Support for Dask clustered tasks in Flyte
Issue created by
kumare3
Why would this plugin be helpful to the Flyte community
Users could write very short running distributed array jobs using DASK. This makes it possible to have very small runtime jobs multi-plexed onto same set of nodes.
Type of Plugin
☐ Web Service (e.g. AWS Sagemaker, GCP DataFlow, Qubole etc...)
☑︎ Kubernetes Operator (e.g. TfOperator, SparkOperator, FlinkK8sOperator, etc...)
☐ Customized Plugin using native kubernetes constructs
☐ Other
Can you help us with the implementation?
☐ Yes
☐ No
Additional context
This would really help express some ideas that are not Spark, or heavyweight like Flyte batch jobs.
flyteorg/flyte