Flyte enables production-grade orchestration for machine learning workflows and data processing created to accelerate local workflows to production.

Flyte

Hey folks.. quick question: If flyte is deployed on a k8s cluster, can it submit spark jobs on a different k8s cluster?

hey <@U03HQE6THNV> do you mean submit jobs to an existing spark deployment or bring up the <https://docs.flyte.org/projects/cookbook/en/latest/auto/integrations/kubernetes/k8s_spark/pyspark_pi.html#writing-a-pyspark-task|virtual cluster> on a separate k8s cluster?

hey <@U03HQE6THNV> i don't believe this is possible since a single propeller instance runs the entirety of the tasks within a flyte workflow on the same cluster. there are options for <https://docs.flyte.org/en/latest/deployment/cluster_config/performance.html#automatic-scale-out|scaling out> flytepropeller or customizing execution cluster placement within a <https://docs.flyte.org/en/latest/deployment/cluster_config/performance.html#multi-cluster-mode|multi-cluster> set-up

thanks.. I just stumbled upon the multi cluster documentation. I'll look into it

great, please let us know if you have any questions! fyi here are more docs on <https://docs.flyte.org/en/latest/deployment/multicluster.html|multi-cluster> (not sure if those are the ones you've already found :smile:)

<@UPBBNMXD1> i think this is actually possible

cc <@UNW4VP36V> do you remember if we made it to the plugin manager - cc <@U03HQE6THNV>

ideally <@U03HQE6THNV> better to co-locate on the same cluster

<@U03HQE6THNV> so you can create a custom kubeclient for some plugins - <https://github.com/flyteorg/flytepropeller/blob/d5ba4c70dccae77747f5f7902de324a7a06debfe/pkg/controller/nodes/task/k8s/plugin_manager.go#L478>

so for every plugin (k8s) you have to implement this - <https://github.com/flyteorg/flyteplugins/blob/9e00636f9b2fcb84b5bc2c8dcece2b9b2bd0935e/go/tasks/pluginmachinery/k8s/plugin.go#L32>

But <@UPBBNMXD1> is right in a way, it is not currently implemented for Spark, but you can