Rupsha Chaudhuri
1 month agoHey folks.. quick question: If flyte is deployed on a k8s cluster, can it submit spark jobs on a different k8s cluster?- k
katrina
1 month agohey @Rupsha Chaudhuri do you mean submit jobs to an existing spark deployment or bring up the virtual cluster on a separate k8s cluster? Rupsha Chaudhuri
1 month agothe latter- k
katrina
1 month agohey @Rupsha Chaudhuri i don't believe this is possible since a single propeller instance runs the entirety of the tasks within a flyte workflow on the same cluster. there are options for scaling out flytepropeller or customizing execution cluster placement within a multi-cluster set-up Rupsha Chaudhuri
1 month agothanks.. I just stumbled upon the multi cluster documentation. I'll look into it- k
katrina
1 month agogreat, please let us know if you have any questions! fyi here are more docs on multi-cluster (not sure if those are the ones you've already found 😄) 
Ketan (kumare3)
1 month ago@katrina i think this is actually possible- cc @Haytham Abuelfutuh do you remember if we made it to the plugin manager - cc @Rupsha Chaudhuri
- ideally @Rupsha Chaudhuri better to co-locate on the same cluster
- better management
- and you can simply scale flyte clusters
- @Rupsha Chaudhuri so you can create a custom kubeclient for some plugins - https://github.com/flyteorg/flytepropeller/blob/d5ba4c70dccae77747f5f7902de324a7a06debfe/pkg/controller/nodes/task/k8s/plugin_manager.go#L478
Rupsha Chaudhuri
1 month agothanks! let me look into thatKetan (kumare3)
1 month agoso for every plugin (k8s) you have to implement this - https://github.com/flyteorg/flyteplugins/blob/9e00636f9b2fcb84b5bc2c8dcece2b9b2bd0935e/go/tasks/pluginmachinery/k8s/plugin.go#L32- But @katrina is right in a way, it is not currently implemented for Spark, but you can
Join thread in Slack
View count: 5