Hey folks.. quick question: If flyte is deployed on a k8s cluster, can it submit spark jobs on a different k8s cluster?
06/15/2022, 8:45 PM
hey @Rupsha Chaudhuri do you mean submit jobs to an existing spark deployment or bring up the virtual cluster on a separate k8s cluster?
06/15/2022, 8:46 PM
06/15/2022, 8:54 PM
hey @Rupsha Chaudhuri i don't believe this is possible since a single propeller instance runs the entirety of the tasks within a flyte workflow on the same cluster. there are options for scaling out flytepropeller or customizing execution cluster placement within a multi-cluster set-up
06/15/2022, 8:56 PM
thanks.. I just stumbled upon the multi cluster documentation. I'll look into it
06/15/2022, 9:00 PM
great, please let us know if you have any questions! fyi here are more docs on multi-cluster (not sure if those are the ones you've already found 😄)
06/15/2022, 11:44 PM
@katrina i think this is actually possible
cc @Haytham Abuelfutuh do you remember if we made it to the plugin manager - cc @Rupsha Chaudhuri
ideally @Rupsha Chaudhuri better to co-locate on the same cluster