hey @Rupsha Chaudhuri do you mean submit jobs to an existing spark deployment or bring up the virtual cluster on a separate k8s cluster?
Rupsha Chaudhuri
1 month ago
the latter
k
katrina
1 month ago
hey @Rupsha Chaudhuri i don't believe this is possible since a single propeller instance runs the entirety of the tasks within a flyte workflow on the same cluster. there are options for scaling out flytepropeller or customizing execution cluster placement within a multi-cluster set-up
Rupsha Chaudhuri
1 month ago
thanks.. I just stumbled upon the multi cluster documentation. I'll look into it
k
katrina
1 month ago
great, please let us know if you have any questions! fyi here are more docs on multi-cluster (not sure if those are the ones you've already found 😄)