Hey folks.. quick question: If flyte is deployed o...
# flyte-support
l
Hey folks.. quick question: If flyte is deployed on a k8s cluster, can it submit spark jobs on a different k8s cluster?
👀 1
a
hey @little-cricket-84530 do you mean submit jobs to an existing spark deployment or bring up the virtual cluster on a separate k8s cluster?
l
the latter
a
hey @little-cricket-84530 i don't believe this is possible since a single propeller instance runs the entirety of the tasks within a flyte workflow on the same cluster. there are options for scaling out flytepropeller or customizing execution cluster placement within a multi-cluster set-up
l
thanks.. I just stumbled upon the multi cluster documentation. I'll look into it
a
great, please let us know if you have any questions! fyi here are more docs on multi-cluster (not sure if those are the ones you've already found 😄)
f
@acceptable-policeman-57188 i think this is actually possible
👀 1
cc @high-park-82026 do you remember if we made it to the plugin manager - cc @little-cricket-84530
ideally @little-cricket-84530 better to co-locate on the same cluster
better management
and you can simply scale flyte clusters
l
thanks! let me look into that
f
But @acceptable-policeman-57188 is right in a way, it is not currently implemented for Spark, but you can
169 Views