https://flyte.org logo
#ask-the-community
Title
# ask-the-community
r

Rupsha Chaudhuri

06/15/2022, 8:38 PM
Hey folks.. quick question: If flyte is deployed on a k8s cluster, can it submit spark jobs on a different k8s cluster?
👀 1
k

katrina

06/15/2022, 8:45 PM
hey @Rupsha Chaudhuri do you mean submit jobs to an existing spark deployment or bring up the virtual cluster on a separate k8s cluster?
r

Rupsha Chaudhuri

06/15/2022, 8:46 PM
the latter
k

katrina

06/15/2022, 8:54 PM
hey @Rupsha Chaudhuri i don't believe this is possible since a single propeller instance runs the entirety of the tasks within a flyte workflow on the same cluster. there are options for scaling out flytepropeller or customizing execution cluster placement within a multi-cluster set-up
r

Rupsha Chaudhuri

06/15/2022, 8:56 PM
thanks.. I just stumbled upon the multi cluster documentation. I'll look into it
k

katrina

06/15/2022, 9:00 PM
great, please let us know if you have any questions! fyi here are more docs on multi-cluster (not sure if those are the ones you've already found 😄)
k

Ketan (kumare3)

06/15/2022, 11:44 PM
@katrina i think this is actually possible
👀 1
cc @Haytham Abuelfutuh do you remember if we made it to the plugin manager - cc @Rupsha Chaudhuri
ideally @Rupsha Chaudhuri better to co-locate on the same cluster
better management
and you can simply scale flyte clusters
r

Rupsha Chaudhuri

06/16/2022, 12:00 AM
thanks! let me look into that
k

Ketan (kumare3)

06/16/2022, 12:00 AM
But @katrina is right in a way, it is not currently implemented for Spark, but you can
84 Views