• Rupsha Chaudhuri

    Rupsha Chaudhuri

    1 month ago
    Hey folks.. quick question: If flyte is deployed on a k8s cluster, can it submit spark jobs on a different k8s cluster?
  • k

    katrina

    1 month ago
    hey @Rupsha Chaudhuri do you mean submit jobs to an existing spark deployment or bring up the virtual cluster on a separate k8s cluster?
  • Rupsha Chaudhuri

    Rupsha Chaudhuri

    1 month ago
    the latter
  • k

    katrina

    1 month ago
    hey @Rupsha Chaudhuri i don't believe this is possible since a single propeller instance runs the entirety of the tasks within a flyte workflow on the same cluster. there are options for scaling out flytepropeller or customizing execution cluster placement within a multi-cluster set-up
  • Rupsha Chaudhuri

    Rupsha Chaudhuri

    1 month ago
    thanks.. I just stumbled upon the multi cluster documentation. I'll look into it
  • k

    katrina

    1 month ago
    great, please let us know if you have any questions! fyi here are more docs on multi-cluster (not sure if those are the ones you've already found 😄)
  • Ketan (kumare3)

    Ketan (kumare3)

    1 month ago
    @katrina i think this is actually possible
  • cc @Haytham Abuelfutuh do you remember if we made it to the plugin manager - cc @Rupsha Chaudhuri
  • ideally @Rupsha Chaudhuri better to co-locate on the same cluster
  • better management
  • and you can simply scale flyte clusters
  • Rupsha Chaudhuri

    Rupsha Chaudhuri

    1 month ago
    thanks! let me look into that
  • Ketan (kumare3)

    Ketan (kumare3)

    1 month ago
  • But @katrina is right in a way, it is not currently implemented for Spark, but you can