Hey folks quick question If flyte is deployed on a k8s clust

Question

Hey folks quick question If flyte is deployed on a k8s cluster can it submit spark jobs on a different k8s cluster

katrina · Answer

hey < Rupsha Chaudhuri> do you mean submit jobs to an existing spark deployment or bring up the on a separate k8s cluster

Rupsha Chaudhuri · Answer

the latter

katrina · Answer

hey < Rupsha Chaudhuri> i don t believe this is possible since a single propeller instance runs the entirety of the tasks within a flyte workflow on the same cluster there are options for flytepropeller or customizing execution cluster placement within a set up

Rupsha Chaudhuri · Answer

thanks I just stumbled upon the multi cluster documentation I ll look into it

katrina · Answer

great please let us know if you have any questions fyi here are more docs on <https docs flyte org en latest deployment multicluster html|multi cluster> not sure if those are the ones you ve already found smile

Ketan (kumare3) · Answer

< katrina> i think this is actually possible

Ketan (kumare3) · Answer

cc < Haytham Abuelfutuh> do you remember if we made it to the plugin manager cc < Rupsha Chaudhuri>

Ketan (kumare3) · Answer

ideally < Rupsha Chaudhuri> better to co locate on the same cluster

Ketan (kumare3) · Answer

better management

Ketan (kumare3) · Answer

and you can simply scale flyte clusters

Ketan (kumare3) · Answer

< Rupsha Chaudhuri> so you can create a custom kubeclient for some plugins

Rupsha Chaudhuri · Answer

thanks let me look into that

Ketan (kumare3) · Answer

so for every plugin k8s you have to implement this <https github com flyteorg flyteplugins blob 9e00636f9b2fcb84b5bc2c8dcece2b9b2bd0935e go tasks pluginmachinery k8s plugin go L32>

Ketan (kumare3) · Answer

But < katrina> is right in a way it is not currently implemented for Spark but you can