Hi, I have just joined this community after spending about 4 hours watching thru tutorial videos and reading the flyte documents including integrations. In my use case, I have tasks of a DAG that need to be invoked across two k8s clusters. I can do this in Airflow with the use of KubernetesJobOperator, specifying the path to the kuebconfig. This way, task1 is run on cluster1 and once it finishes, task2 is run in cluster2. In Flyte, Is there a way to specify in the task decorator, which k8s cluster (or which kubeconfig file to use) to run on? Thanks!

Hi @Buvaneswari Ramanan and welcome to the Flyte community. I hope the content so far has been worthy of 4 hours of your time 🙏🏽 Find here deployment considerations for a multi-cluster environment and how to leverage it from the workflow perspective: https://docs.flyte.org/en/latest/deployment/deployment/multicluster.html#deployment-deployment-multicluster And please let us know any additional question you may have

@Buvaneswari Ramanan may I ask a question, why do you want to use 2 k8s cluster. Are they some specific requirements or for scale? If it is for scale, Flyte will automatically offload a workflow to one or the other k8s cluster. you can also create rules if it is for some reason. flyte does not natively support sending workloads to different clusters from one cluster. this is because providing high resource limiting, garbage collection, ownership becomes extremely hard across clusters

@David Espejo @Ketan (kumare3), thanks a lot for the prompt response. The link provided is certainly helpful, but does not solve my problem completely. And the requirement is not for auto scaling.

I don’t completely understand the usecase please help us understand