I have just joined this community after spending about 4 hours watching thru tutorial videos and reading the flyte documents including integrations.
In my use case, I have tasks of a DAG that need to be invoked across two k8s clusters. I can do this in Airflow with the use of KubernetesJobOperator, specifying the path to the kuebconfig. This way, task1 is run on cluster1 and once it finishes, task2 is run in cluster2.
In Flyte, Is there a way to specify in the task decorator, which k8s cluster (or which kubeconfig file to use) to run on?
@Buvaneswari Ramanan may I ask a question, why do you want to use 2 k8s cluster. Are they some specific requirements or for scale?
If it is for scale, Flyte will automatically offload a workflow to one or the other k8s cluster. you can also create rules
if it is for some reason. flyte does not natively support sending workloads to different clusters from one cluster. this is because providing high resource limiting, garbage collection, ownership becomes extremely hard across clusters
also thank you for watching 4 hours of content. hopefully you liked it. Please share with us any comments
also happy to talk more - do you work with @Sujith Samuel?
02/12/2023, 7:29 AM
@David Espejo@Ketan (kumare3), thanks a lot for the prompt response. The link provided is certainly helpful, but does not solve my problem completely. And the requirement is not for auto scaling.
I was introduced to Flyte by Sujith although I work in a different unit within Nokia
02/12/2023, 7:44 AM
I don’t completely understand the usecase please help us understand