Flyte enables production-grade orchestration for machine learning workflows and data processing created to accelerate local workflows to production.

Flyte

Multi-cluster flyte permissions question :thread:

We're currently evaluating switching to a <https://docs.flyte.org/en/latest/deployment/multicluster.htm|multi-cluster> setup to get a cleaner split between environments (i.e. `development`, `staging` and `production`). We do have a POC running and are currently testing.

One of the hurdles we're seeing at the moment is related to <https://docs.flyte.org/en/latest/deployment/multicluster.htm|reference_tasks> (and workflows/etc.). The rough outline is, that some of our teams publish workflows which are then consumed by downstream workflows from other teams. To keep the teams docker environments separate (and thus enable multiple Python envs, etc), we're using reference tasks to connect the workflows.
The problem now is that whenever a reference task is started, it will run in the same domain (i.e. k8s-cluster/k8s-namespace) as the workflow that started it.
Imagine a workflow which creates some output based on data in the platform. This workflow uses a launch plan from another team to ensure required data is ingested into the platform. For separation reasons, the service account in the `development` domain only has read access to the data. The import workflow (from the other team) would need write access though (which it would have, if it would be running in `production` domain/cluster).

I would think this is a limitation of propeller (as it receives the whole workflow CRD), and thus can't schedule a subworkflow to a different domain or cluster as it would need to leave the data plane then? Or are we missing something here?

Scheduling across domains is not part of the design and is a limitation across the system. The idea is tasks are portable and that permissions should be tied to the execution requester

That being said, what you are saying is finer granularity permissions at a task level

<@UNZB4NW3S> Thank you for your quick response here (as always)!
My fear is, that when we would give the task permissions to write to the production bucket from within the development stage, this would circumvent the environment split that we were aiming for?
That being said though, I don't have a lot of experience setting up permissions, so I might just be missing a concept here. As a note, we would have each environment in a different GCP project, thus accessing cross-project resources is potentially harder

development to production seems like an odd usecase

would love to have a chat and understand the problem and help triage a solution if you want