https://flyte.org logo
Powered by
b

Bernhard Stadlbauer

07/27/2022, 10:26 AM
Multi-cluster flyte permissions question 🧵
We're currently evaluating switching to a multi-cluster setup to get a cleaner split between environments (i.e. 
development
, 
staging
and 
production
). We do have a POC running and are currently testing. One of the hurdles we're seeing at the moment is related to reference_tasks (and workflows/etc.). The rough outline is, that some of our teams publish workflows which are then consumed by downstream workflows from other teams. To keep the teams docker environments separate (and thus enable multiple Python envs, etc), we're using reference tasks to connect the workflows. The problem now is that whenever a reference task is started, it will run in the same domain (i.e. k8s-cluster/k8s-namespace) as the workflow that started it. Imagine a workflow which creates some output based on data in the platform. This workflow uses a launch plan from another team to ensure required data is ingested into the platform. For separation reasons, the service account in the 
development
domain only has read access to the data. The import workflow (from the other team) would need write access though (which it would have, if it would be running in 
production
domain/cluster). I would think this is a limitation of propeller (as it receives the whole workflow CRD), and thus can't schedule a subworkflow to a different domain or cluster as it would need to leave the data plane then? Or are we missing something here?
k

Ketan (kumare3)

07/27/2022, 3:34 PM
Scheduling across domains is not part of the design and is a limitation across the system. The idea is tasks are portable and that permissions should be tied to the execution requester
That being said, what you are saying is finer granularity permissions at a task level
This is doable using pod tasks currently
b

Bernhard Stadlbauer

07/27/2022, 9:25 PM
@Ketan (kumare3) Thank you for your quick response here (as always)! My fear is, that when we would give the task permissions to write to the production bucket from within the development stage, this would circumvent the environment split that we were aiming for? That being said though, I don't have a lot of experience setting up permissions, so I might just be missing a concept here. As a note, we would have each environment in a different GCP project, thus accessing cross-project resources is potentially harder
k

Ketan (kumare3)

07/27/2022, 9:45 PM
yes I think its better to isolate perms
development to production seems like an odd usecase
would love to have a chat and understand the problem and help triage a solution if you want
6 Views
Powered by