Thread
#announcements
    Bernhard Stadlbauer

    Bernhard Stadlbauer

    2 months ago
    Multi-cluster flyte permissions question 🧵
    We're currently evaluating switching to a multi-cluster setup to get a cleaner split between environments (i.e.
    development
    ,
    staging
    and
    production
    ). We do have a POC running and are currently testing. One of the hurdles we're seeing at the moment is related to reference_tasks (and workflows/etc.). The rough outline is, that some of our teams publish workflows which are then consumed by downstream workflows from other teams. To keep the teams docker environments separate (and thus enable multiple Python envs, etc), we're using reference tasks to connect the workflows. The problem now is that whenever a reference task is started, it will run in the same domain (i.e. k8s-cluster/k8s-namespace) as the workflow that started it. Imagine a workflow which creates some output based on data in the platform. This workflow uses a launch plan from another team to ensure required data is ingested into the platform. For separation reasons, the service account in the
    development
    domain only has read access to the data. The import workflow (from the other team) would need write access though (which it would have, if it would be running in
    production
    domain/cluster). I would think this is a limitation of propeller (as it receives the whole workflow CRD), and thus can't schedule a subworkflow to a different domain or cluster as it would need to leave the data plane then? Or are we missing something here?
    Ketan (kumare3)

    Ketan (kumare3)

    2 months ago
    Scheduling across domains is not part of the design and is a limitation across the system. The idea is tasks are portable and that permissions should be tied to the execution requester
    That being said, what you are saying is finer granularity permissions at a task level
    This is doable using pod tasks currently
    Bernhard Stadlbauer

    Bernhard Stadlbauer

    2 months ago
    @Ketan (kumare3) Thank you for your quick response here (as always)! My fear is, that when we would give the task permissions to write to the production bucket from within the development stage, this would circumvent the environment split that we were aiming for? That being said though, I don't have a lot of experience setting up permissions, so I might just be missing a concept here. As a note, we would have each environment in a different GCP project, thus accessing cross-project resources is potentially harder
    Ketan (kumare3)

    Ketan (kumare3)

    2 months ago
    yes I think its better to isolate perms
    development to production seems like an odd usecase
    would love to have a chat and understand the problem and help triage a solution if you want