Alex Bain

    Alex Bain

    4 months ago
    @katrina there were some changes to the auth in Flyte Admin a couple weeks ago that are now in
    v1.0.0
    that have partially broken the
    kubernetes_service_account
    , i.e. some of our workflows are executing with the declared service account and some are just executing with
    default
    . I'm still working on debugging it.
    If I describe executions, I'm getting the correct thing for some of my (newer) workflows, e.g.
    flytectl get execution --admin.endpoint avflyteadmin.pdx.l5.woven-planet.tech:443 -p avexampleworkflows -d dev atbnkl5564m49swgmqxk --output yaml
    ...
      securityContext:
        runAs:
          k8sServiceAccount: avexampleworkflows
    But describing the execution for some of my older workflows ends up with:
    securityContext:
        runAs:
          k8sServiceAccount: default
    The launch plan version for this exact execution above ^^^^ has:
    spec:
      annotations: {}
      authRole:
        kubernetesServiceAccount: avexampleworkflows
    but it is ending up as
    default
    in the execution
    These two workflows - one with the correct k8sServiceAccount and one with "default" are in the same Flyte project and domain.
    I think
    flytekit==0.23.0b1
    is just too old. The workflows using
    default
    are using that
    flytekit
    version. Workflows picking up the correct service account are using a newer version of
    flytekit
    . Working on confirming this...
    It turned out not to be a
    flytekit
    versioning issue. @katrina the issue is that in new Admin version when users run
    flyte-cli register-files
    now
    --output-location-prefix
    is required in order for
    ----kubernetes-service-account
    to actually take effect.
    p

    Prafulla Mahindrakar

    4 months ago
    Hi @Alex Bain this was decision we made here https://github.com/flyteorg/flyteadmin/pull/378#discussion_r834785632 We have a proposal to go back to fallbacks https://github.com/flyteorg/flyteadmin/pull/412 In my view having fallbacks can be confusing for the user since we have multiple level of overrides but i will let you guyz take a consensus call on this. But until then i agree you would have to pass this in as parameter for execution create request or if you feel this is common across project-domain then you can use the matchable attribute for workflow execution config which we introduced now to make it easier https://docs.flyte.org/projects/flytectl/en/latest/gen/flytectl_update_workflow-execution-config.html You can override this for a workflow aswell. so you can have • project-domain wf config : sa : sa1 This will use sa1 if execution or lpSpec doesn’t have any workflow execution config (Meaning of no workflow exeuction config is right now based on non emptiness of the fields) • project -domain -workflowname wf config : sa : s2 This will use sa2 if execution or lpSpec doesn’t have any workflow execution config
    k

    katrina

    4 months ago
    hey can I ask why you're using a beta flytekit release in production? i'd recommend upgrading 👍 also can you elaborate on
    now --output-location-prefix is required in order for ----kubernetes-service-account to actually take effect.
    Alex Bain

    Alex Bain

    4 months ago
    @katrina with Flyte release v1.0.0 on EKS and any version of flytekit when I
    flyte-cli register-files --kubernetes-service-account <accountName>
    my task pods no longer run with the declared service account (they run with "default") unless you also pass --output-location-prefix
    I had Flyte v0.19.3 backend components before and passing
    --kubernetes-service-account <accountName>
    alone was good enough.
    We're fine now, but it took me all day to discover and diagnose this behavior. Just letting you know, since this will assuredly cause someone else problems.
    k

    katrina

    4 months ago
    hey @Alex Bain sorry for the late reply (been out for conference stuff) but thank you for raising this! i had a proposal that fixes this and this is valuable feedback to revisit that. thanks as always for your patience and detailed feedback
    hey @Alex Bain this is is very belated, but I wanted to let you know the proposal got merged and is part of the 1.0.1 release (which should also include the fix for subworkflows service accounts not being propagated) in case you'd like to upgrade 🙂