< acceptable policeman 57188> there were some changes to the Flyte #announcements

<@UPBBNMXD1> there were some changes to the auth i...

astonishing-lizard-78628

04/28/2022, 12:48 AM

@acceptable-policeman-57188 there were some changes to the auth in Flyte Admin a couple weeks ago that are now in

v1.0.0

that have partially broken the

kubernetes_service_account

, i.e. some of our workflows are executing with the declared service account and some are just executing with

default

. I'm still working on debugging it.

astonishing-lizard-78628

04/28/2022, 1:16 AM

If I describe executions, I'm getting the correct thing for some of my (newer) workflows, e.g.

Copy code

flytectl get execution --admin.endpoint avflyteadmin.pdx.l5.woven-planet.tech:443 -p avexampleworkflows -d dev atbnkl5564m49swgmqxk --output yaml
...
  securityContext:
    runAs:
      k8sServiceAccount: avexampleworkflows

astonishing-lizard-78628

04/28/2022, 1:17 AM

But describing the execution for some of my older workflows ends up with:

Copy code

securityContext:
    runAs:
      k8sServiceAccount: default

astonishing-lizard-78628

04/28/2022, 1:18 AM

The launch plan version for this exact execution above ^^^^ has:

Copy code

spec:
  annotations: {}
  authRole:
    kubernetesServiceAccount: avexampleworkflows

but it is ending up as

default

in the execution

astonishing-lizard-78628

04/28/2022, 1:20 AM

Clearly https://github.com/flyteorg/flyteadmin/pull/407 must have broken it (or https://github.com/flyteorg/flyteadmin/pull/401 or https://github.com/flyteorg/flyteadmin/pull/400)

astonishing-lizard-78628

04/28/2022, 1:24 AM

These two workflows - one with the correct k8sServiceAccount and one with "default" are in the same Flyte project and domain.

astonishing-lizard-78628

04/28/2022, 2:36 AM

I think

flytekit==0.23.0b1

is just too old. The workflows using

default

are using that

flytekit

version. Workflows picking up the correct service account are using a newer version of

flytekit

. Working on confirming this...

astonishing-lizard-78628

04/28/2022, 4:47 AM

It turned out not to be a

flytekit

versioning issue. @acceptable-policeman-57188 the issue is that in new Admin version when users run

flyte-cli register-files

now

--output-location-prefix

is required in order for

----kubernetes-service-account

to actually take effect.

👍 1

icy-agent-73298

04/28/2022, 5:17 AM

Hi @astonishing-lizard-78628 this was decision we made here https://github.com/flyteorg/flyteadmin/pull/378#discussion_r834785632 We have a proposal to go back to fallbacks https://github.com/flyteorg/flyteadmin/pull/412 In my view having fallbacks can be confusing for the user since we have multiple level of overrides but i will let you guyz take a consensus call on this. But until then i agree you would have to pass this in as parameter for execution create request or if you feel this is common across project-domain then you can use the matchable attribute for workflow execution config which we introduced now to make it easier https://docs.flyte.org/projects/flytectl/en/latest/gen/flytectl_update_workflow-execution-config.html You can override this for a workflow aswell. so you can have • project-domain wf config : sa : sa1 This will use sa1 if execution or lpSpec doesn’t have any workflow execution config (Meaning of no workflow exeuction config is right now based on non emptiness of the fields) • project -domain -workflowname wf config : sa : s2 This will use sa2 if execution or lpSpec doesn’t have any workflow execution config

acceptable-policeman-57188

04/28/2022, 4:09 PM

hey can I ask why you're using a beta flytekit release in production? i'd recommend upgrading 👍 also can you elaborate on

now --output-location-prefix is required in order for ----kubernetes-service-account to actually take effect.

astonishing-lizard-78628

04/28/2022, 5:24 PM

@acceptable-policeman-57188 with Flyte release v1.0.0 on EKS and any version of flytekit when I

flyte-cli register-files --kubernetes-service-account <accountName>

my task pods no longer run with the declared service account (they run with "default") unless you also pass --output-location-prefix

astonishing-lizard-78628

04/28/2022, 5:26 PM

I had Flyte v0.19.3 backend components before and passing

--kubernetes-service-account <accountName>

alone was good enough.

astonishing-lizard-78628

04/28/2022, 5:28 PM

We're fine now, but it took me all day to discover and diagnose this behavior. Just letting you know, since this will assuredly cause someone else problems.

acceptable-policeman-57188

05/03/2022, 4:38 PM

hey @astonishing-lizard-78628 sorry for the late reply (been out for conference stuff) but thank you for raising this! i had a proposal that fixes this and this is valuable feedback to revisit that. thanks as always for your patience and detailed feedback

acceptable-policeman-57188

05/17/2022, 9:13 PM

hey @astonishing-lizard-78628 this is is very belated, but I wanted to let you know the proposal got merged and is part of the 1.0.1 release (which should also include the fix for subworkflows service accounts not being propagated) in case you'd like to upgrade 🙂

169 Views

Open in Slack

Previous Next