https://flyte.org
Join Slack
<@UPBBNMXD1> there were some changes to the auth in Flyte Admin a couple weeks ago that are now in `...
# announcements
a
@acceptable-policeman-57188 there were some changes to the auth in Flyte Admin a couple weeks ago that are now in 
v1.0.0
that have partially broken the 
kubernetes_service_account
, i.e. some of our workflows are executing with the declared service account and some are just executing with 
default
. I'm still working on debugging it.
If I describe executions, I'm getting the correct thing for some of my (newer) workflows, e.g.
Copy code
flytectl get execution --admin.endpoint avflyteadmin.pdx.l5.woven-planet.tech:443 -p avexampleworkflows -d dev atbnkl5564m49swgmqxk --output yaml
...
  securityContext:
    runAs:
      k8sServiceAccount: avexampleworkflows
But describing the execution for some of my older workflows ends up with:
Copy code
securityContext:
    runAs:
      k8sServiceAccount: default
The launch plan version for this exact execution above ^^^^ has:
Copy code
spec:
  annotations: {}
  authRole:
    kubernetesServiceAccount: avexampleworkflows
but it is ending up as 
default
in the execution
These two workflows - one with the correct k8sServiceAccount and one with "default" are in the same Flyte project and domain.
I think 
flytekit==0.23.0b1
is just too old. The workflows using 
default
are using that 
flytekit
version. Workflows picking up the correct service account are using a newer version of 
flytekit
. Working on confirming this...
It turned out not to be a 
flytekit
versioning issue. @acceptable-policeman-57188 the issue is that in new Admin version when users run 
flyte-cli register-files
now 
--output-location-prefix
is required in order for 
----kubernetes-service-account
to actually take effect.
👍 1
i
Hi @astonishing-lizard-78628 this was decision we made here https://github.com/flyteorg/flyteadmin/pull/378#discussion_r834785632 We have a proposal to go back to fallbacks https://github.com/flyteorg/flyteadmin/pull/412 In my view having fallbacks can be confusing for the user since we have multiple level of overrides but i will let you guyz take a consensus call on this. But until then i agree you would have to pass this in as parameter for execution create request or if you feel this is common across project-domain then you can use the matchable attribute for workflow execution config which we introduced now to make it easier https://docs.flyte.org/projects/flytectl/en/latest/gen/flytectl_update_workflow-execution-config.html You can override this for a workflow aswell. so you can have • project-domain wf config : sa : sa1 This will use sa1 if execution or lpSpec doesn’t have any workflow execution config (Meaning of no workflow exeuction config is right now based on non emptiness of the fields) • project -domain -workflowname wf config : sa : s2 This will use sa2 if execution or lpSpec doesn’t have any workflow execution config
a
hey can I ask why you're using a beta flytekit release in production? i'd recommend upgrading 👍 also can you elaborate on 
now --output-location-prefix is required in order for ----kubernetes-service-account to actually take effect.
a
@acceptable-policeman-57188 with Flyte release v1.0.0 on EKS and any version of flytekit when I 
flyte-cli register-files --kubernetes-service-account <accountName>
my task pods no longer run with the declared service account (they run with "default") unless you also pass --output-location-prefix
I had Flyte v0.19.3 backend components before and passing 
--kubernetes-service-account <accountName>
alone was good enough.
We're fine now, but it took me all day to discover and diagnose this behavior. Just letting you know, since this will assuredly cause someone else problems.
a
hey @astonishing-lizard-78628 sorry for the late reply (been out for conference stuff) but thank you for raising this! i had a proposal that fixes this and this is valuable feedback to revisit that. thanks as always for your patience and detailed feedback
hey @astonishing-lizard-78628 this is is very belated, but I wanted to let you know the proposal got merged and is part of the 1.0.1 release (which should also include the fix for subworkflows service accounts not being propagated) in case you'd like to upgrade 🙂
168 Views
Open in Slack
PreviousNext
Powered by

Flyte

Flyte enables production-grade orchestration for machine learning workflows and data processing created to accelerate local workflows to production.

Powered by