https://flyte.org
Join Slack
<@UPBBNMXD1> <@UPTRGR537> <@UNW4VP36V> I updated to Flyte `1.0.1` and now workflows executed with `f...
# announcements
a
@acceptable-policeman-57188 @icy-agent-73298 @high-park-82026 I updated to Flyte 
1.0.1
and now workflows executed with 
flyte-cli execute-launch-plan
(with flytekit 0.26.0) lose their k8s service account that was declared with 
flyte-cli register-files --kubernetes-service-account
. When I execute the same workflow from Flyte Console the k8s service account is fine.
However, our blocker is that subworkflows are also losing the k8s service account, i.e. the parent workflow uses the declared account but the subworkflow execution goes back to 
default
-- this one is the actual blocker for us.
Now I need to revert and go back to the previous version of Flyte, so I'll lose the fix for https://github.com/flyteorg/flyte/issues/2424
a
hey @astonishing-lizard-78628 the latest flyte should actually include a fix explicitly to handle execution spec attributes formerly being dropped, so this is really unexpected! can you share the full command you use to execute workflows?
a
Let me double check the commands
h
Is it ok with you to update flytekit version?
a
flyte-cli -p avexampleworkflows -d dev -h avflyteadmin.pdx.l5.woven-planet.tech execute-launch-plan -u lp:avexampleworkflows:dev:app.workflows.fabrik.spark_test_workflow.spark_workflow:542c917bfabfd61be18d9c4c71a51d552d4f8428 -r abain
loses the declared k8s service account for me (while running from the UI is fine). This is on flytekit 0.26.0.
^^^^ I could upgrade to flytectl but this won't fix the subworkflow executions (which also lose the k8s service account even when the parent workflow is executed from the UI).
h
OH that's a much bigger issue... so you are on flyte 1.0.1 for all server-side components? propeller, admin... etc.?
so sorry you are having these issues with 1.0.0 upgrade 😞
a
Yes, I'm generally on 1.0.1 on the backend and flytekit 0.26.0 (since we have some old-style SDK tasks). Would updating to the latest flytekit solve the subworkflow issue?
a
hey @astonishing-lizard-78628 I suspect the flyte-cli issue might be due to an older version of flytekit with a bug that's since been fixed, but to verify could i ask you to use flytectl to fetch the execution spec?
h
Would updating to the latest flytekit solve the subworkflow issue?
I highly doubt that... looking into this
a
something like 
flytectl get execution -p flytesnacks -d development <name>
a
flytectl get execution --admin.endpoint avflyteadmin.pdx.l5.woven-planet.tech:443 -p avexampleworkflows -d dev f08d22ec484784cf4a06 -o json
Untitled.txt
Untitled.txt
^^^^^ That snippet is when I started the execution with 
flyte-cli execute-launch-plan
I can get users to replace 
flyte-cli execute-launch-plan
with 
flytectl create execution
, but my real blocker is subworkflows losing the k8s service account
a
gotcha. the service account issue should be fixed in newer versions of flytekit - the one you're currently on is inserting "default" as the actual default k8s service account but this has since been fixed
👍 1
a
Would upgrading flytekit also fix the issue with subworkflows? Even when I launch the parent workflow from the UI (which shows the correct k8s service account), the subworkflow execution goes back to "default"
a
yeah this seems like a back-end issue that isn't scoped to just flytekit 😞 I think @high-park-82026 was looking into this
a
Ok, thanks for letting me know. I need to revert from Flyte 1.0.1 backend components to 1.0.0 admin / datacatalog / propeller for now since we have a few big subworkflow users.
a
hey @astonishing-lizard-78628 to debug the broader issue with subworkflows (so sorry about this!), could i ask you to flytectl get an execution you launched in the UI?
a
@acceptable-policeman-57188 here is an execution (that has a subworkflow) that I launched from console: 
flytectl get execution --admin.endpoint avflyteadmin.pdx.l5.woven-planet.tech:443 -p avfleetscenes -d dev *adb2nvlmk42w44g2f98g* -o json
Untitled.txt
Untitled.txt
a
thank you! also if you do end up reverting your flyte deployment could I ask you to flytectl get a working parent workflow execution? just want to verify if the annotations are being updated in the older deployment
a
Note ^^^^ corrected the execution, now it is the 
flyte.workflows.fs1_scene_generation.FS1SceneReconstructionNoIndexSparkWorkflow
parent workflow (whose subworkflow fails)
Yes, once we finished reverting I'll re-register the workflow and describe it again for you.
FYI the failing subworkflow 
flytectl get execution --admin.endpoint avflyteadmin.pdx.l5.woven-planet.tech:443 -p avfleetscenes -d dev fsar4pvq -o json
Untitled.txt
Untitled.txt
🙏 1
@acceptable-policeman-57188 after reverting to Flyte 1.0.0 my parent workflow is now
Untitled.txt
Untitled.txt
a
hey @astonishing-lizard-78628 thanks so much! i think i found the issue with https://github.com/flyteorg/flyteadmin/pull/422
a
Here is its child workflow that is using the declared k8s service account again
Untitled.txt
Untitled.txt
a
hey @astonishing-lizard-78628 thanks again for your patience! the latest flyteadmin release has the fix for the subworkflow service accounts (confirmed this on my end with our flyte deployment) out of curiosity what flyte version did you revert back to? we're trying to understand just how broken this has been 😞
a
@acceptable-policeman-57188 I reverted back to admin, propeller and datacatalog 1.0.0 and it is working fine for me on those versions
✅ 1
164 Views
Open in Slack
PreviousNext
Powered by

Flyte

Flyte enables production-grade orchestration for machine learning workflows and data processing created to accelerate local workflows to production.

Powered by