Channels
announcements
ask-the-community
auth
conference-talks
contribute
databricks-integration
datahub-flyte
deployment
ecosystem-unionml
engineeringlabs
events
feature-discussions
flyte-bazel
flyte-build
flyte-console
flyte-deployment
flyte-documentation
flyte-github
flyte-on-gcp
flyte-ui-ux
flytekit
flytekit-java
flytelab
great-content
hacktoberfest-2022
helsing-flyte
in-flyte-conversations
integrations
introductions
jobs
konan-integration
linkedin-flyte
random
ray-integration
ray-on-flyte
release
scipy-2022-sprint
show-and-tell
sig-large-models
torch-elastic
workflow-building-ui-proj
writing-w-sfloris
Title
b
Bernhard Stadlbauer
02/07/2023, 1:17 PMI’m seeing a strange error whilst executing a fairly nested workflow:
Workflow[<worfklow-id>] failed. RuntimeExecutionError: max number of system retry attempts [51/50] exhausted. Last known status message: Failed to check Catalog for previous results: DataCatalog failed to get dataset for ID resource_type:TASK name:"<task-id>" version:"<task-version>" : rpc error: code = InvalidArgument desc = missing projectprojectWe’re running Flyte 1.3.0
d
Dan Rammer (hamersaw)
02/07/2023, 11:16 PMThis is very strange. There should be no situation where this lookup does not have an associated project. Do you have any more information? Ex. is it always the same task? Is it a reference task? this only happens with multiple layers of nested workflows? Is there any reproducible minimal example?
b
Bernhard Stadlbauer
02/08/2023, 6:24 AMGood morning @Dan Rammer (hamersaw).
I’ll try to produce a minimal example today (and maybe even debug propeller in case I find one).
It’s slighly tricky to reproduce, as this everything Flyte has to offer - heavily subworkflow nesting, some reference tasks, dynamic task etc.
Will report back once I know more
d
Dan Rammer (hamersaw)
02/08/2023, 10:18 AMOk, keep me posted! I'm wondering if it may be related to reference tasks, there was an issue filed regarding project declaration issues with reference launch plans. I may have time today to try and quickly repro this.
b
Bernhard Stadlbauer
02/08/2023, 10:20 AMI found a minimum example, turns out that it has nothing to do with reference launch plans. You can find it here.
Currently debugging in propeller to check where this occurs
@Dan Rammer (hamersaw) Could it be that dynamic workflows are generally broken at the moment?
The following already breaks for me:
from flytekit import workflow, task, dynamic
@task
def some_task():
print("Do something")
@dynamic
def dynamic_task():
some_task()
@workflow
def my_workflow():
dynamic_task()my_workflow()I think the offending PR is this one.
Without this change, things work for me
d
Dan Rammer (hamersaw)
02/08/2023, 1:55 PMOK, thank you so much for looking into this. I pinged the Union team to get as many eyes on this ASAP. We'll need to do a retrospect on this, regressions like this should not occur.
b
Bernhard Stadlbauer
02/08/2023, 2:21 PMThank you!
For anyone needing a quick fix I’ve ported this to my fork:
pip install git+<https://github.com/bstadlbauer/flytekit.git@fix-dynamic-serialization>pip install git+<https://github.com/bstadlbauer/flytekit.git@5e0a270>d
Dan Rammer (hamersaw)
02/09/2023, 1:01 PMI think there is a brief comment on the issue, but for more clarity. The flytekit PR broke project / domain settings on dynamic tasks - this should have been caught in our e2e tests (that is being addressed). We still need to get this change in because of the way reference entities are executed across project / domains. In taking a look through propeller, I think there is a way we can correctly set project / domain on dynamic tasks through the compiler. Labeling this as a "bug" is incorrect, rather this is a backend change we need to make to support the flytekit PR. Hopefully this resolves the issue in the near-term and we can transparently get this patched up. Thanks so much for diving through the code and debugging this! All your effort is what makes this community so great!
b
Bernhard Stadlbauer
02/13/2023, 10:53 AMThanks @Dan Rammer (hamersaw) for the clarification here 👍