I’m seeing a strange error whilst executing a fair...
# flyte-supporta
agreeable-kitchen-44189
02/07/2023, 1:17 PMI’m seeing a strange error whilst executing a fairly nested workflow:
Copy code
Workflow[<worfklow-id>] failed. RuntimeExecutionError: max number of system retry attempts [51/50] exhausted. Last known status message: Failed to check Catalog for previous results: DataCatalog failed to get dataset for ID resource_type:TASK name:"<task-id>" version:"<task-version>" : rpc error: code = InvalidArgument desc = missing projectprojectagreeable-kitchen-44189
02/07/2023, 1:20 PMWe’re running Flyte 1.3.0
h
hallowed-mouse-14616
02/07/2023, 11:16 PMThis is very strange. There should be no situation where this lookup does not have an associated project. Do you have any more information? Ex. is it always the same task? Is it a reference task? this only happens with multiple layers of nested workflows? Is there any reproducible minimal example?
a
agreeable-kitchen-44189
02/08/2023, 6:24 AMGood morning @hallowed-mouse-14616.
I’ll try to produce a minimal example today (and maybe even debug propeller in case I find one).
It’s slighly tricky to reproduce, as this everything Flyte has to offer - heavily subworkflow nesting, some reference tasks, dynamic task etc.
Will report back once I know more
h
hallowed-mouse-14616
02/08/2023, 10:18 AMOk, keep me posted! I'm wondering if it may be related to reference tasks, there was an issue filed regarding project declaration issues with reference launch plans. I may have time today to try and quickly repro this.
a
agreeable-kitchen-44189
02/08/2023, 10:20 AMI found a minimum example, turns out that it has nothing to do with reference launch plans. You can find it here.
Currently debugging in propeller to check where this occurs
🙏 1
agreeable-kitchen-44189
02/08/2023, 11:26 AM@hallowed-mouse-14616 Could it be that dynamic workflows are generally broken at the moment?
The following already breaks for me:
Copy code
from flytekit import workflow, task, dynamic
@task
def some_task():
print("Do something")
@dynamic
def dynamic_task():
some_task()
@workflow
def my_workflow():
dynamic_task()my_workflow()agreeable-kitchen-44189
02/08/2023, 1:54 PMI think the offending PR is this one.
Without this change, things work for me
h
hallowed-mouse-14616
02/08/2023, 1:55 PMOK, thank you so much for looking into this. I pinged the Union team to get as many eyes on this ASAP. We'll need to do a retrospect on this, regressions like this should not occur.
a
agreeable-kitchen-44189
02/08/2023, 2:21 PMThank you!
For anyone needing a quick fix I’ve ported this to my fork:
Copy code
pip install git+<https://github.com/bstadlbauer/flytekit.git@fix-dynamic-serialization> Copy code
pip install git+<https://github.com/bstadlbauer/flytekit.git@5e0a270>🙌 2
h
hallowed-mouse-14616
02/09/2023, 1:01 PMI think there is a brief comment on the issue, but for more clarity. The flytekit PR broke project / domain settings on dynamic tasks - this should have been caught in our e2e tests (that is being addressed). We still need to get this change in because of the way reference entities are executed across project / domains. In taking a look through propeller, I think there is a way we can correctly set project / domain on dynamic tasks through the compiler. Labeling this as a "bug" is incorrect, rather this is a backend change we need to make to support the flytekit PR. Hopefully this resolves the issue in the near-term and we can transparently get this patched up. Thanks so much for diving through the code and debugging this! All your effort is what makes this community so great!
a
agreeable-kitchen-44189
02/13/2023, 10:53 AMThanks @hallowed-mouse-14616 for the clarification here 👍
288 Views