https://flyte.org logo
#ask-the-community
Title
# ask-the-community
d

Dennis O'Brien

01/10/2024, 10:33 PM
My team has run into an issue since upgrading to
flytekit
1.10.2 (from 1.7.0) and I'm curious if others have run into something similar. If we have a python file that contains the definitions for the workflow, tasks, and launch plans, it fails to run in fast mode. It runs in non-fast mode fine. The error looks like this:
Copy code
$ python3 workflow.py --inference_date=2024-01-07 --n_days_horizon=180

╭─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────── Traceback (most recent call last) ────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╮
│ /.../workflow.py:319 in <module>                                                                                                                                                                                                                              │
│                                                                                                                                                                                                                                                                                                                                                                          │
│ ❱ 319 launch_plan = LaunchPlan.get_or_create(                                                                                                                                                                                                                                                                                                                            │
│                                                                                                                                                                                                                                                                                                                                                                          │
│ /.../lib/python3.10/site-packages/flytekit/core/launch_plan.py:273 in get_or_create                                                                                                                                                                                                                                              │
│                                                                                                                                                                                                                                                                                                                                                                          │
│ ❱ 273 │   │   │   │   raise AssertionError("The cached values aren't the same as the current c                                                                                                                                                                                                                                                                           │
╰──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
AssertionError: The cached values aren't the same as the current call arguments
If we move the launch plans to a separate file, we can run in fast mode. I created a github issue here: https://github.com/flyteorg/flyte/issues/4716
I found one other thread here that mentions "cached values do not match the inputs" but I don't think this is a related issue. Including it here in case it is a helpful clue. https://flyte-org.slack.com/archives/CP2HDHKE1/p1701098778395889
k

Ketan (kumare3)

01/11/2024, 5:57 AM
cc @Yee can you help with this. this seems like the bug in the caching logic - https://github.com/flyteorg/flytekit/blob/949bc1b40e1966b409c821b96011d900241b22fd/flytekit/core/launch_plan.py#L262
y

Yee

01/11/2024, 5:40 PM
this is in local execution right?
could you try clearing the local cache?
@Dennis O'Brien ignore that sorry, i was remembering the wrong error message. this is unrelated to local caching. could you paste the workflow.py code by chance?
d

Dennis O'Brien

02/01/2024, 2:10 AM
Sorry to miss this @Yee! Someone posted a minimal example in the issue thread. https://github.com/flyteorg/flyte/issues/4716#issuecomment-1893423240
y

Yee

02/01/2024, 2:12 AM
thanks.
i will try to spend a day on taking care of these issues tomorrow