https://flyte.org logo
Title
d

Derek Yu

02/17/2023, 2:06 PM
Hi there! Anyone have ideas on what can cause a
task
to succeed but never run/schedule a pod ? Or know where to start troubleshooting? 🕵️‍♂️ -- More details -- The task in question has caching disabled, and is a
map
task. Have searched all flyte component logs for the executions and the only errors I see are these warning ⚠️ messages
Failed to fetch override values when assigning task resource default values

Failed to fetch override values when assigning execution queue
Trying to disable the overrides didn't help either. And the task repeatedly succeeds without ever running the task. cc: @Heidi Hurst
k

Ketan (kumare3)

02/17/2023, 3:13 PM
Without caching enabled this is not possible- are you are the subtasks don’t have cache enabled
d

Derek Yu

02/17/2023, 3:16 PM
Hm good to know, we see
Caching was disabled for this execution.
in the console for the task in question That being said, we did not disable in the task decorator, but via the new
Overwrite cached outputs
flag. Perhaps that may not be working properly...
d

Dan Rammer (hamersaw)

02/17/2023, 3:25 PM
Ah, ok. So it sounds like the task definition has caching enabled, You are using it in a map task with something like:
@task(cache=True, cache_version="1.0")
def a_mappable_task

@workflow
def my_map_workflow(a: List[int]):
    mapped_out = map_task(a_mappable_task)(a=a)
And then when you execute from the UI (or elsewhere) you're selecting to overwrite the cache values?
d

Derek Yu

02/17/2023, 3:25 PM
yep exactly!
d

Dan Rammer (hamersaw)

02/17/2023, 3:27 PM
If so, this is a construct of how map tasks are implemented. Namely, as a separate plugin rather than a core Flyte component. The "cache overwrite" functionality is not currently not support in map tasks. It is possible to integrate it - would you mind filing a bug describing how cache overwrite does not work with map tasks? [flyte-bug]
d

Dan Rammer (hamersaw)

02/17/2023, 3:27 PM
but your current execution should be just reading the values from the cache right?
k

Ketan (kumare3)

02/17/2023, 3:28 PM
@Dan Rammer (hamersaw) we need to find you time
d

Derek Yu

02/17/2023, 3:31 PM
Ah I see..good to know that it's not supported. Will file an issue! If that's the case then I guess yeah it may still be reading from cache. We will try disabling cache in the task decorator and test. Thanks you two 🙏 Issue filed