Any suggestions on how to invalidate the cache whe...
# ask-the-communityg
Gopal Vashishtha
10/12/2023, 1:15 AMAny suggestions on how to invalidate the cache when running a map task? Seeing this bug where tasks complete immediately even though I've tried:
1. disabling caching by setting
cache=Falsej
Jay Ganbat
10/12/2023, 2:43 AMCan you register a new version and run ?
k
Ketan (kumare3)
10/12/2023, 3:29 AM
Use experimental.map_task
g
Gopal Vashishtha
10/12/2023, 4:53 AMI tried a new version, that didn't work. Ended up breaking one of the tasks in my workflow into two (needed to do that anyway) so that seemed to invalidate the cache.
Gopal Vashishtha
10/12/2023, 4:54 AMKetan, I was having an issue with map_task that could only be resolved by wrapping it in a dynamic task.
Copy code
@task
def t(a: int) -> int:
return a + 1
@dynamic
def t_wrapper(xs: List[int]) -> None:
map_task(t)(a=xs)
@workflow
def experimental_map_task(xs: List[int]) -> None:
t_wrapper(xs=xs)s
Samhita Alla
10/12/2023, 6:48 AM
What's the error you're seeing, @Gopal Vashishtha?
g
Gopal Vashishtha
10/12/2023, 8:04 AMIf I don't do that strange wrapper thing, then when I try to register the task I get:
Copy code
Error 0: Code: ParameterNotBound, Node Id: end-node, Description: Parameter not bound [o0]. Error 1: Code: ValueRequired, Node Id: n2, Description: Value required [Target]. Error 2: Code: ValueRequired, Node Id: n3, Description: Value required [Target]. Error 3: Code: VariableNameNotFound, Node Id: n2, Description: Variable [o0] not found on node [n2]s
Samhita Alla
10/12/2023, 8:58 AM
Are all your flyte components updated to 1.9.1?
g
Gopal Vashishtha
10/12/2023, 2:52 PMI'm using flytekit 1.9.0
s
Samhita Alla
10/12/2023, 2:57 PM
1.9.0 should work as well. @Dan Rammer (hamersaw), what could be the cause of this error?
d
Dan Rammer (hamersaw)
10/16/2023, 2:57 PMwill get to looking into this - seems to potentially be a flytekit issue where the dynamic is executed with a different version?!
k
Ketan (kumare3)
10/16/2023, 3:06 PM
This is the same question Tom is also asking
d
Dan Rammer (hamersaw)
10/16/2023, 3:07 PMand others
k
Ketan (kumare3)
10/16/2023, 3:07 PM
I can help from flytekit side
d
Dan Rammer (hamersaw)
10/16/2023, 4:16 PM@Gopal Vashishtha how are you registering this? flytectl?
k
Ketan (kumare3)
10/17/2023, 2:15 PM
@Dan Rammer (hamersaw) is this the same problem as hashing? If not can you share an example I can help
g
Gopal Vashishtha
10/19/2023, 3:17 AMended up just passing a nonce to the task as an input to overwrite cache. Yes registering with
flytectl registerk
Ketan (kumare3)
10/19/2023, 3:22 AM
Ohh is this the same as the older version of admin
d
Dan Rammer (hamersaw)
10/19/2023, 1:12 PM> Ohh is this the same as the older version of admin
Yes, @Gopal Vashishtha for some context - https://flyte-org.slack.com/archives/CP2HDHKE1/p1697574227413249?thread_ts=1696568069.202449&cid=CP2HDHKE1. TL;DR the 1.10 release will support ArrayNode.
3 Views