Thomas Blom01/18/2024, 10:19 PM
from the CLI, but rather just executing the python workflow function directly.
In this case, how best to disable caching? I see that work has been done to give
some local-cache options, but that doesn't apply here.
In my codebase, I use a global flag that is typically always on to allow/disallow caching like
If I take care to turn this flag OFF in my test module BEFORE the module containing the task is loaded, I can disable caching this way. But it would be nice if there were a more programmatic way to do this that didn't rely as much on order of module loading - just something to say, "hey, flytekit, turn off local caching please!"
From the flyte console when "relaunching" a workflow on our k8s cluster, or from code that launches jobs programmatically via FlyteRemote, I can pass an
@task( cache=GLOBAL_CACHING_ENABLED ... )
parameter. Is there some flytekit API I can call to turn off local caching in a similar way? I see reference to
in this PR, but I don't find docs about it. The other option would seem to be executing
in a subprocess during the test, but this feels pretty hacky in a test that's running a very-fast workflow a few times to ensure results are either all different or all the same (which caching gets in the way of).
pyflyte local-cache clear
in the testing environment to cause the effect I'm looking for -- no use of local caching. I missed this initially, because I first thought one was required to modify whatever config file holds the config option discussed:
Am I reading this correctly? One can disable all use of the local-cache by setting the environment variable? Presumably this will become available in a forthcoming release of flytekit? (I see it was approved 8 hours ago!)
Bernhard Stadlbauer01/19/2024, 2:07 PM
Thomas Blom01/19/2024, 2:08 PM
Bernhard Stadlbauer01/19/2024, 2:08 PM