https://flyte.org
Join Slack
Is there a way to have `pyflyte backfill` ignore the cache? This would be similar to launching a wo...
# flyte-support
f
Is there a way to have 
pyflyte backfill
ignore the cache? This would be similar to launching a workflow and selecting the advanced option "Overwrite cached outputs". I don't see this in the 
pyflyte backfill
help page (BTW, the doc is blank currently) but maybe there is a setting I can include in my config? The alternative is for me to push a change that disables the cache for the specific tasks I want to ignore cache.
👀 1
f
ya the doc seems to be down, we are figuring out the problem
also you are right, we do not have ovewrite cache in the cli
you can use the programmatic interface or i can create a PR to add it in
@fancy-yak-23698 I see that actually we do not have cache support in node metadata in flyte today - https://github.com/flyteorg/flyte/blob/5d24d23ab6c70f6d1ec3a5d8dc3ba5ea4edf7f5b/flyteidl/protos/flyteidl/core/workflow.proto#L137 Union has this - cc @hallowed-mouse-14616
f
Thanks for looking into it. I'd be happy to create an issue if that helps.
f
yes please do
f
Just getting back to this. I created this issue: https://github.com/flyteorg/flyte/issues/4917
f
@fancy-yak-23698 we cannot commit to it right now.
f
Understood. My workaround has been to disable cache on tasks that I know will have restated data for the purpose of the backfill. It's not that bad because for most runs of the workflow, the cache is not used because the inputs change with each daily run. So disabling cache might really be what I want in general for these tasks.
I see this was addressed in this PR: https://github.com/flyteorg/flytekit/pull/2214 and it's available in 
flytekit==1.11.0
according to the release notes. Thanks flyte team!
6 Views
Open in Slack
PreviousNext
Powered by

Flyte

Flyte enables production-grade orchestration for machine learning workflows and data processing created to accelerate local workflows to production.

Powered by