acoustic-carpenter-78188
09/05/2023, 12:38 AM@task(cache=True, cache_version="v1")
def t(log_level: int, a: str) -> str:
...
According to the docs, one of the inputs to cache key calculation is the task signature, but in the case of this example, it'd be great if we could ignore log_level
as part of the cache key calculation.
Goal: What should the final outcome look like, ideally?
We should be able to do something along the lines of:
@task(cache=True, cache_version="v1", ignore_input_vars=["log_level"])
def t(log_level: int, a: str) -> str:
...
This would essentially skip some of the parameters for cache key calculation purposes.
Describe alternatives you've considered
We have the ability to override the hashing mechanism used to translate python types into Flyte types, as described in https://docs.flyte.org/projects/cookbook/en/latest/auto/core/flyte_basics/task_cache.html#caching-of-non-flyte-offloaded-objects.
One could use this idea and provide constant hashes for the arguments they want to ignore, for example:
def constant_function(x: int) -> str:
return "const"
@task
def t_produce_annotated_literals() -> Annotated[int, HashMethod(constant_function)]:
log_level = ...
return log_level
@task(cache=True, cache_version="v1")
def t(log_level: int, a: str) -> str:
...
@workflow
def wf() -> str:
log_level = t_produce_annotated_literals()
return t(log_level=log_level, a="some string")
Propose: Link/Inline OR Additional context
Expose ignore_input_vars
in the @task
decorator and ensure the new interface is used during cache key calculation in both local and remote executions.
Are you sure this issue hasn't been raised already?
☑︎ Yes
Have you read the Code of Conduct?
☑︎ Yes
flyteorg/flytecuddly-jelly-27016
09/05/2023, 12:38 AM