<#105 Updated HashMethod approach> Pull request op...
# flyte-github
c
#105 Updated HashMethod approach Pull request opened by wild-endeavor New approach to the
HashMethod
concept from flytekit. ## HashMethod In
flytekit
, users could annotate output types of tasks with
HashMethod(...)
where the
...
was a lambda function that would be computed and then added into the
hash
field of the Flyte Literal message. This
hash
field was then used in downstream tasks to compute the cache key for discovery (i.e. whether to run the task or read from cache). ### New approach A couple things: • The fact that hashes were computed during outputs makes a lot less sense in the dynamic v2 approach. Users can take the output of a Task, and modify the contents completely, before submitting it again to the next task. There's no easy way to know if an object has been touched and if so meaningfully so the system can't reasonably know if a hash value computed or provided at output time is still relevant. • In many instances, the inputs to cache-enabled Tasks will not be outputs of other tasks. A parent task can generate inputs, and immediately pass them downstream. so we're moving hash computation to the input side now. However, since this concept was really only ever useful in flytekit for FlyteFile, FlyteDirectory, and StructuredDataset types, this pr adds this functionality directly on the types instead of relying on the
typing.Annotated
approach. Also this only contains the change for File and Dir so far - DataFrame to be added later. ### Changes • Removed the old HashMethod object and removed the calculate hash function in the type engine. • Added a new HashMethod protocol with a standard PrecomputedValue implementation if you just want to provide a value. • Updated constructors for File/Dir to take this object, either as is, or as a str in cases where I/O isn't relevant (e.g.
from_existing_remote
). • Added I/O classes that capture and scan bytes as they are written/read. • Updated the File class to have both hash and hash_method and Dir to just have hash. The reason for not adding hash_method yet to Dir is just because the storage layer would need a much bigger change to support the scan and hash updating approach used for Files. We will add this to Dir in the future if there is demand, or maybe remove from File if we find it's never used. • Update cache key generation to respect Literals with a hash value, including cases where the Literal is a container type that has literals underneath that have hash values. (Also, this change to only look at the literals rather than the whole Inputs object is needed as we added more fields to the Inputs object (like context)). flyteorg/flyte-sdk