cuddly-jelly-27016
10/21/2024, 4:01 PMFlyteFile
. Everything went well until I tried to define a custom TypeTransformer
for it, which I wanted to behave exactly like `FlyteFile`'s type transformer with a few additions.
Pseudocode:
class Resource(Generic[T], FlyteFile):
...
class ResourceTransformer(TypeTransformer[Resource]):
def to_literal(ctx, python_val, python_type, expected):
lit = TypeEngine().to_literal(ctx, python_val, FlyteFile, None) # <- construct a literal representation of the FlyteFile
... # now, do something with the literal like embedding a hash etc.
return lit
TypeEngine.register(ResourceTransformer(...))
The problem is that the current type resolution will now resolve FlyteFile
as the appropriate transformer for a Resource
object, as per this code:
https://github.com/flyteorg/flytekit/blob/28da983bba36e243bc671f9ba1aa53a0791efd62/flytekit/core/type_engine.py#L756-L760
There is a TODO comment a little above that sort of acknowledges this problem, and proposes to use an ordered dict, presumably to walk it in reverse and resolve the Resource
transformer before the FlyteFile
transformer. This might be brittle though for multiple inheritance use cases like the one here, maybe even with a third resource type sprinkled in that gets registered way after the other two.
What I propose instead is trying to walk down the `python_val`'s class hierarchy and greedily match the first type transformer available, roughly like so:
# flytekit/core/type_engine.py
# begins at the code block linked above
# Step
# To facilitate cases where users may specify one transformer for multiple types that all inherit from
# parent.
if hasattr(python_type, "__mro__"):
for base_type in inspect.getmro(python_type):
try:
return cls._REGISTRY[base_type]
except KeyError:
continue
For an inheritance chain like mine with MRO Resource -> FlyteFile -> ... -> object
, this would lead to a match directly at the Resource
level. What do you think?
Expected behavior
TypeEngine().to_literal(ctx, python_val, python_type, expected)
calls the ResourceTransformer
class to construct the literal, but instead it calls `FlyteFile`'s associated transformer.
Additional context to reproduce
No response
Screenshots
No response
Are you sure this issue hasn't been raised already?
• Yes
Have you read the Code of Conduct?
• Yes
flyteorg/flytecuddly-jelly-27016
10/21/2024, 4:01 PM