https://flyte.org
Join Slack
hi community, is it possible to support dictionary with Any value type in literal types `def task(i...
# flyte-support
l
hi community, is it possible to support dictionary with Any value type in literal types 
def task(input: Dict[str, Any])
? Right now it is transformed into a picklefile
t
no it needs to map to something on the flyteidl side, it needs to be mapped to a known thing
and the only known thing that can handle python’s Any is currently a pickling it.
what behavior are you looking for? is union not enough?
is it really an any?
l
uh yes, it is something like: 
{'location': {'com.linkedin.photon.connect.kernel.configs.shared.HdfsLocation': {'path': '/user/test/pcv2-pipeline-home-dir-keras-pipeline'}}}
so it contains {str: str, str: Dict[str: str]} and nested dict. what is inside the dictionary is unknown beforehead
I think LiteralMap can store Any types in the value, but there is a missing LiteralType for it.
because LiteralType.map_value_type can only have one subtype
t
yeah… that was by design, kinda
we actually talked about the idea of adding multi-variate map types a few months ago.
and that was rejected for now just from the sheer complexity of implementation
but i’m still confused…
are you saying that the type of the value of the dictionary is unknown? or can be multiple things?
if it’s multiple things you should be able to make that a union.
if it’s truly unknown, then pickling does feel correct
l
I guess I can iterate through each dictionary entry to determine a Union of all the types
t
i think i need to see a more concrete example.
so in python, you’re saying the correct signature is Any? like that’s the legitimate return type
l
let me explore using Union types. I think that could potentially solve our case. If not, I will provide you with a more concrete example. Thanks yee!
36 Views
Open in Slack
PreviousNext
Powered by

Flyte

Flyte enables production-grade orchestration for machine learning workflows and data processing created to accelerate local workflows to production.

Powered by