hi community is it possible to support dictionary with Any v

Question

hi community is it possible to support dictionary with Any value type in literal types `def task input Dict str Any ` Right now it is transformed into a picklefile

Yee · Answer

no it needs to map to something on the flyteidl side it needs to be mapped to a known thing

Yee · Answer

and the only known thing that can handle python s Any is currently a pickling it

Yee · Answer

what behavior are you looking for is union not enough

Yee · Answer

is it really an any

Yubo Wang · Answer

uh yes it is something like ` location com linkedin photon connect kernel configs shared HdfsLocation path user test pcv2 pipeline home dir keras pipeline `

Yubo Wang · Answer

so it contains str str str Dict str str and nested dict what is inside the dictionary is unknown beforehead

Yubo Wang · Answer

I think LiteralMap can store Any types in the value but there is a missing LiteralType for it

Yubo Wang · Answer

because LiteralType map value type can only have one subtype

Yee · Answer

yeah that was by design kinda

Yee · Answer

we actually talked about the idea of adding multi variate map types a few months ago

Yee · Answer

and that was rejected for now just from the sheer complexity of implementation

Yee · Answer

but i m still confused

Yee · Answer

are you saying that the type of the value of the dictionary is unknown or can be multiple things

Yee · Answer

if it s multiple things you should be able to make that a union

Yee · Answer

if it s truly unknown then pickling does feel correct

Yubo Wang · Answer

I guess I can iterate through each dictionary entry to determine a Union of all the types

Yee · Answer

i think i need to see a more concrete example

Yee · Answer

so in python you re saying the correct signature is Any like that s the legitimate return type

Yubo Wang · Answer

let me explore using Union types I think that could potentially solve our case If not I will provide you with a more concrete example Thanks yee