Hi everyone, I have a question: I want to use dic...
# flytekit
m
Hi everyone, I have a question: I want to use dicts as input to tasks, such that I do not have to touch every intermediate function if i want to add parameters at some lower level function that should get a config value from the top. Is it possible to have these dicts (recursive type definition: Literal=dict[str, Literal] | Iterable[Literal] | Scalar(python primitives) ) as inputs to tasks that still allow caching? Can flyte to this nativly? I experimented a bit already and found out that a recursive type definition is not allowed currently by flyte, because recurisve type definitions make use of forward declaration of types which flyte complains about as it cannot resolve those. But is it planned to have this functionality at some point (I think these kind of config dicts are very common in python?) What would a current work-around best-practices look like? (transformer for this custom literal type? Can I do something with flytes literal type? etc...
p
We've recently merged the
flytekitplugins-omegaconf
, it is available with the latest flytekit. Using DictConfigs as inputs to pipeline / tasks, I think you should be able to achieve what you want. Also, I think #CP2HDHKE1 is generally better suited for these sort of inquiries
m
Ok sure, description might be a bit misleading then:
# flytekit If you have any questions about our Python SDK, feel free to ask us here!
omegaconf sounds nice, although for my purposes a bit over the top i guess 😉 but that is not something i need to worry to much about, so will probably use that 😄