https://flyte.org logo
#flytekit
Title
# flytekit
r

Rahul Mehta

10/21/2022, 8:24 PM
Is there way to specify a custom type transformer for a JSON type? We'd like to use JSON over the wire instead of pickling/some other serialization format for a particular type, but wasn't able to find any examples (most of them were around types that were serialized as blobs)
e

Eduardo Apolinario (eapolinario)

10/21/2022, 9:42 PM
and what would be the type of this object?
r

Rahul Mehta

10/22/2022, 12:55 AM
A dataclass (but one that isn't decorated w/
@dataclass_json
). It has a
Generated
mixin that we'd like to register a type transformer for, and each instance of
Generated
will have a `to_dict`/`from_dict` method
Friendly bump on this thread -- is there a way to register a typetransformer that maps a class to JSON?
e

Eduardo Apolinario (eapolinario)

10/25/2022, 12:15 AM
@Rahul Mehta, you're saying that after https://github.com/flyteorg/flytekit/commit/b7ecdf60768bf8b03a849947d5e2cab0fccde1e3 you are not able to define a type transformer for
Generated
?
r

Rahul Mehta

10/25/2022, 12:16 AM
Ah, that’s what im trying to implement now - just not sure what to pass in to the LiteralType. Is the recommended way to just use to/from_json to write the json to a file and mimic the behavior of one of the other type transformers?
That change will enable me to register it once I figure this out though
e

Eduardo Apolinario (eapolinario)

10/25/2022, 12:29 AM
Take a look at the DictTransformer. You'll notice that in the simple case we simply struct the protobuf struct. Since you're defining your own transformer, you get to choose how the literal is going to be represented (i.e. if you want you can simply dump the json to a string and load from it, without worrying about writing that string to a file.)
r

Rahul Mehta

10/25/2022, 12:36 AM
Ah that sounds exactly like what I'm trying to do, thanks @Eduardo Apolinario (eapolinario)!