Hi all, I have a custom object that implements `St...
# flyte-support
m
Hi all, I have a custom object that implements
StructuredDatasetEncoder/Decoder
and I’d like to compose this object inside another object. For example:
Copy code
class A:
    ...
class AEncoder(StructuredDatasetEncoder):
    ...
class ADecoder(StructuredDatasetDecoder):
    ...

class B(BaseModel):
    a1: A
    a2: A
It’s not clear to me how to get this to work though. I can define encoder and decoder classes for
B
but not sure how to get them to use the existing encoder/decoder functionality from
A
. Running this without defining and encoder/decoder for
B
runs locally but fails with
Failed to bind data… with literal type…
remotely. Appreciate your help.
m
Yes, I registered the encoder/decoder for the nested class (
A
above) like this:
Copy code
StructuredDatasetTransformerEngine.register(
    ADecoder(python_type=A, supported_format=PARQUET)
)
StructuredDatasetTransformerEngine.register(
    AEncoder(python_type=A, supported_format=PARQUET)
but not sure how to write encoder/decoders for
B
that use `A`’s encoder/decoders. For example, encoders are expected to return a
literals.StructuredDataset
but not sure how to compose these to contain other
literals.StructuredDataset
instances. The error message above is without an encoder/decoder for
B
g
qq: what is BaseModel? is it the pydantic baseModel? would you also want to serialize B to structured dataset?
m
Yes, sorry, pydantic BaseModel. Just an example of how I could compose
A
inside
B
. I don’t have strong feelings about how
B
is serialized, I just want that to call the encoder/decoder logic we have for
A
because it’s got a bunch of details in it. I looked into writing an encoder/decoder for
B
that calls the methods from
A
, but I can’t figure out how to combine
literals.StructuredDataset
into a collection (e.g.
LiteralCollection
) that can be returned from the encoder.
g
ok, I see. one more question, are you using pydantic plugin, it’s used to serialize the pydantic model? https://github.com/flyteorg/flytekit/tree/master/plugins/flytekit-pydantic/flytekitplugins/pydantic
actually, you don’t need to write the encoder/decoder for
B
. If you install pydantic plugin, it contains pydantic transformer, which will serialize all the value (a1, a2) in your pydantic model, and add them to protobuf. It will find corresponding transformer for a1, a2, if you already registered.
m
Ah great! I’ll try that.