Hi all, I have a custom object that implements

StructuredDatasetEncoder/Decoder

and I’d like to compose this object inside another object. For example:

class A:
    ...
class AEncoder(StructuredDatasetEncoder):
    ...
class ADecoder(StructuredDatasetDecoder):
    ...

class B(BaseModel):
    a1: A
    a2: A

It’s not clear to me how to get this to work though. I can define encoder and decoder classes for

B

but not sure how to get them to use the existing encoder/decoder functionality from

A

. Running this without defining and encoder/decoder for

B

runs locally but fails with

Failed to bind data… with literal type…

remotely. Appreciate your help.

did you register the encoder & decoder? https://github.com/flyteorg/flytekit/blob/26291f192b6760e23d2e6aff418a14dd8c1880a2/flytekit/types/structured/__init__.py#L40-L42

Yes, I registered the encoder/decoder for the nested class (

A

above) like this:

StructuredDatasetTransformerEngine.register(
    ADecoder(python_type=A, supported_format=PARQUET)
)
StructuredDatasetTransformerEngine.register(
    AEncoder(python_type=A, supported_format=PARQUET)

but not sure how to write encoder/decoders for

B

that use `A`’s encoder/decoders. For example, encoders are expected to return a

literals.StructuredDataset

but not sure how to compose these to contain other

literals.StructuredDataset

instances. The error message above is without an encoder/decoder for

B

qq: what is BaseModel? is it the pydantic baseModel? would you also want to serialize B to structured dataset?

Yes, sorry, pydantic BaseModel. Just an example of how I could compose

A

inside

B

. I don’t have strong feelings about how

B

is serialized, I just want that to call the encoder/decoder logic we have for

A

because it’s got a bunch of details in it. I looked into writing an encoder/decoder for

B

that calls the methods from

A

, but I can’t figure out how to combine

literals.StructuredDataset

into a collection (e.g.

LiteralCollection

) that can be returned from the encoder.

ok, I see. one more question, are you using pydantic plugin, it’s used to serialize the pydantic model? https://github.com/flyteorg/flytekit/tree/master/plugins/flytekit-pydantic/flytekitplugins/pydantic

Ah great! I’ll try that.