miniature-barista-44022
09/15/2023, 7:58 PMStructuredDatasetEncoder/Decoder
and I’d like to compose this object inside another object. For example:
class A:
...
class AEncoder(StructuredDatasetEncoder):
...
class ADecoder(StructuredDatasetDecoder):
...
class B(BaseModel):
a1: A
a2: A
It’s not clear to me how to get this to work though. I can define encoder and decoder classes for B
but not sure how to get them to use the existing encoder/decoder functionality from A
. Running this without defining and encoder/decoder for B
runs locally but fails with Failed to bind data… with literal type…
remotely. Appreciate your help.glamorous-carpet-83516
09/15/2023, 8:02 PMminiature-barista-44022
09/15/2023, 8:10 PMA
above) like this:
StructuredDatasetTransformerEngine.register(
ADecoder(python_type=A, supported_format=PARQUET)
)
StructuredDatasetTransformerEngine.register(
AEncoder(python_type=A, supported_format=PARQUET)
but not sure how to write encoder/decoders for B
that use `A`’s encoder/decoders. For example, encoders are expected to return a literals.StructuredDataset
but not sure how to compose these to contain other literals.StructuredDataset
instances. The error message above is without an encoder/decoder for B
glamorous-carpet-83516
09/15/2023, 8:22 PMminiature-barista-44022
09/15/2023, 9:16 PMA
inside B
. I don’t have strong feelings about how B
is serialized, I just want that to call the encoder/decoder logic we have for A
because it’s got a bunch of details in it. I looked into writing an encoder/decoder for B
that calls the methods from A
, but I can’t figure out how to combine literals.StructuredDataset
into a collection (e.g. LiteralCollection
) that can be returned from the encoder.glamorous-carpet-83516
09/15/2023, 9:22 PMglamorous-carpet-83516
09/15/2023, 9:26 PMB
. If you install pydantic plugin, it contains pydantic transformer, which will serialize all the value (a1, a2) in your pydantic model, and add them to protobuf. It will find corresponding transformer for a1, a2, if you already registered.miniature-barista-44022
09/15/2023, 9:35 PM