Greg Friedland09/15/2023, 7:58 PM
and I’d like to compose this object inside another object. For example:
It’s not clear to me how to get this to work though. I can define encoder and decoder classes for
class A: ... class AEncoder(StructuredDatasetEncoder): ... class ADecoder(StructuredDatasetDecoder): ... class B(BaseModel): a1: A a2: A
but not sure how to get them to use the existing encoder/decoder functionality from
. Running this without defining and encoder/decoder for
runs locally but fails with
remotely. Appreciate your help.
Failed to bind data… with literal type…
Kevin Su09/15/2023, 8:02 PM
Greg Friedland09/15/2023, 8:10 PM
above) like this:
but not sure how to write encoder/decoders for
StructuredDatasetTransformerEngine.register( ADecoder(python_type=A, supported_format=PARQUET) ) StructuredDatasetTransformerEngine.register( AEncoder(python_type=A, supported_format=PARQUET)
that use `A`’s encoder/decoders. For example, encoders are expected to return a
but not sure how to compose these to contain other
instances. The error message above is without an encoder/decoder for
Kevin Su09/15/2023, 8:22 PM
Greg Friedland09/15/2023, 9:16 PM
. I don’t have strong feelings about how
is serialized, I just want that to call the encoder/decoder logic we have for
because it’s got a bunch of details in it. I looked into writing an encoder/decoder for
that calls the methods from
, but I can’t figure out how to combine
into a collection (e.g.
) that can be returned from the encoder.
Kevin Su09/15/2023, 9:22 PM
. If you install pydantic plugin, it contains pydantic transformer, which will serialize all the value (a1, a2) in your pydantic model, and add them to protobuf. It will find corresponding transformer for a1, a2, if you already registered.
Greg Friedland09/15/2023, 9:35 PM