melodic-magician-71351
02/21/2023, 4:09 PMclass MyDataClass:
foo: Foo = field(metadata=dict(encoder=encoder, decoder=decoder))
Is the only alternative to define and register a custom TypeTransformer
? What's the logic behind requiring something comparatively cumbersome?melodic-magician-71351
02/21/2023, 4:22 PMclass MyDataClass:
foo_str: str
def __post_init__(self):
self.foo = decoder(self.foo_str)
@classmethod
def from_foo(cls, foo: Foo):
return cls(foo_str=encoder(foo))
tall-lock-23197
Foo
, has to be a valid Flyte type. If that isn't the case, I believe the data gets pickled. @glamorous-carpet-83516, could you please confirm?glamorous-carpet-83516
02/22/2023, 5:55 AMglamorous-carpet-83516
02/22/2023, 5:56 AMuser
02/22/2023, 5:56 AMmelodic-magician-71351
02/22/2023, 8:59 AMmelodic-magician-71351
02/22/2023, 9:01 AMmm_field
to specify an alternate intermediate marshmallow type in the schema, which then needs to match up with the return/call signature of your encoder/decoder respectively.melodic-magician-71351
02/22/2023, 9:01 AMmelodic-magician-71351
02/22/2023, 9:13 AMdef import_from_str(obj_qual_name: str):
module_str, obj_qual_name = obj_qual_name.rsplit(sep='.', maxsplit=1)
module = importlib.import_module(module_str)
return getattr(module, obj_qual_name)
def full_name_from_obj(obj) -> str:
return f'{inspect.getmodule(obj).__name__}.{obj.__qualname__}'
TYPE_SERIALIZER = config(
encoder=full_name_from_obj,
decoder=import_from_str
)
@dataclass_json
@dataclass
class ModelSpec:
model: Type[PreTrainedModel]
preprocessor_factory: Type[PreprocessorFactory] = field(metadata=TYPE_SERIALIZER)
training_args: Dict = field(default_factory=dict)
model_weights: PyTorchCheckpoint
model_args: Dict = field(default_factory=dict)
weight_preprocessing: Callable = field(metadata=TYPE_SERIALIZER, default=noop)
glamorous-carpet-83516
02/22/2023, 3:26 PMmelodic-magician-71351
02/22/2023, 3:26 PMglamorous-carpet-83516
02/22/2023, 3:56 PM