cool-lifeguard-49380
03/02/2023, 5:29 PMBaseModel
(which has several benefits over data classes which is why our ML engineers asked for this).
from pydantic import BaseModel
class Conf(BaseModel):
...
@task
def train(conf: Conf):
...
Assuming there is a class BaseModelTransformer(TypeTransformer[BaseModel]):
, would it be invoked if the user specifies train(conf: Conf)
or only in case they specify train(conf: BaseModel)
?cool-lifeguard-49380
03/02/2023, 5:41 PMthankful-minister-83577
thankful-minister-83577
thankful-minister-83577
thankful-minister-83577
StructuredDataset
(or like another separate custom type), the same thing should happen as if it were not in a pydantic modelthankful-minister-83577
cool-lifeguard-49380
03/02/2023, 6:39 PMwe’re actually thinking of rewriting the dataclass transformer as well, would be nice if they looked the sameI think the pydantic base model transformer could be rather simple. For pydantic, I unfortunately need to know which exact class is trying to be deserialized.
def pydantic_encoder(obj):
if isinstance(obj, BaseModel):
return {'__pydantic_model__': obj.__class__.__name__, **obj.dict()}
else:
return obj
def pydantic_decoder(obj):
if '__pydantic_model__' in obj:
model_name = obj.pop('__pydantic_model__')
model_class = globals()[model_name]
return model_class(**obj)
else:
return obj
serialized = json.dumps(m, default=pydantic_encoder)
reconstructed = json.loads(serialized, object_hook=pydantic_decoder)
assert reconstructed == m
(Maybe not get if from globals()
, first try, but save the path from which we can get it with importlib
.)
A base model has a .schema_json()
which I would also save into the protobuf. Then, at deserialization time, we load the class using importlib
, compare the schemas, and load using pydantic_decoder
.
I’m not 100% convinced of the part with importlib
😕 but don’t have a better idea yet since we need to know the class when instantiating the python value again.
Are you aware of a better approach?cool-lifeguard-49380
03/02/2023, 6:47 PMthankful-minister-83577
thankful-minister-83577
thankful-minister-83577
cool-lifeguard-49380
03/02/2023, 6:48 PMlet me play around with this early next week and get back to you?Of course. If you have an idea for a better approach, I can also try it out if you give me the hint ..
agreeable-kitchen-44189
03/03/2023, 6:07 PMcool-lifeguard-49380
03/03/2023, 7:19 PMglobals()
or importlib
anymore that yesterday I thought I would need.)cool-lifeguard-49380
03/07/2023, 5:13 PMcool-lifeguard-49380
03/07/2023, 5:13 PMcool-lifeguard-49380
03/07/2023, 5:13 PMrich-garden-69988
03/07/2023, 5:14 PMcool-lifeguard-49380
03/07/2023, 5:36 PMrich-garden-69988
03/07/2023, 5:37 PMrich-garden-69988
03/07/2023, 5:39 PMbroad-monitor-993
03/07/2023, 5:47 PMrich-garden-69988
03/07/2023, 5:52 PMcool-lifeguard-49380
03/07/2023, 5:53 PMrich-garden-69988
03/07/2023, 6:07 PMcool-lifeguard-49380
03/07/2023, 6:10 PMrich-garden-69988
03/07/2023, 6:10 PMcool-lifeguard-49380
03/07/2023, 6:15 PMrich-garden-69988
03/07/2023, 6:36 PMbroad-monitor-993
03/07/2023, 7:46 PMrich-garden-69988
03/07/2023, 8:18 PMbroad-monitor-993
03/07/2023, 8:26 PMrich-garden-69988
03/07/2023, 8:29 PMrich-garden-69988
03/07/2023, 8:29 PMthankful-minister-83577
broad-monitor-993
03/07/2023, 8:55 PM