I’m thinking about writing a type transformer for pydantic’s

BaseModel

(which has several benefits over data classes which is why our ML engineers asked for this).

from pydantic import BaseModel

class Conf(BaseModel):
    ...

@task
def train(conf: Conf):
   ...

Assuming there is a

class BaseModelTransformer(TypeTransformer[BaseModel]):

, would it be invoked if the user specifies

train(conf: Conf)

or only in case they specify

train(conf: BaseModel)

?

yeah this will work.

If we go forward with this, I will upstream 👍

we’re actually thinking of rewriting the dataclass transformer as well, would be nice if they looked the same

I think the pydantic base model transformer could be rather simple. For pydantic, I unfortunately need to know which exact class is trying to be deserialized.

def pydantic_encoder(obj):
    if isinstance(obj, BaseModel):
        return {'__pydantic_model__': obj.__class__.__name__, **obj.dict()}
    else:
        return obj

def pydantic_decoder(obj):
    if '__pydantic_model__' in obj:
        model_name = obj.pop('__pydantic_model__')
        model_class = globals()[model_name]
        return model_class(**obj)
    else:
        return obj

serialized = json.dumps(m, default=pydantic_encoder)

reconstructed = json.loads(serialized, object_hook=pydantic_decoder)

assert reconstructed == m

(Maybe not get if from

globals()

, first try, but save the path from which we can get it with

importlib

.) A base model has a

.schema_json()

which I would also save into the protobuf. Then, at deserialization time, we load the class using

importlib

, compare the schemas, and load using

pydantic_decoder

. I’m not 100% convinced of the part with

importlib

😕 but don’t have a better idea yet since we need to know the class when instantiating the python value again. Are you aware of a better approach?

let me play around with this early next week and get back to you?

let me play around with this early next week and get back to you?

Of course. If you have an idea for a better approach, I can also try it out if you give me the hint ..

We do have a custom de/serializer like this within Pachama, I remember it having some slight drawbacks but not exactly which. I’m currently OOO but will be back Monday to take a look

@thankful-minister-83577 I gave this some more thoughts and actually, if I’m not mistaken, it can be done way simpler and cleaner than I thought yesterday. What do you think about this implementation? And about what I noted here? (It doesn’t rely on this hack with

globals()

or

importlib

anymore that yesterday I thought I would need.)

Want to have a quick call potentially?

Interested in your thoughts about the implementation and whether this is interesting to upstream like this or with modifications @broad-monitor-993 🙂

I think we'd benefit from automatic type registration, so downstream implementers of new models don't have to explicitly register (or is that handled already?)

let’s have a call to coordinate this effort? @high-accountant-32689 @cool-lifeguard-49380 @rich-garden-69988 what times work for you this week? What timezones are y’all in?

Very flexible today, tomorrow after 2PM PT, pretty flexible Thursday after 10AM PT until 2PM

I’m in CET so it’s ~7pm now. Flexible later today after 8pm. Thursday evening my time/day your time would also work well.

Would be down for later today (if not too late for you)

Depends on when 🙂 In an hour its 8pm here. Would work for me between ~8-10:30

any time in that slot works for me 🙂

Actually I could also do from now on, not 8

@broad-monitor-993?

sorry, my day today is packed with meetings… do you mind if we chat on Thursday 11am PT, 2pm ET, 8pm CET?

Thurs 11AM works for me!

@cool-lifeguard-49380 @rich-garden-69988 mind sending me an email for a calendar invite?

I can send one over

me please

also @high-accountant-32689 probably