Channels
#ask-the-community
Title
# ask-the-community
r
Ryuu
10/31/2023, 8:30 AMHow to defind my own datatype to defind the typedata for each task. I want to defind something not regular such as torch.model or sth like deeplake.dataset, ...
s
Samhita Alla
10/31/2023, 8:48 AM
Hi Ryuu, you can define a type transformer. Docs: https://docs.flyte.org/projects/cookbook/en/latest/auto_examples/extending/custom_types.html
Example plugins: https://github.com/flyteorg/flytekit/tree/master/plugins/flytekit-onnx-pytorch, https://github.com/flyteorg/flytekit/tree/master/plugins/flytekit-polars
r
Ryuu
10/31/2023, 8:55 AM@Samhita Alla I want to use specific this object class Dataset, are there any simple way to do that .
https://github.com/activeloopai/deeplake/blob/main/deeplake/core/dataset/dataset.py#L158
@Samhita Alla Cuz if i want to using the datatype of another lib which may be very complicate to rewrite. In your solution, I can understand that I have to rewrite then register to DataType engine.
s
Samhita Alla
10/31/2023, 9:51 AM
If you want add it as a native type, then you need to add a type transformer. It shouldn't be very complex to add. Else, it usually gets automatically pickled.
2 Views