Hi everyone. Ive been working off the PR that adds a transformer for pydantic basemodels (https://github.com/flyteorg/flytekit/pull/1620) running into an unfortunate issue with flytepaths/directories though. These inherit from os.PathLike which falls outside of pydantic's supported json serialization/deserialization schemas. my understanding is also that flytefiles/directories need to be initialized in a particular way to work (below iv'e implemented a flytedirectory initialized inspired from the dataclass transformer). This works but seems a bit hacky so wanted to hear if anyone else encountered similar challenges. TLDR; how to serialize-deserialize flytedirectories/files in pydantic basemodels (or other objects)

from data_engine import (
    pydantic_transformer,
)  # code from <https://github.com/flyteorg/flytekit/pull/1620>


class Rax(pydantic.BaseModel):
    flytedir: str

    @pydantic.validator("flytedir")
    def validate_hax(cls, v: str) -> str:
        flytedir = make_flytedir(v)
        return str(flytedir.path)

    @classmethod
    def from_json(cls, json: str) -> "Self":
        self = cls.parse_raw(json)
        return self


@flytekit.workflow
def test_wf(rax: Rax) -> str:
    return test(rax=rax)  # type: ignore


@flytekit.task
def test(rax: Rax) -> str:
    return str(os.listdir(rax.flytedir))


def make_flytedir(path: Union[str, os.PathLike]) -> directory.FlyteDirectory:
    context = context_manager.FlyteContext.current_context()
    dimensionality = core_types.BlobType.BlobDimensionality.MULTIPART
    literal = make_literal(uri=path, dimensionality=dimensionality)
    transformer = directory_types.FlyteDirToMultipartBlobTransformer()
    out_dir = transformer.to_python_value(context, literal, directory.FlyteDirectory)
    os.listdir(out_dir)  # the dir isnt synced if we dont do this
    return out_dir


def make_literal(
    uri: Union[str, os.PathLike],
    dimensionality,
) -> literals.Literal:
    scalar = make_scalar(uri, dimensionality)
    return literals.Literal(scalar=scalar)  # type: ignore


def make_scalar(
    uri: Union[str, os.PathLike],
    dimensionality,
) -> literals.Scalar:
    blobtype = core_types.BlobType(format="", dimensionality=dimensionality)
    blob = literals.Blob(metadata=literals.BlobMetadata(type=blobtype), uri=uri)

return literals.Scalar(blob=blob) # type: ignore

Cc @thankful-minister-83577 / @glamorous-carpet-83516 we should hopefully use the streaming api

Thanks! Its fun to be here

😄 wow - ccc @cool-lifeguard-49380 / @average-finland-92144 / @melodic-magician-71351

need to see if i can open a PR for this.

@loud-oil-9899 you should also join the contributor sync we just had today. #C04NJPLRWUX channel. Please open a PR, the community will cheer you on to merge it 😄

🫡

Hey Arthur! You might want to look at https://github.com/flyteorg/flytekit/pull/1565 which I'm currently splitting into pieces. The first piece is submitted here: https://github.com/flyteorg/flytekit/pull/1615

@melodic-magician-71351 check the #C04NJPLRWUX channel there is a PR by Arthur