Flyte enables production-grade orchestration for machine learning workflows and data processing created to accelerate local workflows to production.

Flyte

question about nested/repeated fields for structured datasets.  They don't appear to be supported. is this a backend limitation or just one that exists in flytekit?

It can be anything in the back <https://docs.flyte.org/en/latest/protos/docs/core/core.html#ref-flyteidl-core-structureddatasettype-datasetcolumn|https://docs.flyte.org/en/latest/protos/docs/core/core.html#ref-flyteidl-core-structureddatasettype-datasetcolumn> what limitation do you see

might just be a flytekit side, do you have a code example of defining this.  trying using named tuples, kwtypes etc for nested to no avail

or should it be another structured dataset?

hang on let me share some code :slightly_smiling_face:

none of the following work with flytekit
```# Levels = Annotated[StructuredDataset, kwtypes(level1=str, level2=str)] # TypeError: unhashable type: 'collections.OrderedDict'
# Levels = NamedTuple( # AssertionError: type <class 'onemodel.models.levels'> is currently not supported by StructuredDataset
#     "levels",
#     level1=str,
#     level2=str
# )
Levels = kwtypes(  # TypeError: unhashable type: 'collections.OrderedDict'
    level1=str,
    level2=str
)
Schema = Annotated[StructuredDataset, kwtypes(age=int, levels=Levels)]

@task
def mytask() -> Schema:
    return pd.DataFrame({
        "age": [1],
        "levels": [
            {"level1": "1", "level2": "2"}
        ]
    })```

currently, we provide support for <https://github.com/flyteorg/flytekit/blob/95ea92dbbe11aea202e1044b6e8a45e7a67ae687/flytekit/types/structured/structured_dataset.py#L300|simple types>, but it should be possible to support nested types. cc <@USU6W5ATA>

it’s a known issue. it’s doable, just no one implements it yet. more and more dataframe/database support nested type now, so I think it’s make sense to add that.

seems straightforward. what's the best way to get this rolling? open an issue?

also is this just a root xformer fix or would we need to update each coder?

<@U01E1L78CLA> I will escalate this issue. <https://github.com/flyteorg/flyte/issues/4241> you already created one before.

which structured dataset decoder you are using? `pandasToParquetDecoder`? Let us know, we can work on that first.