Hello what the best way to represent a complex schema in a d

Question

Hello what the best way to represent a complex schema in a dataframe passed between tasks I think its not supported natively in `StructuredDataset` but can that be somehow represented as `FlyteSchema`

Nada Saiyed · Answer

Can a `List` or `Dict` type be specified as part of the schema

Nada Saiyed · Answer

Or for example if i have a spark dataframe with this schema can this be enforced somehow ``` schema = StructType StructField name StructType StructField firstname StringType True StructField middlename StringType True StructField lastname StringType True StructField state StringType True StructField gender StringType True ```

Niels Bantilan · Answer

are you using a pandas or spark dataframe Not sure if nested dict struct like objects are supported currently < Yee> < Kevin Su> < Eduardo Apolinario eapolinario >

Nada Saiyed · Answer

spark

Nada Saiyed · Answer

would be good if could define it in pandas as well some less complex datatypes like a `list` or a `set`

Niels Bantilan · Answer

to be clear you want to support the use case of arbitrary collection map type objects as individual values in a dataframe

Nada Saiyed · Answer

for pandas dataframes yes and for spark dataframes different `StructType` s

Samhita Alla · Answer

< Nada Saiyed> I believe you can use pyarrow schema in a StructuredDataset like Pyarrow also supports `StructType` ref

Samhita Alla · Answer

cc < Shivay Lamba>