Hello.. what the best way to represent a complex s...
# announcementsn
Nada Saiyed
08/15/2022, 7:47 PMHello.. what the best way to represent a complex schema in a dataframe passed between tasks? I think its not supported natively in
StructuredDatasetFlyteSchema👀 1
Nada Saiyed
08/15/2022, 7:50 PMCan a
ListDictNada Saiyed
08/15/2022, 7:50 PMOr, for example, if i have a spark dataframe with this schema, can this be enforced somehow?
Copy code
schema = StructType([
StructField('name', StructType([
StructField('firstname', StringType(), True),
StructField('middlename', StringType(), True),
StructField('lastname', StringType(), True)
])),
StructField('state', StringType(), True),
StructField('gender', StringType(), True)
])n
Niels Bantilan
08/15/2022, 7:53 PMare you using a pandas or spark dataframe?
Not sure if nested dict/struct-like objects are supported currently… @Yee @Kevin Su @Eduardo Apolinario (eapolinario)?
n
Nada Saiyed
08/15/2022, 7:54 PMspark
Nada Saiyed
08/15/2022, 7:56 PMwould be good if could define it in pandas as well.. some less complex datatypes like a
listsetn
Niels Bantilan
08/15/2022, 7:57 PMto be clear, you want to support the use case of arbitrary collection/map-type objects as individual values in a dataframe?
n
Nada Saiyed
08/15/2022, 8:00 PMfor pandas dataframes, yes.
and for spark dataframes different
StructTypes
Samhita Alla
08/16/2022, 5:18 AM
@Nada Saiyed, I believe you can use pyarrow schema in a StructuredDataset like https://github.com/flyteorg/flytekit/blob/771aa8a72fbc3ded437b6ff8498404767fc438db[…]it/types/structured_dataset/test_structured_dataset_workflow.py.
Pyarrow also supports
StructType👍 1
Samhita Alla
08/16/2022, 1:47 PM
cc: @Shivay Lamba
160 Views