Hello.. what the best way to represent a complex s...
# announcementsp
plain-carpenter-67621
08/15/2022, 7:47 PMHello.. what the best way to represent a complex schema in a dataframe passed between tasks? I think its not supported natively in
StructuredDatasetFlyteSchema👀 1
plain-carpenter-67621
08/15/2022, 7:50 PMCan a
ListDictplain-carpenter-67621
08/15/2022, 7:50 PMOr, for example, if i have a spark dataframe with this schema, can this be enforced somehow?
Copy code
schema = StructType([
StructField('name', StructType([
StructField('firstname', StringType(), True),
StructField('middlename', StringType(), True),
StructField('lastname', StringType(), True)
])),
StructField('state', StringType(), True),
StructField('gender', StringType(), True)
])b
broad-monitor-993
08/15/2022, 7:53 PMare you using a pandas or spark dataframe?
Not sure if nested dict/struct-like objects are supported currently… @thankful-minister-83577 @glamorous-carpet-83516 @high-accountant-32689?
p
plain-carpenter-67621
08/15/2022, 7:54 PMspark
plain-carpenter-67621
08/15/2022, 7:56 PMwould be good if could define it in pandas as well.. some less complex datatypes like a
listsetb
broad-monitor-993
08/15/2022, 7:57 PMto be clear, you want to support the use case of arbitrary collection/map-type objects as individual values in a dataframe?
p
plain-carpenter-67621
08/15/2022, 8:00 PMfor pandas dataframes, yes.
and for spark dataframes different
StructTypet
tall-lock-23197
08/16/2022, 5:18 AM
@plain-carpenter-67621, I believe you can use pyarrow schema in a StructuredDataset like https://github.com/flyteorg/flytekit/blob/771aa8a72fbc3ded437b6ff8498404767fc438db[…]it/types/structured_dataset/test_structured_dataset_workflow.py.
Pyarrow also supports
StructType👍 1
tall-lock-23197
08/16/2022, 1:47 PM
cc: @victorious-park-53030
162 Views