Hello.. what the best way to represent a complex schema in a dataframe passed between tasks? I think...

plain-carpenter-67621

08/15/2022, 7:47 PM

Hello.. what the best way to represent a complex schema in a dataframe passed between tasks? I think its not supported natively in

StructuredDataset

, but can that be somehow represented as

FlyteSchema

👀 1

plain-carpenter-67621

08/15/2022, 7:50 PM

Can a

List

Dict

type be specified as part of the schema?

plain-carpenter-67621

08/15/2022, 7:50 PM

Or, for example, if i have a spark dataframe with this schema, can this be enforced somehow?

Copy code

schema = StructType([
        StructField('name', StructType([
            StructField('firstname', StringType(), True),
            StructField('middlename', StringType(), True),
            StructField('lastname', StringType(), True)
            ])),
        StructField('state', StringType(), True),
        StructField('gender', StringType(), True)
        ])

broad-monitor-993

08/15/2022, 7:53 PM

are you using a pandas or spark dataframe? Not sure if nested dict/struct-like objects are supported currently… @thankful-minister-83577 @glamorous-carpet-83516 @high-accountant-32689?

plain-carpenter-67621

08/15/2022, 7:54 PM

spark

plain-carpenter-67621

08/15/2022, 7:56 PM

would be good if could define it in pandas as well.. some less complex datatypes like a

list

or a

set

broad-monitor-993

08/15/2022, 7:57 PM

to be clear, you want to support the use case of arbitrary collection/map-type objects as individual values in a dataframe?

plain-carpenter-67621

08/15/2022, 8:00 PM

for pandas dataframes, yes. and for spark dataframes different

StructType

tall-lock-23197

08/16/2022, 5:18 AM

@plain-carpenter-67621, I believe you can use pyarrow schema in a StructuredDataset like https://github.com/flyteorg/flytekit/blob/771aa8a72fbc3ded437b6ff8498404767fc438db[…]it/types/structured_dataset/test_structured_dataset_workflow.py. Pyarrow also supports

StructType

(ref: https://arrow.apache.org/docs/python/generated/pyarrow.StructType.html)

👍 1

tall-lock-23197

08/16/2022, 1:47 PM

cc: @victorious-park-53030

161 Views

Open in Slack

Previous Next

Flyte

Flyte enables production-grade orchestration for machine learning workflows and data processing created to accelerate local workflows to production.