https://flyte.org logo
#flytekit
Title
# flytekit
d

Dylan Wilder

02/22/2024, 1:50 PM
question about nested/repeated fields for structured datasets. They don't appear to be supported. is this a backend limitation or just one that exists in flytekit?
k

Ketan (kumare3)

02/22/2024, 3:38 PM
d

Dylan Wilder

02/22/2024, 3:52 PM
might just be a flytekit side, do you have a code example of defining this. trying using named tuples, kwtypes etc for nested to no avail
or should it be another structured dataset?
hang on let me share some code 🙂
none of the following work with flytekit
Copy code
# Levels = Annotated[StructuredDataset, kwtypes(level1=str, level2=str)] # TypeError: unhashable type: 'collections.OrderedDict'
# Levels = NamedTuple( # AssertionError: type <class 'onemodel.models.levels'> is currently not supported by StructuredDataset
#     "levels",
#     level1=str,
#     level2=str
# )
Levels = kwtypes(  # TypeError: unhashable type: 'collections.OrderedDict'
    level1=str,
    level2=str
)
Schema = Annotated[StructuredDataset, kwtypes(age=int, levels=Levels)]

@task
def mytask() -> Schema:
    return pd.DataFrame({
        "age": [1],
        "levels": [
            {"level1": "1", "level2": "2"}
        ]
    })
s

Samhita Alla

02/23/2024, 6:26 AM
currently, we provide support for simple types, but it should be possible to support nested types. cc @Kevin Su
k

Kevin Su

02/23/2024, 7:44 AM
it’s a known issue. it’s doable, just no one implements it yet. more and more dataframe/database support nested type now, so I think it’s make sense to add that.
d

Dylan Wilder

02/27/2024, 6:37 PM
seems straightforward. what's the best way to get this rolling? open an issue?
also is this just a root xformer fix or would we need to update each coder?
@Kevin Su thoughts on the above?
k

Kevin Su

03/04/2024, 7:49 AM
@Dylan Wilder I will escalate this issue. https://github.com/flyteorg/flyte/issues/4241 you already created one before.
which structured dataset decoder you are using?
pandasToParquetDecoder
? Let us know, we can work on that first.
d

Dylan Wilder

03/04/2024, 1:42 PM
pandas bq 🙂
k

Kevin Su

03/04/2024, 5:18 PM
got it, we will work on that first.
d

Dylan Wilder

03/04/2024, 5:31 PM
Thanks!!
k

Kevin Su

03/04/2024, 5:34 PM
cc @Austin Liu (Austinnn) ^^^
a

Austin Liu (Austinnn)

03/04/2024, 5:56 PM
@Kevin Su Got it!