<#3116 [wip] Pandas Dataframe in Dataclass> Pull r...
# flyte-github
c
#3116 [wip] Pandas Dataframe in Dataclass Pull request opened by Future-Outlier ## Tracking issue ## Why are the changes needed? we want to improve UX. import flytekit from flytekit import task, workflow from dataclasses import dataclass, field import pandas as pd # from pydantic import BaseModel from flytekit.types.structured.structured_dataset import ( PARQUET, StructuredDataset, StructuredDatasetDecoder, StructuredDatasetEncoder, StructuredDatasetTransformerEngine, ) @dataclass class smallDC: df: pd.DataFrame = field(default_factory=lambda: pd.DataFrame([[1, 2, 3], [4, 5, 6]])) @dataclass class DC: dc: smallDC df: pd.DataFrame = field(default_factory=lambda: pd.DataFrame([[1, 2, 3], [4, 5, 6]])) @task def t1() -> DC: return DC(dc=smallDC()) if name == "__main__": print(t1()) ## What changes were proposed in this pull request? ## How was this patch tested? ### Setup process ### Screenshots ## Check all the applicable boxes • I updated the documentation accordingly. • All new and existing tests passed. • All commits are signed-off. ## Related PRs ## Docs link ### Summary by Bito This PR enhances Flytekit's functionality in multiple areas: 1) Improves the DataclassTransformer with automatic Pandas DataFrame to StructuredDataset conversion and better type handling, 2) Enhances error handling and validation for msgpack data, authentication, and secret mount management, 3) Implements recursive transformation capabilities and improved JSON decoding, 4) Introduces configurable caching policies and strengthens type validation features. Unit tests added: True Estimated effort to review (1-5, lower is better): 5 flyteorg/flytekit