#3116 [wip] Pandas Dataframe in Dataclass
Pull request opened by
Future-Outlier
## Tracking issue
## Why are the changes needed?
we want to improve UX.
import flytekit
from flytekit import task, workflow
from dataclasses import dataclass, field
import pandas as pd
# from pydantic import BaseModel
from flytekit.types.structured.structured_dataset import (
PARQUET,
StructuredDataset,
StructuredDatasetDecoder,
StructuredDatasetEncoder,
StructuredDatasetTransformerEngine,
)
@dataclass
class smallDC:
df: pd.DataFrame = field(default_factory=lambda: pd.DataFrame([[1, 2, 3], [4, 5, 6]]))
@dataclass
class DC:
dc: smallDC
df: pd.DataFrame = field(default_factory=lambda: pd.DataFrame([[1, 2, 3], [4, 5, 6]]))
@task
def t1() -> DC:
return DC(dc=smallDC())
if
name == "__main__":
print(t1())
## What changes were proposed in this pull request?
## How was this patch tested?
### Setup process
### Screenshots
## Check all the applicable boxes
• I updated the documentation accordingly.
• All new and existing tests passed.
• All commits are signed-off.
## Related PRs
## Docs link
### Summary by Bito
This PR enhances Flytekit's functionality in multiple areas: 1) Improves the DataclassTransformer with automatic Pandas DataFrame to StructuredDataset conversion and better type handling, 2) Enhances error handling and validation for msgpack data, authentication, and secret mount management, 3) Implements recursive transformation capabilities and improved JSON decoding, 4) Introduces configurable caching policies and strengthens type validation features.
Unit tests added: True
Estimated effort to review (1-5, lower is better): 5
flyteorg/flytekit