#629 feat: Add Hugging Face Datasets plugin Pull request opened by andreahlert Summary Port of the flytekit-huggingface plugin to flyte-sdk v2, enabling native support for

datasets.Dataset

as a Flyte DataFrame type. • DataFrameEncoder/Decoder for

datasets.Dataset

with Parquet serialization • Cloud storage support (S3, GCS, Azure) via fsspec-compatible storage options • Anonymous S3 fallback for public datasets (mirrors Polars plugin pattern) • Column filtering on both encode and decode via type annotations • Auto-registration via

flyte.plugins.types

entry point Follows the same patterns as the existing Polars plugin. Usage Example import flyte import datasets env = flyte.TaskEnvironment("hf-example") @env.task async def create_dataset() -> datasets.Dataset: return datasets.Dataset.from_dict({ "text": ["hello", "world"], "label": [0, 1], }) @env.task async def process(ds: datasets.Dataset) -> int: return len(ds) Test plan • Type recognition tests (Dataset, with columns, with format annotations) • Handler registration and property tests • Encode/decode roundtrip tests • DataFrame wrapper integration test • Raw task I/O test via

flyte.TaskEnvironment

• Column subsetting on decode • Various data types (int, float, str, bool) • Empty dataset roundtrip • Storage options tests (S3, GCS, Azure, anonymous, unknown protocol) flyteorg/flyte-sdk