<#992 add HF dataset plugin> Pull request opened b...
# flyte-github
a
#992 add HF dataset plugin Pull request opened by samhita-alla This PR adds a Hugging Face datasets plugin for Flyte. It lets tasks consume Hub datasets directly via
from_hf(...)
and transparently materializes them as
datasets.Dataset
or
datasets.IterableDataset
with optional shared caching through
cache_root
. from flyteplugins.huggingface.datasets import from_hf @env.task async def train( ds: datasets.Dataset = from_hf( "stanfordnlp/imdb", name="plain_text", split="train", cache_root="s3://my-bucket/flyte-hf-cache", ), ) -> int: return len(ds) The plugin streams converted parquet shards from the Hub, stores them as sharded parquet artifacts, and reuses cached materializations across runs when available. You can also return and pass real
datasets.Dataset
objects between tasks: @env.task async def create_dataset() -> datasets.Dataset: return datasets.Dataset.from_dict( { "text": ["hello", "world", "flyte"], "label": [0, 1, 0], } ) @env.task async def filter_positive(ds: datasets.Dataset) -> datasets.Dataset: return ds.filter(lambda row: row["label"] == 1) Screenshot 2026-04-24 at 3 24 30 PM flyteorg/flyte-sdk