acoustic-carpenter-78188
04/21/2026, 9:56 AMfrom_hf(...) and transparently materializes them as datasets.Dataset or datasets.IterableDataset with optional shared caching through cache_root.
from flyteplugins.huggingface.datasets import from_hf
@env.task
async def train(
ds: datasets.Dataset = from_hf(
"stanfordnlp/imdb",
name="plain_text",
split="train",
cache_root="s3://my-bucket/flyte-hf-cache",
),
) -> int:
return len(ds)
The plugin streams converted parquet shards from the Hub, stores them as sharded parquet artifacts, and reuses cached materializations across runs when available.
You can also return and pass real datasets.Dataset objects between tasks:
@env.task
async def create_dataset() -> datasets.Dataset:
return datasets.Dataset.from_dict(
{
"text": ["hello", "world", "flyte"],
"label": [0, 1, 0],
}
)
@env.task
async def filter_positive(ds: datasets.Dataset) -> datasets.Dataset:
return ds.filter(lambda row: row["label"] == 1)
Screenshot 2026-04-24 at 3 24 30 PM
flyteorg/flyte-sdkacoustic-carpenter-78188
04/28/2026, 5:24 AM