https://flyte.org
Join Slack
Hello, Can Polars dataframe be used as flyte task input & output params out of the box? Since wh...
# flyte-support
s
Hello, Can Polars dataframe be used as flyte task input & output params out of the box? Since which version of flyte it’s supported. I am using flyte 1.2.4.
b
Have you tried installing 
flytekitplugins-polars
? It has a type transformer for polars dataframes!
Let me know if that isn't working for you
Here is a code snippet 🙂
Copy code
# pip install flytekitplugins-polars

from flytekit import task, workflow
import polars as pl
import pandas as pd

@task
def t1() -> pl.DataFrame:
    return pl.DataFrame(pd.DataFrame({"a": [1, 2, 3]}))

@task
def t2(df: pl.DataFrame) -> int:
    return len(df)

@workflow
def wf() -> int:
    df = t1()
    return t2(df=df)

wf()
s
@abundant-hamburger-66584, thanks. I will try it out. Looks like there is not much offered in flytekitplugins-polars yet, Therefore I will not install flytekitplugins-polars.
b
The plugin only adds type transformers so you can use them for inputs and outputs by themselves or as StructuredDatasets @salmon-refrigerator-32115. I think having it support lazy execution would be a fantastic addition, but please let me know if you have other feature ideas for the plugin!
223 Views
Open in Slack
PreviousNext
Powered by

Flyte

Flyte enables production-grade orchestration for machine learning workflows and data processing created to accelerate local workflows to production.

Powered by