Hello, Can Polars dataframe be used as flyte task ...
# ask-the-community
f
Hello, Can Polars dataframe be used as flyte task input & output params out of the box? Since which version of flyte it’s supported. I am using flyte 1.2.4.
e
Have you tried installing
flytekitplugins-polars
? It has a type transformer for polars dataframes!
Let me know if that isn't working for you
Here is a code snippet 🙂
Copy code
# pip install flytekitplugins-polars

from flytekit import task, workflow
import polars as pl
import pandas as pd

@task
def t1() -> pl.DataFrame:
    return pl.DataFrame(pd.DataFrame({"a": [1, 2, 3]}))

@task
def t2(df: pl.DataFrame) -> int:
    return len(df)

@workflow
def wf() -> int:
    df = t1()
    return t2(df=df)

wf()
f
@Evan Sadler, thanks. I will try it out. Looks like there is not much offered in flytekitplugins-polars yet, Therefore I will not install flytekitplugins-polars.
e
The plugin only adds type transformers so you can use them for inputs and outputs by themselves or as StructuredDatasets @Frank Shen. I think having it support lazy execution would be a fantastic addition, but please let me know if you have other feature ideas for the plugin!
170 Views