Question about passing polars dataframes between tasks: In theory the

flytekitplugin-polars

package should automatically get the return signature from the following function

def random_dataframe(num_rows: int) -> pl.DataFrame:
    feature_1_list = np.random.randint(low=100, high=999, size=(num_rows,))
    feature_2_list = np.random.normal(loc=0, scale=1, size=(num_rows, ))
    feature_3_list = [make_random_protein(x) for x in feature_1_list]
    return pl.DataFrame({'protein_length': feature_1_list,
                         'protein_feature': feature_2_list,
                         'protein_seq': feature_3_list})

and handle the conversion to

flytekit.types.structured.structured_dataset.StructuredDataset

when moving it between tasks. In practice what I see is

TypeError: Failed to convert outputs of task 'main_workflows.make_dataframe' at position 0. 
Failed to convert type <class 'polars.dataframe.frame.DataFrame'> to type <class 'polars.dataframe.frame.DataFrame'>.
Error Message: 'coroutine' object does not support the asynchronous context manager protocol.

Message:

TypeError: Failed to convert outputs of task 'main_workflows.make_dataframe' at position 0.
Failed to convert type <class 'polars.dataframe.frame.DataFrame'> to type <class 'polars.dataframe.frame.DataFrame'.
Error Message: 'coroutine' object does not support the asynchronous context manager protocol.", "taskName": null}

Any idea as to what's going on?

cc: @thankful-minister-83577 We shipped a change to add support for python`asyncio` in flytekit 1.14.0. Seeing a mention to

coroutine

there, so flagging just as a potential cause. @freezing-tailor-85994, can you confirm which versions of

flytekit

and

flytekitplugin-polars

you're using? Also, can you try running a version other than 1.14.0 ?

It's 1.14.0 on both, I can roll back to 1.13 and retest once I have some bandwidth

thank you. We'll investigate on our side as well and report back here.

@high-accountant-32689 confirming that 1.13 works fine

no we’ll patch 14. let us repro first.

@thankful-minister-83577 I am not, the function is exactly as listed above with a return of a pl.DataFrame and a type hint on the function definition

yeah thanks… we have a fix - this will get patched on monday

beautiful, thank you

https://github.com/flyteorg/flytekit/pull/2998 - will be in 14.1

@thankful-minister-83577 what's the ETA on 1.14.1 release?

need to backport prs…

thank you sir 🫡