Hi Community! Has anyone run into issues with pand...
# ask-the-communitya
Anooj Patel
11/06/2023, 7:07 PMHi Community! Has anyone run into issues with pandas DataFrames when working with them as task output types? I generally have issues running them locally on flytekit 1.10
MWE:
Copy code
import pandas as pd
from flytekit import workflow, task
@task
def gen_df()-> pd.DataFrame:
df = pd.DataFrame({"a":[1], "b":[2]})
return df
@workflow()
def wf() -> pd.DataFrame:
df = gen_df()
return dfAnooj Patel
11/06/2023, 7:08 PM Copy code
│ │
│ ❱ 960 │ │ │ │ sink = self.file_handle = filesystem.open_output_stream( │
│ │
│ in pyarrow._fs.FileSystem.open_output_stream:868 │
│ │
│ in pyarrow.lib.pyarrow_internal_check_status:144 │
│ │
│ in pyarrow.lib.check_status:113 │
╰──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
FileNotFoundError: [Errno 2] Failed to open local file '/tmp/flyte-o_211y4n/raw/7657bf429e60c00a271217d651979014/00000'. Detail: [errno 2] No such file or directoryk
Kevin Su
11/06/2023, 7:09 PMgot the same error, looking
k
Ketan (kumare3)
11/06/2023, 7:12 PM
is this the
0000k
Kevin Su
11/06/2023, 7:42 PMIt seems to have been fixed in the master branch
s
Samhita Alla
11/07/2023, 8:56 AM
It likely relates to fsspec. I suggest downgrading your fsspec to < 2023.10.0.
https://github.com/flyteorg/flyte/issues/4326
4 Views