Hi all, quick question: what is the correct way to pass a raw file or a pandas.DataFrame to a task a...

happy-sunset-85283

04/23/2023, 8:56 PM

Hi all, quick question: what is the correct way to pass a raw file or a pandas.DataFrame to a task as input using

FlyteRemote.execute

happy-sunset-85283

04/23/2023, 8:57 PM

I was trying

Copy code

remote.execute(mytask, {"df": df})

but I get

ValueError: Failed to find a handler for , protocol gcs, fmt ||

(my FlyteRemote is configured with data_upload_location pointing to gcs)

freezing-airport-6809

04/23/2023, 11:17 PM

It should work once you configure gcs correctly

freezing-airport-6809

04/23/2023, 11:24 PM

Ohh what version of flytekit is this

freezing-airport-6809

04/23/2023, 11:25 PM

It seems there is not gcs persistence plugin available

happy-sunset-85283

04/24/2023, 7:48 AM

I have flytekit 1.5.0 (tried with 1.4.2 too, same result). How do I verify the gcs plugin? I see that gcsfs is installed and I can

fsspec.open()

files with the same prefix as I set as data_upload_location.

happy-sunset-85283

04/24/2023, 7:51 AM

I can also download the results of executions (

.outputs.get('o0').open(pd.DataFrame)

), so the gcs libs and permissions seem to be ok.

happy-sunset-85283

04/25/2023, 12:07 PM

oh, I figured out what was missing:

.execute(task, {"df": StructuredDataset(dataframe=df)})

works

freezing-airport-6809

04/25/2023, 1:20 PM

Hmm, that should not be needed, maybe file a bug

160 Views

Flyte

Flyte enables production-grade orchestration for machine learning workflows and data processing created to accelerate local workflows to production.