I'm trying to run an SQLAlchemyTask locally that returns rows for a table that has the uuid type for a few columns and I get the following PyArrow error

ArrowInvalid: ("Could not convert UUID('0655f174-8763-7e9e-8000-3ee86194b973') with type UUID: did not recognize Python value

type when inferring an Arrow data type", 'Conversion failed for column id with type object')

I'm running it locally. Is there a way around this issue?

this is something to do with arrow. you could just simply write your own python task to do this for now and not use arrow?

I'm not sure how I can not use Arrow. I assume arrow is used to serialise task inputs and outputs? So if I create my own task and return a dataframe containing a uuid won't I run into the same issue? I guess now my questions would be how do i use something other tham arron. Do I have to configure flyte or is there something i can configure in the task itself?

No arrow is only used for retrieving data from sql database in dataframes

Thanks for hellping out!. This: https://github.com/apache/iceberg/issues/8248 could be a related issue with pyarrow? Is there a way to upgrade the versions of pyarrow that flyte uses or does it make sense for me to attempt the serialization manually using the newer version of pyarrow?

i believe you can install pyarrow over flytekit

i believe you can install pyarrow over flyteki

I see. Thank you will try that. So when running locally and not on a cluster, just installing the latest version ought to work?

install the latest version of pyarrow yes

Hey! I’m facing an issue and it might be related to this. I can’t use pyarrow’s latest due to flytekit’s cap. Are there plans to tackle this? Maybe something I can help with?

@glamorous-carpet-83516 can we merge this in https://github.com/flyteorg/flytekit/pull/1988

No need to apologies, please 🙂 !

yes

Lovely! This should impact since the next release then, correct? If so, when was it planned for? No intend to push here, just to plan from my end. And thanks once again for the quick fix

This should impact since the next release then, correct?

yes, we will release tomorrow or Friday

Thanks for this 🙂