https://flyte.org logo
#ask-the-community
Title
# ask-the-community
a

arun tom

11/23/2023, 5:12 PM
I'm trying to run an SQLAlchemyTask locally that returns rows for a table that has the uuid type for a few columns and I get the following PyArrow error
ArrowInvalid: ("Could not convert UUID('0655f174-8763-7e9e-8000-3ee86194b973') with type UUID: did not recognize Python value
type when inferring an Arrow data type", 'Conversion failed for column id with type object')
I'm running it locally. Is there a way around this issue?
k

Ketan (kumare3)

11/23/2023, 7:11 PM
this is something to do with arrow. you could just simply write your own python task to do this for now and not use arrow?
a

arun tom

11/24/2023, 4:57 AM
I'm not sure how I can not use Arrow. I assume arrow is used to serialise task inputs and outputs? So if I create my own task and return a dataframe containing a uuid won't I run into the same issue? I guess now my questions would be how do i use something other tham arron. Do I have to configure flyte or is there something i can configure in the task itself?
k

Ketan (kumare3)

11/24/2023, 5:09 AM
No arrow is only used for retrieving data from sql database in dataframes
Arrow is only used for dataframes
But it might just be the arrow version or something
Cc @Yee can you help
Or @Samhita Alla
a

arun tom

11/24/2023, 5:50 AM
Thanks for hellping out!. This: https://github.com/apache/iceberg/issues/8248 could be a related issue with pyarrow? Is there a way to upgrade the versions of pyarrow that flyte uses or does it make sense for me to attempt the serialization manually using the newer version of pyarrow?
y

Yee

11/24/2023, 5:53 AM
i believe you can install pyarrow over flytekit
flytekit currently has an upper bound on pyarrow but we’re removing it soon
a

arun tom

11/24/2023, 7:39 AM
i believe you can install pyarrow over flyteki
I see. Thank you will try that. So when running locally and not on a cluster, just installing the latest version ought to work?
y

Yee

11/24/2023, 9:09 AM
install the latest version of pyarrow yes
14.1 i think?
f

Franco Bocci

11/29/2023, 2:08 PM
Hey! I’m facing an issue and it might be related to this. I can’t use pyarrow’s latest due to flytekit’s cap. Are there plans to tackle this? Maybe something I can help with?
y

Yee

11/29/2023, 2:46 PM
@Kevin Su can we merge this in https://github.com/flyteorg/flytekit/pull/1988
we can ship your pandas change after
apologies @Franco Bocci - let’s see what kevin and eduardo say when they get up
f

Franco Bocci

11/29/2023, 3:07 PM
No need to apologies, please 🙂 !
k

Kevin Su

11/29/2023, 6:17 PM
yes
let’s merge you PR first
merged it
f

Franco Bocci

11/30/2023, 5:53 AM
Lovely! This should impact since the next release then, correct? If so, when was it planned for? No intend to push here, just to plan from my end. And thanks once again for the quick fix
k

Kevin Su

11/30/2023, 5:54 AM
This should impact since the next release then, correct?
yes, we will release tomorrow or Friday
f

Franco Bocci

11/30/2023, 8:48 AM
Thanks for this 🙂
Hey hey! Good morning. Daily dose of being a pain in the 🥥
I saw the release. Gonna try this out. Thanks for the estimation, honestly. You said Friday and it ended up being Tuesday, really close to it. Well estimated, kudos for that