We are looking at upgrading some key dependencies across our stack, and as part of that effort we are considering updating pandas to >=2.0. I notice flytekit currently has a ceiling pin for pandas <2.0.
My question: is this due to a genuine known incompatibility, or can we relax this constraint now that pandas 2 has been on the block for awhile now? If it's the former, is there an issue tracking the changes required to make flytekit compatible? The issue I found (https://github.com/flyteorg/flyte/issues/3928) seems to be just requesting the pin be changed.
08/30/2023, 7:16 PM
Need to double check with @Yee@Kevin Su but I don’t think there’s a fundamental compatibility issue here. However, we will need to think about the CI testing strategy: e.g. do we want to test both pandas 1.x.x (maybe the latest 1.x.x version?) and pandas 2+, or just pandas 2+.
08/30/2023, 7:49 PM
I think we can just remove pandas from setup.py, but we definitely need to install pandas to test both 1.x and 2.x in the ci.