https://flyte.org
Join Slack
Hey, everyone. Quick question: how viable would it be to bump Pyarrow to a greater version (such as ...
# flytekit
s
Hey, everyone. Quick question: how viable would it be to bump Pyarrow to a greater version (such as 7.*)? We''re developing an internal ML library with a set of tools and I just bumped into a dependency problem: some Pyarrow functions we use were introduced at 7.0.0, but Flytekit requires version 6.0.1. According to the official release, 7.0.0 biggest change is the drop in support for Python 3.6. Since Flytekit's official support is >=3.7, this shouldn't be an issue, right?
🚀 3
t
can you make the PR? but run requirements using python 3.7
s
ok, will do
A lot of dependencies changed 😅 but mostly in minor/patch levels
I am being unable to run 
make requirements
. I'm getting this error:
Copy code
Could not find a version that matches flatbuffers>=2.0,~=1.12 (from tf2onnx==1.12.0->-r <http://doc-requirements.in|doc-requirements.in> (line 38))
Tried: 1.9, 1.9, 1.10, 1.10, 1.11, 1.11, 1.12, 1.12, 2.0, 2.0, 2.0.7, 2.0.7
There are incompatible versions in the resolved dependencies:
  flatbuffers>=2.0 (from tensorflow==2.10.0->-r <http://doc-requirements.in|doc-requirements.in> (line 39))
  flatbuffers~=1.12 (from tf2onnx==1.12.0->-r <http://doc-requirements.in|doc-requirements.in> (line 38))
Even without changing anything in the code (i.e. not bumping Pyarrow), this error persists
t
can you just push changes to the other files for now?
this is only for building docs.
i saw this last week when testing.
s
oh ok
tried bumping to 9.0.0, then 8.0.0 then 7.0.0. all versions are failing with 
ImportError: cannot import name '_message' from 'google.protobuf.pyext' (/opt/hostedtoolcache/Python/3.7.13/x64/lib/python3.7/site-packages/google/protobuf/pyext/__init__.py)
, which doesn't necessarily seem like an error from Pyarrow itself. Maybe when I'm running 
make requirements
some other library is being updated and affecting the compatibility?
164 Views
Open in Slack
PreviousNext
Powered by

Flyte

Flyte enables production-grade orchestration for machine learning workflows and data processing created to accelerate local workflows to production.

Powered by