https://flyte.org logo
Join Slack
Powered by
I'm facing deps issues when trying to `make dev-re...
# flytekit
c
I'm facing deps issues when trying to 
make dev-requirements.txt
I've 
grpcio-status<1.49.0
added to the 
<http://dev-requirements.in|dev-requirements.in>
Copy code
Could not find a version that matches protobuf!=3.20.0,!=3.20.1,!=4.21.0,!=4.21.1,!=4.21.2,!=4.21.3,!=4.21.4,!=4.21.5,<3.20,<5.0.0dev,==3.20.3,>=3.19.5,>=3.9.2 (from -c requirements.txt (line 121))
Tried: 2.0.3, 2.3.0, 2.4.1, 2.5.0, 2.6.0, 2.6.1, 3.0.0, 3.0.0, 3.1.0, 3.1.0.post1, 3.1.0.post1, 3.2.0, 3.2.0, 3.3.0, 3.4.0, 3.4.0, 3.5.0.post1, 3.5.0.post1, 3.5.1, 3.5.1, 3.5.2, 3.5.2, 3.5.2.post1, 3.5.2.post1, 3.6.0, 3.6.0, 3.6.1, 3.6.1, 3.7.0, 3.7.0, 3.7.1, 3.7.1, 3.8.0, 3.8.0, 3.9.0, 3.9.0, 3.9.1, 3.9.1, 3.9.2, 3.9.2, 3.10.0, 3.10.0, 3.11.0, 3.11.0, 3.11.1, 3.11.1, 3.11.2, 3.11.2, 3.11.2, 3.11.3, 3.11.3, 3.11.3, 3.12.0, 3.12.1, 3.12.2, 3.12.2, 3.12.2, 3.12.4, 3.12.4, 3.12.4, 3.13.0, 3.13.0, 3.13.0, 3.14.0, 3.14.0, 3.14.0, 3.15.0, 3.15.0, 3.15.0, 3.15.1, 3.15.1, 3.15.1, 3.15.2, 3.15.2, 3.15.2, 3.15.3, 3.15.3, 3.15.3, 3.15.4, 3.15.4, 3.15.4, 3.15.5, 3.15.5, 3.15.5, 3.15.6, 3.15.6, 3.15.6, 3.15.7, 3.15.7, 3.15.7, 3.15.8, 3.15.8, 3.15.8, 3.16.0, 3.16.0, 3.16.0, 3.17.0, 3.17.0, 3.17.0, 3.17.1, 3.17.1, 3.17.1, 3.17.2, 3.17.2, 3.17.2, 3.17.3, 3.17.3, 3.17.3, 3.18.0, 3.18.0, 3.18.0, 3.18.1, 3.18.1, 3.18.1, 3.18.3, 3.18.3, 3.18.3, 3.19.0, 3.19.0, 3.19.0, 3.19.1, 3.19.1, 3.19.1, 3.19.2, 3.19.2, 3.19.2, 3.19.3, 3.19.3, 3.19.3, 3.19.4, 3.19.4, 3.19.4, 3.19.5, 3.19.5, 3.19.5, 3.19.6, 3.19.6, 3.19.6, 3.20.0, 3.20.0, 3.20.0, 3.20.1, 3.20.1, 3.20.1, 3.20.2, 3.20.2, 3.20.2, 3.20.3, 3.20.3, 3.20.3, 4.21.0, 4.21.0, 4.21.0, 4.21.0, 4.21.1, 4.21.1, 4.21.1, 4.21.1, 4.21.2, 4.21.2, 4.21.2, 4.21.2, 4.21.3, 4.21.3, 4.21.3, 4.21.3, 4.21.4, 4.21.4, 4.21.4, 4.21.4, 4.21.5, 4.21.5, 4.21.5, 4.21.5, 4.21.6, 4.21.6, 4.21.6, 4.21.6, 4.21.7, 4.21.7, 4.21.7, 4.21.7, 4.21.8, 4.21.8, 4.21.8, 4.21.8
Skipped pre-versions: 2.0.0b0, 3.0.0a2, 3.0.0a3, 3.0.0b1, 3.0.0b1.post1, 3.0.0b1.post2, 3.0.0b2, 3.0.0b2, 3.0.0b2.post1, 3.0.0b2.post1, 3.0.0b2.post2, 3.0.0b2.post2, 3.0.0b3, 3.0.0b4, 3.0.0b4, 3.2.0rc1, 3.2.0rc1, 3.2.0rc1.post1, 3.2.0rc1.post1, 3.2.0rc2, 3.2.0rc2, 3.7.0rc2, 3.7.0rc2, 3.7.0rc3, 3.7.0rc3, 3.8.0rc1, 3.8.0rc1, 3.9.0rc1, 3.9.0rc1, 3.10.0rc1, 3.10.0rc1, 3.11.0rc1, 3.11.0rc1, 3.11.0rc2, 3.11.0rc2, 3.12.0rc1, 3.12.0rc2, 3.13.0rc3, 3.13.0rc3, 3.13.0rc3, 3.14.0rc1, 3.14.0rc1, 3.14.0rc1, 3.14.0rc2, 3.14.0rc2, 3.14.0rc2, 3.14.0rc3, 3.14.0rc3, 3.14.0rc3, 3.15.0rc1, 3.15.0rc1, 3.15.0rc1, 3.15.0rc2, 3.15.0rc2, 3.15.0rc2, 3.16.0rc1, 3.16.0rc1, 3.16.0rc1, 3.16.0rc2, 3.16.0rc2, 3.16.0rc2, 3.17.0rc1, 3.17.0rc1, 3.17.0rc1, 3.17.0rc2, 3.17.0rc2, 3.17.0rc2, 3.18.0rc1, 3.18.0rc1, 3.18.0rc1, 3.18.0rc2, 3.18.0rc2, 3.18.0rc2, 3.19.0rc1, 3.19.0rc1, 3.19.0rc1, 3.19.0rc2, 3.19.0rc2, 3.19.0rc2, 3.20.0rc1, 3.20.0rc1, 3.20.0rc1, 3.20.0rc2, 3.20.0rc2, 3.20.0rc2, 3.20.1rc1, 3.20.1rc1, 3.20.1rc1, 4.0.0rc1, 4.0.0rc1, 4.0.0rc1, 4.0.0rc2, 4.0.0rc2, 4.0.0rc2, 4.21.0rc1, 4.21.0rc1, 4.21.0rc1, 4.21.0rc1, 4.21.0rc2, 4.21.0rc2, 4.21.0rc2, 4.21.0rc2
There are incompatible versions in the resolved dependencies:
  protobuf==3.20.3 (from -c requirements.txt (line 121))
  protobuf<3.20,>=3.9.2 (from tensorflow==2.10.0->-r <http://dev-requirements.in|dev-requirements.in> (line 16))
  protobuf!=3.20.0,!=3.20.1,!=4.21.0,!=4.21.1,!=4.21.2,!=4.21.3,!=4.21.4,!=4.21.5,<5.0.0dev,>=3.19.5 (from google-cloud-bigquery-storage==2.16.2->-r <http://dev-requirements.in|dev-requirements.in> (line 13))
  protobuf!=3.20.0,!=3.20.1,!=4.21.0,!=4.21.1,!=4.21.2,!=4.21.3,!=4.21.4,!=4.21.5,<5.0.0dev,>=3.19.5 (from google-cloud-bigquery==3.3.5->-r <http://dev-requirements.in|dev-requirements.in> (line 12))
make: *** [Makefile:66: dev-requirements.txt] Error 2
I'm using a slightly older clone of the repo which was the one before @Dennis O'Brien’s fix
y
cc @Eduardo Apolinario (eapolinario)
can you add
Copy code
grpcio<=1.47.0
grpcio-status<=1.47.0
we really need to release this pin
c
I've gone through that PR and tried doing that too, but I still got a make error
Copy code
Could not find a version that matches protobuf!=3.20.0,!=3.20.1,!=4.21.0,!=4.21.1,!=4.21.2,!=4.21.3,!=4.21.4,!=4.21.5,<3.20,<5.0.0dev,==3.20.3,>=3.19.5,>=3.9.2 (from -c requirements.txt (line 122))
Tried: 2.0.3, 2.3.0, 2.4.1, 2.5.0, 2.6.0, 2.6.1, 3.0.0, 3.0.0, 3.1.0, 3.1.0.post1, 3.1.0.post1, 3.2.0, 3.2.0, 3.3.0, 3.4.0, 3.4.0, 3.5.0.post1, 3.5.0.post1, 3.5.1, 3.5.1, 3.5.2, 3.5.2, 3.5.2.post1, 3.5.2.post1, 3.6.0, 3.6.0, 3.6.1, 3.6.1, 3.7.0, 3.7.0, 3.7.1, 3.7.1, 3.8.0, 3.8.0, 3.9.0, 3.9.0, 3.9.1, 3.9.1, 3.9.2, 3.9.2, 3.10.0, 3.10.0, 3.11.0, 3.11.0, 3.11.1, 3.11.1, 3.11.2, 3.11.2, 3.11.2, 3.11.3, 3.11.3, 3.11.3, 3.12.0, 3.12.1, 3.12.2, 3.12.2, 3.12.2, 3.12.4, 3.12.4, 3.12.4, 3.13.0, 3.13.0, 3.13.0, 3.14.0, 3.14.0, 3.14.0, 3.15.0, 3.15.0, 3.15.0, 3.15.1, 3.15.1, 3.15.1, 3.15.2, 3.15.2, 3.15.2, 3.15.3, 3.15.3, 3.15.3, 3.15.4, 3.15.4, 3.15.4, 3.15.5, 3.15.5, 3.15.5, 3.15.6, 3.15.6, 3.15.6, 3.15.7, 3.15.7, 3.15.7, 3.15.8, 3.15.8, 3.15.8, 3.16.0, 3.16.0, 3.16.0, 3.17.0, 3.17.0, 3.17.0, 3.17.1, 3.17.1, 3.17.1, 3.17.2, 3.17.2, 3.17.2, 3.17.3, 3.17.3, 3.17.3, 3.18.0, 3.18.0, 3.18.0, 3.18.1, 3.18.1, 3.18.1, 3.18.3, 3.18.3, 3.18.3, 3.19.0, 3.19.0, 3.19.0, 3.19.1, 3.19.1, 3.19.1, 3.19.2, 3.19.2, 3.19.2, 3.19.3, 3.19.3, 3.19.3, 3.19.4, 3.19.4, 3.19.4, 3.19.5, 3.19.5, 3.19.5, 3.19.6, 3.19.6, 3.19.6, 3.20.0, 3.20.0, 3.20.0, 3.20.1, 3.20.1, 3.20.1, 3.20.2, 3.20.2, 3.20.2, 3.20.3, 3.20.3, 3.20.3, 4.21.0, 4.21.0, 4.21.0, 4.21.0, 4.21.1, 4.21.1, 4.21.1, 4.21.1, 4.21.2, 4.21.2, 4.21.2, 4.21.2, 4.21.3, 4.21.3, 4.21.3, 4.21.3, 4.21.4, 4.21.4, 4.21.4, 4.21.4, 4.21.5, 4.21.5, 4.21.5, 4.21.5, 4.21.6, 4.21.6, 4.21.6, 4.21.6, 4.21.7, 4.21.7, 4.21.7, 4.21.7, 4.21.8, 4.21.8, 4.21.8, 4.21.8
Skipped pre-versions: 2.0.0b0, 3.0.0a2, 3.0.0a3, 3.0.0b1, 3.0.0b1.post1, 3.0.0b1.post2, 3.0.0b2, 3.0.0b2, 3.0.0b2.post1, 3.0.0b2.post1, 3.0.0b2.post2, 3.0.0b2.post2, 3.0.0b3, 3.0.0b4, 3.0.0b4, 3.2.0rc1, 3.2.0rc1, 3.2.0rc1.post1, 3.2.0rc1.post1, 3.2.0rc2, 3.2.0rc2, 3.7.0rc2, 3.7.0rc2, 3.7.0rc3, 3.7.0rc3, 3.8.0rc1, 3.8.0rc1, 3.9.0rc1, 3.9.0rc1, 3.10.0rc1, 3.10.0rc1, 3.11.0rc1, 3.11.0rc1, 3.11.0rc2, 3.11.0rc2, 3.12.0rc1, 3.12.0rc2, 3.13.0rc3, 3.13.0rc3, 3.13.0rc3, 3.14.0rc1, 3.14.0rc1, 3.14.0rc1, 3.14.0rc2, 3.14.0rc2, 3.14.0rc2, 3.14.0rc3, 3.14.0rc3, 3.14.0rc3, 3.15.0rc1, 3.15.0rc1, 3.15.0rc1, 3.15.0rc2, 3.15.0rc2, 3.15.0rc2, 3.16.0rc1, 3.16.0rc1, 3.16.0rc1, 3.16.0rc2, 3.16.0rc2, 3.16.0rc2, 3.17.0rc1, 3.17.0rc1, 3.17.0rc1, 3.17.0rc2, 3.17.0rc2, 3.17.0rc2, 3.18.0rc1, 3.18.0rc1, 3.18.0rc1, 3.18.0rc2, 3.18.0rc2, 3.18.0rc2, 3.19.0rc1, 3.19.0rc1, 3.19.0rc1, 3.19.0rc2, 3.19.0rc2, 3.19.0rc2, 3.20.0rc1, 3.20.0rc1, 3.20.0rc1, 3.20.0rc2, 3.20.0rc2, 3.20.0rc2, 3.20.1rc1, 3.20.1rc1, 3.20.1rc1, 4.0.0rc1, 4.0.0rc1, 4.0.0rc1, 4.0.0rc2, 4.0.0rc2, 4.0.0rc2, 4.21.0rc1, 4.21.0rc1, 4.21.0rc1, 4.21.0rc1, 4.21.0rc2, 4.21.0rc2, 4.21.0rc2, 4.21.0rc2
There are incompatible versions in the resolved dependencies:
  protobuf==3.20.3 (from -c requirements.txt (line 122))
  protobuf<3.20,>=3.9.2 (from tensorflow==2.10.0->-r <http://dev-requirements.in|dev-requirements.in> (line 16))
  protobuf!=3.20.0,!=3.20.1,!=4.21.0,!=4.21.1,!=4.21.2,!=4.21.3,!=4.21.4,!=4.21.5,<5.0.0dev,>=3.19.5 (from google-cloud-bigquery==3.3.5->-r <http://dev-requirements.in|dev-requirements.in> (line 12))
  protobuf!=3.20.0,!=3.20.1,!=4.21.0,!=4.21.1,!=4.21.2,!=4.21.3,!=4.21.4,!=4.21.5,<5.0.0dev,>=3.19.5 (from google-cloud-bigquery-storage==2.16.2->-r <http://dev-requirements.in|dev-requirements.in> (line 13))
this is what I got after adding 1.47.0 @Yee which is pretty much the same
a
The error you're seeing now links the issue to tensorflow
```dependencies:
protobuf==3.20.3 (from -c requirements.txt (line 122))
protobuf<3.20,>=3.9.2 (from tensorflow==2.10.0->-r dev-requirements.in (line 16))```
https://github.com/tensorflow/tensorflow/blob/v2.10.0/tensorflow/tools/pip_package/setup.py#L96 tf 2.10.0 requires protobuf < 3.20 but your requirements.txt seems like it explicitly wants 3.20.3 try 3.19.6?
c
@Andrew Achkar Thank you, you were right, i'd to set the version inside 
<http://requirements.in|requirements.in>
.
is it worth opening an issue on this? @Yee
that protobuf versioning is odd
@Eduardo Apolinario (eapolinario) Would it be the right solution to pin the version of protobuf?
e
Are you changing the tensorflow version?
d
I see that @Ryan Nazareth has worked on a few PRs related to tensorflow, but I didn't notice any version pinning. @cryptic Are you including tensorflow with a version constraint in 
<http://dev-requirements.in|dev-requirements.in>
or 
<http://requirements.in|requirements.in>
?
r
@cryptic Actually i’ve made a comment regarding this https://github.com/flyteorg/flytekit/pull/1240#discussion_r1002499826 - this may or may not be due to 
grpcio-status
version pinned but when Ive pinned 
grpcio-status<1.49.0
in 
<http://dev-requirements.in|dev-requirements.in>
as @Kevin Su suggested here https://github.com/flyteorg/flytekit/pull/1241#issuecomment-1283098765 and it seemed to work for me and bypass that 
protobuf
conflict (before syncing with master). However, 
grpcio-status
is also pinned in 
<http://requirements.in|requirements.in>
master as @Dennis O'Brien mentioned with his PR, but pinned to 
grpcio-status<=1.47.0
. When i tried to 
make <http://dev-requirements.in|dev-requirements.in>
after syncing with master - i get the same error, when installing tensorflow. Also reported in another PR , although the author of that PR seemed to be experiencing the same issue even with pinning 
grpcio-status<1.49.0
, which worked in my case, so not sure https://github.com/flyteorg/flytekit/pull/1243 i haven’t tried pinning the version of protobuf though.
c
@Eduardo Apolinario (eapolinario) Sorry for the late response, but I never tried changing the version of tf as i never felt the need to. It picks up the latest 2.10 when i make 
<http://dev-requirements.in|dev-requirements.in>
by default
@Dennis O'Brien Yes i've made the same observation as you did. But in other people's PRs, especially from @Ryan Nazareth's PR i've seen the default version of TensorFlow installed was 2.8.1 without pinning or doing anything.
@Ryan Nazareth Well, i'm the author of that PR#1243 😄. What i'm thinking to do is stick to TensorFlow 2.8.1 as i think this is the version that others are using too.. I wanted @Eduardo Apolinario (eapolinario)'s opinion on this
r
@cryptic on i see 😃. Does it work for you if you pin it to 2.8.1 - and is this with syncing with version of 
grpcio-status
pinned in master or pinning 
grpcio-status
in 
<http://dev-requirements.in|dev-requirements.in>
? Ive also noticed in this other PR https://github.com/flyteorg/flytekit/pull/1241/files - he has not pinned tensorflow but pinned 
grpcio-status<1.49.0
in 
<http://requirements.in|requirements.in>
. Yeh i guess lets wait for the one of the core team to confirm what the do.
c
@Ryan Nazareth This is in sync with 1.47 pinned in 
<http://requirements.in|requirements.in>
And I remember putting 
grpcio-status<1.49.0
in dev-requirements twice or thrice but still it was giving me an error. Like it was pointed out - this error is being raised by the protobuf version requirements set for TensorFlow v2.10
e
The explanation for this strict version in 
tensorflow
is captured in this comment. Essentially by adding this dependency to flytekit we will be stuck with whatever version of protobuf tensorflow needs (and will block the move to protobuf 4) cc: @Yee
c
So that essentially means we should be pinning tensorflow version to 2.8.1 to prevent changing the protobuf version @Eduardo Apolinario (eapolinario)
e
@cryptic,just following up to your last question here. I'm not sure if we have to pin the tensorflow version to 2.8.1 or this was a bug that the tensorflow team only discovered later. For now, I agree that we should be setting that upper bound to 2.8.1.
c
Okay, thank you for confirming 👍🏻
161 Views
Open in Slack
PreviousNext
Powered by