https://flyte.org logo
Powered by
#flytekit
Title
# flytekit
c

cryptic

10/21/2022, 11:34 AM
I'm facing deps issues when trying to 
make dev-requirements.txt
I've 
grpcio-status<1.49.0
added to the 
<http://dev-requirements.in|dev-requirements.in>
Copy code
Could not find a version that matches protobuf!=3.20.0,!=3.20.1,!=4.21.0,!=4.21.1,!=4.21.2,!=4.21.3,!=4.21.4,!=4.21.5,<3.20,<5.0.0dev,==3.20.3,>=3.19.5,>=3.9.2 (from -c requirements.txt (line 121))
Tried: 2.0.3, 2.3.0, 2.4.1, 2.5.0, 2.6.0, 2.6.1, 3.0.0, 3.0.0, 3.1.0, 3.1.0.post1, 3.1.0.post1, 3.2.0, 3.2.0, 3.3.0, 3.4.0, 3.4.0, 3.5.0.post1, 3.5.0.post1, 3.5.1, 3.5.1, 3.5.2, 3.5.2, 3.5.2.post1, 3.5.2.post1, 3.6.0, 3.6.0, 3.6.1, 3.6.1, 3.7.0, 3.7.0, 3.7.1, 3.7.1, 3.8.0, 3.8.0, 3.9.0, 3.9.0, 3.9.1, 3.9.1, 3.9.2, 3.9.2, 3.10.0, 3.10.0, 3.11.0, 3.11.0, 3.11.1, 3.11.1, 3.11.2, 3.11.2, 3.11.2, 3.11.3, 3.11.3, 3.11.3, 3.12.0, 3.12.1, 3.12.2, 3.12.2, 3.12.2, 3.12.4, 3.12.4, 3.12.4, 3.13.0, 3.13.0, 3.13.0, 3.14.0, 3.14.0, 3.14.0, 3.15.0, 3.15.0, 3.15.0, 3.15.1, 3.15.1, 3.15.1, 3.15.2, 3.15.2, 3.15.2, 3.15.3, 3.15.3, 3.15.3, 3.15.4, 3.15.4, 3.15.4, 3.15.5, 3.15.5, 3.15.5, 3.15.6, 3.15.6, 3.15.6, 3.15.7, 3.15.7, 3.15.7, 3.15.8, 3.15.8, 3.15.8, 3.16.0, 3.16.0, 3.16.0, 3.17.0, 3.17.0, 3.17.0, 3.17.1, 3.17.1, 3.17.1, 3.17.2, 3.17.2, 3.17.2, 3.17.3, 3.17.3, 3.17.3, 3.18.0, 3.18.0, 3.18.0, 3.18.1, 3.18.1, 3.18.1, 3.18.3, 3.18.3, 3.18.3, 3.19.0, 3.19.0, 3.19.0, 3.19.1, 3.19.1, 3.19.1, 3.19.2, 3.19.2, 3.19.2, 3.19.3, 3.19.3, 3.19.3, 3.19.4, 3.19.4, 3.19.4, 3.19.5, 3.19.5, 3.19.5, 3.19.6, 3.19.6, 3.19.6, 3.20.0, 3.20.0, 3.20.0, 3.20.1, 3.20.1, 3.20.1, 3.20.2, 3.20.2, 3.20.2, 3.20.3, 3.20.3, 3.20.3, 4.21.0, 4.21.0, 4.21.0, 4.21.0, 4.21.1, 4.21.1, 4.21.1, 4.21.1, 4.21.2, 4.21.2, 4.21.2, 4.21.2, 4.21.3, 4.21.3, 4.21.3, 4.21.3, 4.21.4, 4.21.4, 4.21.4, 4.21.4, 4.21.5, 4.21.5, 4.21.5, 4.21.5, 4.21.6, 4.21.6, 4.21.6, 4.21.6, 4.21.7, 4.21.7, 4.21.7, 4.21.7, 4.21.8, 4.21.8, 4.21.8, 4.21.8
Skipped pre-versions: 2.0.0b0, 3.0.0a2, 3.0.0a3, 3.0.0b1, 3.0.0b1.post1, 3.0.0b1.post2, 3.0.0b2, 3.0.0b2, 3.0.0b2.post1, 3.0.0b2.post1, 3.0.0b2.post2, 3.0.0b2.post2, 3.0.0b3, 3.0.0b4, 3.0.0b4, 3.2.0rc1, 3.2.0rc1, 3.2.0rc1.post1, 3.2.0rc1.post1, 3.2.0rc2, 3.2.0rc2, 3.7.0rc2, 3.7.0rc2, 3.7.0rc3, 3.7.0rc3, 3.8.0rc1, 3.8.0rc1, 3.9.0rc1, 3.9.0rc1, 3.10.0rc1, 3.10.0rc1, 3.11.0rc1, 3.11.0rc1, 3.11.0rc2, 3.11.0rc2, 3.12.0rc1, 3.12.0rc2, 3.13.0rc3, 3.13.0rc3, 3.13.0rc3, 3.14.0rc1, 3.14.0rc1, 3.14.0rc1, 3.14.0rc2, 3.14.0rc2, 3.14.0rc2, 3.14.0rc3, 3.14.0rc3, 3.14.0rc3, 3.15.0rc1, 3.15.0rc1, 3.15.0rc1, 3.15.0rc2, 3.15.0rc2, 3.15.0rc2, 3.16.0rc1, 3.16.0rc1, 3.16.0rc1, 3.16.0rc2, 3.16.0rc2, 3.16.0rc2, 3.17.0rc1, 3.17.0rc1, 3.17.0rc1, 3.17.0rc2, 3.17.0rc2, 3.17.0rc2, 3.18.0rc1, 3.18.0rc1, 3.18.0rc1, 3.18.0rc2, 3.18.0rc2, 3.18.0rc2, 3.19.0rc1, 3.19.0rc1, 3.19.0rc1, 3.19.0rc2, 3.19.0rc2, 3.19.0rc2, 3.20.0rc1, 3.20.0rc1, 3.20.0rc1, 3.20.0rc2, 3.20.0rc2, 3.20.0rc2, 3.20.1rc1, 3.20.1rc1, 3.20.1rc1, 4.0.0rc1, 4.0.0rc1, 4.0.0rc1, 4.0.0rc2, 4.0.0rc2, 4.0.0rc2, 4.21.0rc1, 4.21.0rc1, 4.21.0rc1, 4.21.0rc1, 4.21.0rc2, 4.21.0rc2, 4.21.0rc2, 4.21.0rc2
There are incompatible versions in the resolved dependencies:
  protobuf==3.20.3 (from -c requirements.txt (line 121))
  protobuf<3.20,>=3.9.2 (from tensorflow==2.10.0->-r <http://dev-requirements.in|dev-requirements.in> (line 16))
  protobuf!=3.20.0,!=3.20.1,!=4.21.0,!=4.21.1,!=4.21.2,!=4.21.3,!=4.21.4,!=4.21.5,<5.0.0dev,>=3.19.5 (from google-cloud-bigquery-storage==2.16.2->-r <http://dev-requirements.in|dev-requirements.in> (line 13))
  protobuf!=3.20.0,!=3.20.1,!=4.21.0,!=4.21.1,!=4.21.2,!=4.21.3,!=4.21.4,!=4.21.5,<5.0.0dev,>=3.19.5 (from google-cloud-bigquery==3.3.5->-r <http://dev-requirements.in|dev-requirements.in> (line 12))
make: *** [Makefile:66: dev-requirements.txt] Error 2
I'm using a slightly older clone of the repo which was the one before @Dennis O'Brien’s fix
y

Yee

10/21/2022, 4:44 PM
cc @Eduardo Apolinario (eapolinario)
can you add
Copy code
grpcio<=1.47.0
grpcio-status<=1.47.0
we really need to release this pin
c

cryptic

10/21/2022, 4:47 PM
I've gone through that PR and tried doing that too, but I still got a make error
Copy code
Could not find a version that matches protobuf!=3.20.0,!=3.20.1,!=4.21.0,!=4.21.1,!=4.21.2,!=4.21.3,!=4.21.4,!=4.21.5,<3.20,<5.0.0dev,==3.20.3,>=3.19.5,>=3.9.2 (from -c requirements.txt (line 122))
Tried: 2.0.3, 2.3.0, 2.4.1, 2.5.0, 2.6.0, 2.6.1, 3.0.0, 3.0.0, 3.1.0, 3.1.0.post1, 3.1.0.post1, 3.2.0, 3.2.0, 3.3.0, 3.4.0, 3.4.0, 3.5.0.post1, 3.5.0.post1, 3.5.1, 3.5.1, 3.5.2, 3.5.2, 3.5.2.post1, 3.5.2.post1, 3.6.0, 3.6.0, 3.6.1, 3.6.1, 3.7.0, 3.7.0, 3.7.1, 3.7.1, 3.8.0, 3.8.0, 3.9.0, 3.9.0, 3.9.1, 3.9.1, 3.9.2, 3.9.2, 3.10.0, 3.10.0, 3.11.0, 3.11.0, 3.11.1, 3.11.1, 3.11.2, 3.11.2, 3.11.2, 3.11.3, 3.11.3, 3.11.3, 3.12.0, 3.12.1, 3.12.2, 3.12.2, 3.12.2, 3.12.4, 3.12.4, 3.12.4, 3.13.0, 3.13.0, 3.13.0, 3.14.0, 3.14.0, 3.14.0, 3.15.0, 3.15.0, 3.15.0, 3.15.1, 3.15.1, 3.15.1, 3.15.2, 3.15.2, 3.15.2, 3.15.3, 3.15.3, 3.15.3, 3.15.4, 3.15.4, 3.15.4, 3.15.5, 3.15.5, 3.15.5, 3.15.6, 3.15.6, 3.15.6, 3.15.7, 3.15.7, 3.15.7, 3.15.8, 3.15.8, 3.15.8, 3.16.0, 3.16.0, 3.16.0, 3.17.0, 3.17.0, 3.17.0, 3.17.1, 3.17.1, 3.17.1, 3.17.2, 3.17.2, 3.17.2, 3.17.3, 3.17.3, 3.17.3, 3.18.0, 3.18.0, 3.18.0, 3.18.1, 3.18.1, 3.18.1, 3.18.3, 3.18.3, 3.18.3, 3.19.0, 3.19.0, 3.19.0, 3.19.1, 3.19.1, 3.19.1, 3.19.2, 3.19.2, 3.19.2, 3.19.3, 3.19.3, 3.19.3, 3.19.4, 3.19.4, 3.19.4, 3.19.5, 3.19.5, 3.19.5, 3.19.6, 3.19.6, 3.19.6, 3.20.0, 3.20.0, 3.20.0, 3.20.1, 3.20.1, 3.20.1, 3.20.2, 3.20.2, 3.20.2, 3.20.3, 3.20.3, 3.20.3, 4.21.0, 4.21.0, 4.21.0, 4.21.0, 4.21.1, 4.21.1, 4.21.1, 4.21.1, 4.21.2, 4.21.2, 4.21.2, 4.21.2, 4.21.3, 4.21.3, 4.21.3, 4.21.3, 4.21.4, 4.21.4, 4.21.4, 4.21.4, 4.21.5, 4.21.5, 4.21.5, 4.21.5, 4.21.6, 4.21.6, 4.21.6, 4.21.6, 4.21.7, 4.21.7, 4.21.7, 4.21.7, 4.21.8, 4.21.8, 4.21.8, 4.21.8
Skipped pre-versions: 2.0.0b0, 3.0.0a2, 3.0.0a3, 3.0.0b1, 3.0.0b1.post1, 3.0.0b1.post2, 3.0.0b2, 3.0.0b2, 3.0.0b2.post1, 3.0.0b2.post1, 3.0.0b2.post2, 3.0.0b2.post2, 3.0.0b3, 3.0.0b4, 3.0.0b4, 3.2.0rc1, 3.2.0rc1, 3.2.0rc1.post1, 3.2.0rc1.post1, 3.2.0rc2, 3.2.0rc2, 3.7.0rc2, 3.7.0rc2, 3.7.0rc3, 3.7.0rc3, 3.8.0rc1, 3.8.0rc1, 3.9.0rc1, 3.9.0rc1, 3.10.0rc1, 3.10.0rc1, 3.11.0rc1, 3.11.0rc1, 3.11.0rc2, 3.11.0rc2, 3.12.0rc1, 3.12.0rc2, 3.13.0rc3, 3.13.0rc3, 3.13.0rc3, 3.14.0rc1, 3.14.0rc1, 3.14.0rc1, 3.14.0rc2, 3.14.0rc2, 3.14.0rc2, 3.14.0rc3, 3.14.0rc3, 3.14.0rc3, 3.15.0rc1, 3.15.0rc1, 3.15.0rc1, 3.15.0rc2, 3.15.0rc2, 3.15.0rc2, 3.16.0rc1, 3.16.0rc1, 3.16.0rc1, 3.16.0rc2, 3.16.0rc2, 3.16.0rc2, 3.17.0rc1, 3.17.0rc1, 3.17.0rc1, 3.17.0rc2, 3.17.0rc2, 3.17.0rc2, 3.18.0rc1, 3.18.0rc1, 3.18.0rc1, 3.18.0rc2, 3.18.0rc2, 3.18.0rc2, 3.19.0rc1, 3.19.0rc1, 3.19.0rc1, 3.19.0rc2, 3.19.0rc2, 3.19.0rc2, 3.20.0rc1, 3.20.0rc1, 3.20.0rc1, 3.20.0rc2, 3.20.0rc2, 3.20.0rc2, 3.20.1rc1, 3.20.1rc1, 3.20.1rc1, 4.0.0rc1, 4.0.0rc1, 4.0.0rc1, 4.0.0rc2, 4.0.0rc2, 4.0.0rc2, 4.21.0rc1, 4.21.0rc1, 4.21.0rc1, 4.21.0rc1, 4.21.0rc2, 4.21.0rc2, 4.21.0rc2, 4.21.0rc2
There are incompatible versions in the resolved dependencies:
  protobuf==3.20.3 (from -c requirements.txt (line 122))
  protobuf<3.20,>=3.9.2 (from tensorflow==2.10.0->-r <http://dev-requirements.in|dev-requirements.in> (line 16))
  protobuf!=3.20.0,!=3.20.1,!=4.21.0,!=4.21.1,!=4.21.2,!=4.21.3,!=4.21.4,!=4.21.5,<5.0.0dev,>=3.19.5 (from google-cloud-bigquery==3.3.5->-r <http://dev-requirements.in|dev-requirements.in> (line 12))
  protobuf!=3.20.0,!=3.20.1,!=4.21.0,!=4.21.1,!=4.21.2,!=4.21.3,!=4.21.4,!=4.21.5,<5.0.0dev,>=3.19.5 (from google-cloud-bigquery-storage==2.16.2->-r <http://dev-requirements.in|dev-requirements.in> (line 13))
this is what I got after adding 1.47.0 @Yee which is pretty much the same
a

Andrew Achkar

10/21/2022, 5:38 PM
The error you're seeing now links the issue to tensorflow
```dependencies:
protobuf==3.20.3 (from -c requirements.txt (line 122))
protobuf<3.20,>=3.9.2 (from tensorflow==2.10.0->-r dev-requirements.in (line 16))```
https://github.com/tensorflow/tensorflow/blob/v2.10.0/tensorflow/tools/pip_package/setup.py#L96 tf 2.10.0 requires protobuf < 3.20 but your requirements.txt seems like it explicitly wants 3.20.3 try 3.19.6?
c

cryptic

10/21/2022, 6:11 PM
@Andrew Achkar Thank you, you were right, i'd to set the version inside 
<http://requirements.in|requirements.in>
.
is it worth opening an issue on this? @Yee
that protobuf versioning is odd
@Eduardo Apolinario (eapolinario) Would it be the right solution to pin the version of protobuf?
e

Eduardo Apolinario (eapolinario)

10/22/2022, 4:53 PM
Are you changing the tensorflow version?
d

Dennis O'Brien

10/22/2022, 4:57 PM
I see that @Ryan Nazareth has worked on a few PRs related to tensorflow, but I didn't notice any version pinning. @cryptic Are you including tensorflow with a version constraint in 
<http://dev-requirements.in|dev-requirements.in>
or 
<http://requirements.in|requirements.in>
?
r

Ryan Nazareth

10/22/2022, 7:01 PM
@cryptic Actually i’ve made a comment regarding this https://github.com/flyteorg/flytekit/pull/1240#discussion_r1002499826 - this may or may not be due to 
grpcio-status
version pinned but when Ive pinned 
grpcio-status<1.49.0
in 
<http://dev-requirements.in|dev-requirements.in>
as @Kevin Su suggested here https://github.com/flyteorg/flytekit/pull/1241#issuecomment-1283098765 and it seemed to work for me and bypass that 
protobuf
conflict (before syncing with master). However, 
grpcio-status
is also pinned in 
<http://requirements.in|requirements.in>
master as @Dennis O'Brien mentioned with his PR, but pinned to 
grpcio-status<=1.47.0
. When i tried to 
make <http://dev-requirements.in|dev-requirements.in>
after syncing with master - i get the same error, when installing tensorflow. Also reported in another PR , although the author of that PR seemed to be experiencing the same issue even with pinning 
grpcio-status<1.49.0
, which worked in my case, so not sure https://github.com/flyteorg/flytekit/pull/1243 i haven’t tried pinning the version of protobuf though.
c

cryptic

10/23/2022, 7:12 AM
@Eduardo Apolinario (eapolinario) Sorry for the late response, but I never tried changing the version of tf as i never felt the need to. It picks up the latest 2.10 when i make 
<http://dev-requirements.in|dev-requirements.in>
by default
@Dennis O'Brien Yes i've made the same observation as you did. But in other people's PRs, especially from @Ryan Nazareth's PR i've seen the default version of TensorFlow installed was 2.8.1 without pinning or doing anything.
@Ryan Nazareth Well, i'm the author of that PR#1243 😄. What i'm thinking to do is stick to TensorFlow 2.8.1 as i think this is the version that others are using too.. I wanted @Eduardo Apolinario (eapolinario)'s opinion on this
r

Ryan Nazareth

10/23/2022, 11:58 PM
@cryptic on i see 😃. Does it work for you if you pin it to 2.8.1 - and is this with syncing with version of 
grpcio-status
pinned in master or pinning 
grpcio-status
in 
<http://dev-requirements.in|dev-requirements.in>
? Ive also noticed in this other PR https://github.com/flyteorg/flytekit/pull/1241/files - he has not pinned tensorflow but pinned 
grpcio-status<1.49.0
in 
<http://requirements.in|requirements.in>
. Yeh i guess lets wait for the one of the core team to confirm what the do.
c

cryptic

10/24/2022, 3:21 AM
@Ryan Nazareth This is in sync with 1.47 pinned in 
<http://requirements.in|requirements.in>
And I remember putting 
grpcio-status<1.49.0
in dev-requirements twice or thrice but still it was giving me an error. Like it was pointed out - this error is being raised by the protobuf version requirements set for TensorFlow v2.10
e

Eduardo Apolinario (eapolinario)

10/25/2022, 3:38 PM
The explanation for this strict version in 
tensorflow
is captured in this comment. Essentially by adding this dependency to flytekit we will be stuck with whatever version of protobuf tensorflow needs (and will block the move to protobuf 4) cc: @Yee
c

cryptic

10/25/2022, 4:15 PM
So that essentially means we should be pinning tensorflow version to 2.8.1 to prevent changing the protobuf version @Eduardo Apolinario (eapolinario)
e

Eduardo Apolinario (eapolinario)

10/26/2022, 5:06 PM
@cryptic,just following up to your last question here. I'm not sure if we have to pin the tensorflow version to 2.8.1 or this was a bug that the tensorflow team only discovered later. For now, I agree that we should be setting that upper bound to 2.8.1.
c

cryptic

10/26/2022, 5:07 PM
Okay, thank you for confirming 👍🏻
5 Views
Powered by