Flyte enables production-grade orchestration for machine learning workflows and data processing created to accelerate local workflows to production.

Flyte

I just discovered that protobuf version 5 is out and that the newest grpcio already depends on protobuf &gt; 5.
<https://github.com/grpc/grpc/blob/master/requirements.txt>

The entire ML ecosystem of course still depends on version 4.x, let’s please be mindful of when we upgrade to 5 since users can’t really upgrade to 5.x before the most common ML libraries support this :confused:

Sad - this is why we are just trying to migrate to rust grpc 

Yes, protobuf is the issue, not grpc :confused:

The ML ecosystem needs to collectively migrate to the newer version whenever protobuf has a new major version and most packges support either the old or the new version but not both at the same time. This leads to dependency hell between many ml packages for several months whenever protobuf decides to make a breaking change :cry:

So if it would be possible for the same flytekit version to support protobuf 4.x and 5.x at the same time for a while this would help many ML engineers.

Judging from the 3.x to 4.x bump I would expect that tensorflow, torch, … will take several months until they start supporting 5.x so there is no rush.

<https://github.com/onnx/onnx/issues/4239#issuecomment-1346976108> :wink:

> hi <https://github.com/jcwchen|@jcwchen> - do you think the pr will be closed soon? we’re working on an integration between flytekit and onnx and we’ve internally just bumped proto to 4.x - would be great if onnx supported it as well. Thank you.

And <https://github.com/onnx/onnx/issues/4239#issuecomment-1438859559|this> reply:
&gt; @fg91 unfortunately we don’t have a concrete date for this upgrade yet. It really depends on when the whole ONNX ecosystem is ready (PyTorch and TensorFlow are still using quite old version of Protobuf). I will try to reach out relevant team to understand whether there is any concern about this upgrade. Thank you for your interest and waiting!

Thanks for the heads up - this is going to be fun :wink:

Last time they at least published in the docs that a major version was <https://protobuf.dev/news/2022-05-06/|published> and some of the changes. Scanning the <https://github.com/protocolbuffers/protobuf/releases|v27 releases> doesn't seem to point to any major change in python this time around though.