Hi All, having issue with recent release of

flyteidl

(1.2.7). I see that the protobuf version is bumped and break compatibility with some python packages.

from flytekit.core.context_manager import ExecutionParameters, FlyteContext, FlyteContextManager, FlyteEntities
/usr/local/lib/python3.7/site-packages/flytekit/core/context_manager.py:30: in <module>
    from flytekit.clients import friendly as friendly_client  # noqa
/usr/local/lib/python3.7/site-packages/flytekit/clients/friendly.py:4: in <module>
    from flyteidl.admin import common_pb2 as _common_pb2
/usr/local/lib/python3.7/site-packages/flyteidl/admin/common_pb2.py:5: in <module>
    from google.protobuf.internal import builder as _builder
E   ImportError: cannot import name 'builder' from 'google.protobuf.internal' (/usr/local/lib/python3.7/site-packages/google/protobuf/internal/__init__.py)

ERROR: flyteidl 1.2.7 has requirement protobuf<5.0.0,>=4.21.1, but you'll have protobuf 3.15.6 which is incompatible.

you mean your internal python packages?

Ah yes, some requires protobuf <4.21.1

sorry not sure what we can do about that

also 3.15.6 is around 2 years old

Yes, but it seems they are still maintaining 3.X , last release is 3.20.3 in Sept 2022.

interesting

Personally I’d like to support 4.X since there is huge performance improvement.

@Pradithya Aria Pura, sorry for the breakage. We're working to bring in the new versions of protobuf and also grcp python libraries, but flyteidl 1.2.7 is in a bad state. I just yanked it from pypi. Expect a new version (this time will be 1.3.0) which will allow us a clean break (since flytekit pins the flyteidl version). Again, sorry for the breakage and we will have this fixed asap.

It’s ok. One question, will the next release of flyte backend be compatible with the current flytekit and flyteidl? In our side, we might need to pin the flytekit/flyteidl version temporarily until we can test the compatibility with newer protobuf, but at the same time still can upgrade the flyte backend.

I don’t think python protobuf version affects wire format

Got it, thanks for confirmation!

that's correct. The main change is in the library used to marshal / unmarshal bytes (in other words, no change to the wire format as Ketan mentioned)