flyteidl 1.3.12 depends on protobuf<5.0.0 and >=4.21.1
    sagemaker 2.133.0 depends on protobuf<4.0 and >=3.1

Since 1.3 FlyteKit has version conflict with Sagemaker API and Tensorflow as well by deprecating protobuf 3 making it impossible to use those libraries together

The Flytekit 1.2.x versions have feature parity (from what I know) with the 1.3.x versions but still support protobuf 3.

Why is it hard cutoff on protobuf 4? Can it be relaxed in newer version to include 3? @Eduardo Apolinario (eapolinario) I think it’s unreasonable for orchestrator tool to break compatibility with tensorflow and sagemaker, which are the reason to use orchestrator in the first place.

Are you using sagemaker and tensorflow @Andrew Korzhuev

Yes, we’re using Sagemaker. The issue is version incompatibility with pinned protobuf between flyteidl and sagemaker and tensorflow, plugin wouldn’t solve it

Hmm interesting - let’s find a solution for this

I hate this sagemaker plugin.. It’s also a problem for us (I’m the one that commented above you in the sagemaker issue) so I am following to see if we can avoid this problem with Flyte as well

https://github.com/aws/sagemaker-python-sdk/pull/3138 looks like they pinned it in 2.92.1 release, I’ll try to depend on older version of sagemaker and see if it works, but I would guess it was pinned for a reason, so something will be broken

I think we are in the process of releasing flytekit backend plugins. We should revisit sagemaker plugin

@Andrew Korzhuev, first of all, sorry for the breakage. Let's explore some options.

Why is it hard cutoff on protobuf 4? Can it be relaxed in newer version to include 3?

As @Fabio Grätz pointed out, we're maintaining feature parity on the flytekit 1.2.x releases (including the flytekit plugins). We announced this change in flytekit 1.3.0 (check the release notes). I'll make sure to reinforce the message in all releases going forward. Also worth keeping in mind that tensorflow recently announced support for the next version of

protobuf

in their latest release (2.12.0). For the sagemaker plugin specifically, thanks for bringing up the inconsistency. We're going to yank 1.3.x and 1.4.x versions of the sagemaker plugin from pypi as those are not really usable. This will clear up any confusion that people see when installing the plugin (since only versions 1.2.x and below will be available). One final thing, as Ketan already mentioned, we're in the process of releasing flytekit backend plugins (you can follow the work in https://github.com/flyteorg/flyte/issues/3282). Sagemaker is a great candidate for such plugin.

@Ketan (kumare3) @Eduardo Apolinario (eapolinario) we’re using Sagemaker Python SDK https://github.com/aws/sagemaker-python-sdk, it has hard dependency on protobuf 3 as well, moving to WebAPI would not solve that issue.

Why do you think this is an issue in Flyte design? Grpc in my opinion is a core dependency and tbh I feel Google which owns both Grpc and tensorflow needs to get their act together Also sagemaker cannot hard pin. If you are saying our flaw is in using Grpc I agree Other dependencies are all mitigatable

Yeah I feel like Sagemaker is the problematic one, almost every library we use now requires protobuf>4 and it’s only sagemaker that isn’t doing anything. Tensorflow peeps are currently implementing it from what I’ve seen. Also Sagemaker isn’t even replying to the thread on Github 😓

@Stephen I agree - we have seen this with sagemaker in the past - abandoning things - not replying. Also Grpc just broke people. Btw @Andrew Korzhuev I do hear you and want to help. Over the longer term we will migrate to connect instead of Grpc. But connect does not yet have python. Connect is Grpc over pure http - so we will all love it. Let’s hope it performs as well