https://flyte.org logo
#ask-the-community
Title
# ask-the-community
a

Andrew Korzhuev

03/28/2023, 2:16 PM
Copy code
flyteidl 1.3.12 depends on protobuf<5.0.0 and >=4.21.1
    sagemaker 2.133.0 depends on protobuf<4.0 and >=3.1
Since 1.3 FlyteKit has version conflict with Sagemaker API and Tensorflow as well by deprecating protobuf 3 making it impossible to use those libraries together
f

Fabio Grätz

03/28/2023, 2:17 PM
The Flytekit 1.2.x versions have feature parity (from what I know) with the 1.3.x versions but still support protobuf 3.
a

Andrew Korzhuev

03/28/2023, 2:29 PM
Why is it hard cutoff on protobuf 4? Can it be relaxed in newer version to include 3? @Eduardo Apolinario (eapolinario) I think it’s unreasonable for orchestrator tool to break compatibility with tensorflow and sagemaker, which are the reason to use orchestrator in the first place.
k

Ketan (kumare3)

03/28/2023, 3:01 PM
Are you using sagemaker and tensorflow @Andrew Korzhuev
I think the mistake is to include tensorflow in default core
It should be a plugin
a

Andrew Korzhuev

03/28/2023, 3:03 PM
Yes, we’re using Sagemaker. The issue is version incompatibility with pinned protobuf between flyteidl and sagemaker and tensorflow, plugin wouldn’t solve it
k

Ketan (kumare3)

03/28/2023, 3:04 PM
Hmm interesting - let’s find a solution for this
s

Stephen

03/28/2023, 3:10 PM
I hate this sagemaker plugin.. It’s also a problem for us (I’m the one that commented above you in the sagemaker issue) so I am following to see if we can avoid this problem with Flyte as well
a

Andrew Korzhuev

03/28/2023, 3:11 PM
https://github.com/aws/sagemaker-python-sdk/pull/3138 looks like they pinned it in 2.92.1 release, I’ll try to depend on older version of sagemaker and see if it works, but I would guess it was pinned for a reason, so something will be broken
k

Ketan (kumare3)

03/28/2023, 3:36 PM
I think we are in the process of releasing flytekit backend plugins. We should revisit sagemaker plugin
cc @Eduardo Apolinario (eapolinario) ^
e

Eduardo Apolinario (eapolinario)

03/28/2023, 6:44 PM
@Andrew Korzhuev, first of all, sorry for the breakage. Let's explore some options.
Why is it hard cutoff on protobuf 4? Can it be relaxed in newer version to include 3?
As @Fabio Grätz pointed out, we're maintaining feature parity on the flytekit 1.2.x releases (including the flytekit plugins). We announced this change in flytekit 1.3.0 (check the release notes). I'll make sure to reinforce the message in all releases going forward. Also worth keeping in mind that tensorflow recently announced support for the next version of
protobuf
in their latest release (2.12.0). For the sagemaker plugin specifically, thanks for bringing up the inconsistency. We're going to yank 1.3.x and 1.4.x versions of the sagemaker plugin from pypi as those are not really usable. This will clear up any confusion that people see when installing the plugin (since only versions 1.2.x and below will be available). One final thing, as Ketan already mentioned, we're in the process of releasing flytekit backend plugins (you can follow the work in https://github.com/flyteorg/flyte/issues/3282). Sagemaker is a great candidate for such plugin.
a

Andrew Korzhuev

03/30/2023, 9:48 AM
@Ketan (kumare3) @Eduardo Apolinario (eapolinario) we’re using Sagemaker Python SDK https://github.com/aws/sagemaker-python-sdk, it has hard dependency on protobuf 3 as well, moving to WebAPI would not solve that issue.
This is fundamental problem in Flyte design, so sagemaker, or tensorflow, or any dependency on protobuf 3 is not usable anymore. Given how many dependencies flytekit and flyteidl pull in I imagine there will be more incompatibility issues in the future.
k

Ketan (kumare3)

03/30/2023, 1:19 PM
Why do you think this is an issue in Flyte design? Grpc in my opinion is a core dependency and tbh I feel Google which owns both Grpc and tensorflow needs to get their act together Also sagemaker cannot hard pin. If you are saying our flaw is in using Grpc I agree Other dependencies are all mitigatable
s

Stephen

03/30/2023, 1:22 PM
Yeah I feel like Sagemaker is the problematic one, almost every library we use now requires protobuf>4 and it’s only sagemaker that isn’t doing anything. Tensorflow peeps are currently implementing it from what I’ve seen. Also Sagemaker isn’t even replying to the thread on Github 😓
k

Ketan (kumare3)

03/30/2023, 1:33 PM
@Stephen I agree - we have seen this with sagemaker in the past - abandoning things - not replying. Also Grpc just broke people. Btw @Andrew Korzhuev I do hear you and want to help. Over the longer term we will migrate to connect instead of Grpc. But connect does not yet have python. Connect is Grpc over pure http - so we will all love it. Let’s hope it performs as well
66 Views