https://flyte.org logo
Powered by
#ask-the-community
Title
# ask-the-community
a

Adrian Loy

11/13/2023, 5:50 PM
We have a problem of trying to run a workflow with python dependencies that have conflicts with flyte dependencies. Is there any neat way or tutorial someone can point me to? I could create two venvs within the docker container in which the task gets executed, and the task could then execute its actual workload with a subprocess using a different venv, but it is a bit cumbersome. Any ideas or thoughts on how to keep the flyte dependencies/environment seperate from the python dependencies needed to actually run a task would be appreciated.
k

Ketan (kumare3)

11/13/2023, 5:51 PM
@Adrian Loy could you please help us understand what dependencies are you having a problem with
we are trying to reduce dependencies within flytekit too
cc @Eduardo Apolinario (eapolinario)
a

Adrian Loy

11/13/2023, 5:54 PM
Afaik remember protobuff 😬 Many packages use it by now but need different versions. Its a pain even without Flyte in the mix. But that is out of my head now, I can come back to you in a couple of days after digging deeper into the issue. It just lead me to the question above if there is an easy way to deal with that in general
n

Niels Bantilan

11/13/2023, 10:35 PM
I created this issue as a stub: https://github.com/flyteorg/flyte/issues/4418 @Adrian Loy please feel free to add to the comments section.
e

Eduardo Apolinario (eapolinario)

11/14/2023, 3:27 AM
Yeah, protobuf released a major version a few months ago and the python ecosystem is slowly adopting it. Can you say a bit more? I'm curious to see if there's any other dependency that we should keep an eye on. As Niels mentioned above, we're going to invest in breaking flytekit apart, but it'd be awesome if we had clear low hanging fruit to adopt right off the bat.
a

Adrian Loy

11/14/2023, 7:56 AM
Thanks for all the replies. I will come back after digging deeper into the issue.
Copy code
ERROR: Cannot install flyteidl==1.3.20, flytekit and tensorflow==2.10.0 because these package versions have conflicting dependencies.

The conflict is caused by:
    tensorflow 2.10.0 depends on protobuf<3.20 and >=3.9.2
    flyteidl 1.3.20 depends on protobuf<5.0.0 and >=4.21.1
This is the issue I have. I can probably fix it by upgrading to tensorflow 1.12 as there doesnt seem to be a good way of keeping the dependency seperate
e

Eduardo Apolinario (eapolinario)

11/20/2023, 6:36 PM
@Adrian Loy, protobuf dependencies caused a lot of headaches in the last year, including to flyte. I have a few questions: 1. Are you pinning flyteidl? Why is version 1.3.20 being installed? 2. As you pointed out tensorflow updated their protobuf dependencies in 2.12. Is there any way you could update that also? I'm surprised that you are only seeing this issue with flytekit and its dependencies as most of the python ecosystem already updated to protobuf>=4.21 We were maintaining a branch to support protobuf<4 this last summer (release notes: https://github.com/flyteorg/flytekit/releases/tag/v1.2.15). If you can't really update flytekit to newer versions you can still use 1.2.16 (which backports changes up to 1.6.2)
a

Adrian Loy

11/21/2023, 9:20 AM
Thanks for reaching out again. My issue is less with updating flyte, but updating tensorflow is more of a problem, since it is a different package that brings this dependency and not a environment/code I control. 
pip install flytekit tensorflow==2.11
gives me 
flytekit==1.2.17
. I was assuming this is too old for me, as we also have some internal flytekitplugin developed. But you are saying it backports changes up to 1.6.2, so it actually might be valid. I give it a shot.
Powered by