We have a problem of trying to run a workflow with python dependencies that have conflicts with flyte dependencies. Is there any neat way or tutorial someone can point me to? I could create two venvs within the docker container in which the task gets executed, and the task could then execute its actual workload with a subprocess using a different venv, but it is a bit cumbersome. Any ideas or thoughts on how to keep the flyte dependencies/environment seperate from the python dependencies needed to actually run a task would be appreciated.

@nutritious-manchester-15545 could you please help us understand what dependencies are you having a problem with

Afaik remember protobuff 😬 Many packages use it by now but need different versions. Its a pain even without Flyte in the mix. But that is out of my head now, I can come back to you in a couple of days after digging deeper into the issue. It just lead me to the question above if there is an easy way to deal with that in general

I created this issue as a stub: https://github.com/flyteorg/flyte/issues/4418 @nutritious-manchester-15545 please feel free to add to the comments section.

Yeah, protobuf released a major version a few months ago and the python ecosystem is slowly adopting it. Can you say a bit more? I'm curious to see if there's any other dependency that we should keep an eye on. As Niels mentioned above, we're going to invest in breaking flytekit apart, but it'd be awesome if we had clear low hanging fruit to adopt right off the bat.

Thanks for all the replies. I will come back after digging deeper into the issue.

@nutritious-manchester-15545, protobuf dependencies caused a lot of headaches in the last year, including to flyte. I have a few questions: 1. Are you pinning flyteidl? Why is version 1.3.20 being installed? 2. As you pointed out tensorflow updated their protobuf dependencies in 2.12. Is there any way you could update that also? I'm surprised that you are only seeing this issue with flytekit and its dependencies as most of the python ecosystem already updated to protobuf>=4.21 We were maintaining a branch to support protobuf<4 this last summer (release notes: https://github.com/flyteorg/flytekit/releases/tag/v1.2.15). If you can't really update flytekit to newer versions you can still use 1.2.16 (which backports changes up to 1.6.2)

Thanks for reaching out again. My issue is less with updating flyte, but updating tensorflow is more of a problem, since it is a different package that brings this dependency and not a environment/code I control.

pip install flytekit tensorflow==2.11

gives me

flytekit==1.2.17

. I was assuming this is too old for me, as we also have some internal flytekitplugin developed. But you are saying it backports changes up to 1.6.2, so it actually might be valid. I give it a shot.