https://flyte.org logo
#ask-the-community
Title
# ask-the-community
a

Andrew

01/04/2024, 4:07 PM
Any recommendations for python dependencies that don’t work with
flytekit
dependencies? I’m trying to use
coremltools
in a pipeline task, but with
flytekit==1.10.2
and
coremltools==5.2.0
, which are the most up to date, supposedly compatible versions for these, using
pip-compile
, I’m getting an error when I import
coremltools
Copy code
TypeError: Descriptors cannot not be created directly.
If this call came from a _pb2.py file, your generated code is out of date and must be regenerated with protoc >= 3.19.0.
If you cannot immediately regenerate your protos, some other possible workarounds are:
 1. Downgrade the protobuf package to 3.20.x or lower.
 2. Set PROTOCOL_BUFFERS_PYTHON_IMPLEMENTATION=python (but this will use pure-Python parsing and will be much slower).
However, It doesn’t look like I can lower the
protobuf
version due to
flytekit
dependencies, and the other option doesn’t seem to be working, but I’m seeing if I can get it to work. Is this something I should submit a ticket for, to try to allow for better compatibility? Usually we use
coremltools==6.2.0
or greater, which doesn’t work at all, but would be great to have compatibility for
m

MUhwjR9VF3K2QJhL2VvNGoFY

01/04/2024, 6:57 PM
protobuf package is not backwards compatible, so one has to to decide to use version >=4.0 or <4.0. flytekit and most other packages have upgraded. So it seems like you are going to have to wait for https://github.com/apple/coremltools/issues/1961 or do the work of updating protobuf in coremltools :/
You could also try using very old flytekit back when it used protobuf <4 but I am not sure how workable that is.
a

Andrew

01/04/2024, 7:01 PM
Ok, that’s good to know, thanks. I’m guessing its impossible to run a task without flytekit installed?
m

MUhwjR9VF3K2QJhL2VvNGoFY

01/04/2024, 7:02 PM
You could have a task with flytekit that shells out to a different venv without flytekit maybe? It would be messy and you would be giving up a lot of flyte functionality.
a

Andrew

01/04/2024, 7:03 PM
Interesting, yeah I see what you’re saying. It could maybe work as a workaround until
coremltools
updates
Though I’m not sure on exactly what that would look like
m

MUhwjR9VF3K2QJhL2VvNGoFY

01/04/2024, 7:13 PM
I think your “best” option is probably to just fork coremltools and build your own wheels with updated protobuf if you want to use it with flyte (or pretty much anything else at this point)
a

Andrew

01/04/2024, 7:48 PM
Sounds good, thanks for the help
l

Lee Ning Jie Leon

01/05/2024, 2:38 AM
you can use flytekit<1.3, that is the latest proto3 supported flytekit which my team are still using till date with flyte 1.10 server. You would miss out on features from 1.3 onwards on the downside
a

Andrew

01/05/2024, 2:39 AM
Yeah I’m trying out 1.2.16, I’ll see if that works out. I saw something about updates though 1.6 being back ported to that or something, so figured I’d try it. Thanks!