Any recommendations for python dependencies that don’t work with

flytekit

dependencies? I’m trying to use

coremltools

in a pipeline task, but with

flytekit==1.10.2

and

coremltools==5.2.0

, which are the most up to date, supposedly compatible versions for these, using

pip-compile

, I’m getting an error when I import

coremltools

TypeError: Descriptors cannot not be created directly.
If this call came from a _pb2.py file, your generated code is out of date and must be regenerated with protoc >= 3.19.0.
If you cannot immediately regenerate your protos, some other possible workarounds are:
 1. Downgrade the protobuf package to 3.20.x or lower.
 2. Set PROTOCOL_BUFFERS_PYTHON_IMPLEMENTATION=python (but this will use pure-Python parsing and will be much slower).

However, It doesn’t look like I can lower the

protobuf

version due to

flytekit

dependencies, and the other option doesn’t seem to be working, but I’m seeing if I can get it to work. Is this something I should submit a ticket for, to try to allow for better compatibility? Usually we use

coremltools==6.2.0

or greater, which doesn’t work at all, but would be great to have compatibility for

protobuf package is not backwards compatible, so one has to to decide to use version >=4.0 or <4.0. flytekit and most other packages have upgraded. So it seems like you are going to have to wait for https://github.com/apple/coremltools/issues/1961 or do the work of updating protobuf in coremltools :/

Ok, that’s good to know, thanks. I’m guessing its impossible to run a task without flytekit installed?

You could have a task with flytekit that shells out to a different venv without flytekit maybe? It would be messy and you would be giving up a lot of flyte functionality.

Interesting, yeah I see what you’re saying. It could maybe work as a workaround until

coremltools

updates

I think your “best” option is probably to just fork coremltools and build your own wheels with updated protobuf if you want to use it with flyte (or pretty much anything else at this point)

Sounds good, thanks for the help

you can use flytekit<1.3, that is the latest proto3 supported flytekit which my team are still using till date with flyte 1.10 server. You would miss out on features from 1.3 onwards on the downside

Yeah I’m trying out 1.2.16, I’ll see if that works out. I saw something about updates though 1.6 being back ported to that or something, so figured I’d try it. Thanks!