Another note on our upgrade from flytekit v1.15.X ...
# flyte-v1-supportc
clean-glass-36808
05/12/2026, 6:52 PMAnother note on our upgrade from flytekit v1.15.X to v1.16.20. We're seeing some tasks with 1Gi of memory OOM on our pull request to upgrade Flytekit. Before I go down a rabbit hole is anyone aware of any performance regressions in newer versions of flytekit?
clean-glass-36808
05/12/2026, 7:22 PMOne difference I see is that we're getting many more exceptions logged about Pydantic in newer versions...
f
freezing-airport-6809
05/14/2026, 4:39 AM
it has to be either fsspec (one of the underlying libraries) or msgpack if this is lists etc
g
glamorous-carpet-83516
05/14/2026, 4:54 AMCould you share that exception logs?
glamorous-carpet-83516
05/14/2026, 4:55 AMOr the example you ran
c
clean-glass-36808
05/14/2026, 5:04 AMI can share the exception logs. The workflow is fairly complicated so I don't think I can share that. I figured it might be fsspec as well but I don't think it changed between versions based on the diff I looked at
clean-glass-36808
05/14/2026, 5:06 AMThe exceptions look like this, they are warning logs but based on the code it looks like its coming from a catch block.
Copy code
2026-05-13 22:17:22.386 warn {"asctime": "2026-05-14 02:17:22,385", "name": "flytekit", "levelname": "WARNING", "message": "Field output_sensors_to_update of type <redacted> cannot be converted to a literal type. Error: Type of Generic List type is not supported, Type of Generic Union type is not supported, Only EnumTypes with value of string are supported"}
2026-05-13 22:17:22.386 warn {"asctime": "2026-05-14 02:17:22,386", "name": "flytekit", "levelname": "WARNING", "message": "Field dataset_id of type <enum 'MLDatasetId'> cannot be converted to a literal type. Error: Only EnumTypes with value of string are supported"}
2026-05-13 22:17:22.389 warn {"asctime": "2026-05-14 02:17:22,389", "name": "flytekit", "levelname": "WARNING", "message": "<redacted> cannot be converted to a literal type. Error: Transformer for type <class 'tuple'> is restricted currently"}g
glamorous-carpet-83516
05/14/2026, 5:15 AMI feel like the outputs become pickle. did you see any logs related to pickle?
c
clean-glass-36808
05/14/2026, 2:54 PMUntitled
clean-glass-36808
05/14/2026, 2:54 PMNo mention of pickle in the logs but we do have some custom patches on the flytekit wheel which may affect things.
clean-glass-36808
05/14/2026, 2:58 PMI can revert some of those changes and re-run
5 Views