<#1102 fix: make module-level syncify wrappers clo...
# flyte-github
a
#1102 fix: make module-level syncify wrappers cloudpickleable Pull request opened by EngHabu Summary
_SyncWrapper
transitively references a
_BackgroundLoop
whose asyncio loop owns a
ThreadPoolExecutor
backed by a
_queue.SimpleQueue
. When cloudpickle serializes a user function that captures a module-level syncify-wrapped helper — notably
flyte._trace._fetch_action_outputs
— the default reducer walks into the loop's state and dies with:
Copy code
TypeError: cannot pickle '_queue.SimpleQueue' object
when serializing dict item '_work_queue'
when serializing concurrent.futures.thread.ThreadPoolExecutor state
...
when serializing flyte.syncify._api._BackgroundLoop object
when serializing dict item '_bg_loop'
when serializing flyte.syncify._api._SyncWrapper state
...
This shows up in user-facing
flyte.deploy(...)
flows whenever the user has any
@trace
-decorated task — `wrapper_sync`'s globals capture
_fetch_action_outputs
(a
_SyncWrapper
), and that captures
_bg_loop
. PR #1051 added a generic
try/except
around the deploy-time pickle that converts this into a
ClickException
and points the user at
version=...
, but the underlying SDK object is unpicklable for legitimate reasons we control — the proper fix is to make it serialize. Fix Add a
__reduce__
to
_SyncWrapper
that pickles by reference (module + qualname) whenever the wrapper is importable at its original module path. The consumer just re-imports it on load, which is the only sane contract anyway — the bg loop in the producer is meaningless in the consumer process. Falls back to the default reducer for unnamed (closure-local) wrappers so behavior outside the regression is unchanged. Closes • FLYTE-SDK-38 —
TypeError: cannot pickle '_queue.SimpleQueue' object
from
cloudpickle.dump
during
flyte deploy
fixes FLYTE-SDK-38
Test plan • Added
test_module_level_sync_wrapper_is_cloudpickleable
— direct round-trip of
flyte._trace._fetch_action_outputs
. • Added
test_module_level_sync_wrapper_pickleable_via_closure
— round-trip of a function that closes over the wrapper. • Manual repro:
cloudpickle.dumps(TaskEnvironment(...))
with a
@trace
-decorated task now succeeds (was raising
TypeError
). • Full
tests/flyte/syncify/
suite green (34 passed). •
make fmt
clean. 🤖 Generated with Claude Code flyteorg/flyte-sdk