acoustic-carpenter-78188
05/19/2026, 9:18 PM_token_client.py the line:
response = await http_session.post(token_endpoint, data=body, headers=headers)
can throw an httpx exception - it never gets to the error categorization block we have below starting on line 154.
This bare exception bubbles all the way up to _bg_run from _bg_launch.
Change is to catch these exceptions and convert them to one of our SlowDown errors so it gets properly retried.
Alternatives:
• Why not re-raise a SlowDownError directly? Mixes up errors from different layers, feels really bad.
• Why not catch inside get_token and directly re-raise AuthenticationPending? get_token is called by both device code and client credentials - the latter doesn't handle the pending exception at all. Can add a new one, but using the higher level retry mechanism feels a bit more right.
Separately,
• Also updating the error message to be better, seeing "0/10" is weird.
Stack Trace from user
[1x] Controller failed, system retries 0 / 10 crossed threshold, for action rii0qtww9o8jflxw8k448qvv:
Traceback (most recent call last):
File "/root/.venv/lib/python3.12/site-packages/httpx/_transports/default.py", line 101, in map_httpcore_exceptions
yield
File "/root/.venv/lib/python3.12/site-packages/httpx/_transports/default.py", line 394, in handle_async_request
resp = await self._pool.handle_async_request(req)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/root/.venv/lib/python3.12/site-packages/httpcore/_async/connection_pool.py", line 256, in handle_async_request
raise exc from None
File "/root/.venv/lib/python3.12/site-packages/httpcore/_async/connection_pool.py", line 236, in handle_async_request
response = await connection.handle_async_request(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/root/.venv/lib/python3.12/site-packages/httpcore/_async/connection.py", line 101, in handle_async_request
raise exc
File "/root/.venv/lib/python3.12/site-packages/httpcore/_async/connection.py", line 78, in handle_async_request
stream = await self._connect(request)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/root/.venv/lib/python3.12/site-packages/httpcore/_async/connection.py", line 124, in _connect
stream = await self._network_backend.connect_tcp(**kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/root/.venv/lib/python3.12/site-packages/httpcore/_backends/auto.py", line 31, in connect_tcp
return await self._backend.connect_tcp(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/root/.venv/lib/python3.12/site-packages/httpcore/_backends/anyio.py", line 113, in connect_tcp
with map_exceptions(exc_map):
^^^^^^^^^^^^^^^^^^^^^^^
File "/root/.local/share/uv/python/cpython-3.12.13-linux-x86_64-gnu/lib/python3.12/contextlib.py", line 158, in __exit__
self.gen.throw(value)
File "/root/.venv/lib/python3.12/site-packages/httpcore/_exceptions.py", line 14, in map_exceptions
raise to_exc(exc) from exc
httpcore.ConnectTimeout
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/root/.venv/lib/python3.12/site-packages/flyte/_internal/controllers/remote/_core.py", line 465, in _bg_run
await self._bg_process(action)
File "/root/.venv/lib/python3.12/site-packages/flyte/_internal/controllers/remote/_core.py", line 437, in _bg_process
await self._bg_launch(action)
File "/root/.venv/lib/python3.12/site-packages/flyte/_internal/controllers/remote/_core.py", line 381, in _bg_launch
await self._actions_service.Enqueue(
File "/root/.venv/lib/python3.12/site-packages/grpc/aio/_interceptor.py", line 471, in __await__
call = yield from self._interceptors_task.__await__()
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/root/.venv/lib/python3.12/site-packages/grpc/aio/_interceptor.py", line 701, in _invoke
return await _run_interceptor(
^^^^^^^^^^^^^^^^^^^^^^^
File "/root/.venv/lib/python3.12/site-packages/grpc/aio/_interceptor.py", line 675, in _run_interceptor
call_or_response = await interceptors[0].intercept_unary_unary(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/root/.venv/lib/python3.12/site-packages/flyte/remote/_client/auth/_grpc_utils/default_metadata_interceptor.py", line 103, in intercept_unary_unary
return await (await continuation(updated_call_details, request))
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/root/.venv/lib/python3.12/site-packages/grpc/aio/_interceptor.py", line 675, in _run_interceptor
call_or_response = await interceptors[0].intercept_unary_unary(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/root/.venv/lib/python3.12/site-packages/flyte/remote/_client/auth/_grpc_utils/auth_interceptor.py", line 79, in intercept_unary_unary
await self.authenticator.refresh_credentials(creds_id=creds_id)
File "/root/.venv/lib/python3.12/site-packages/flyte/remote/_client/auth/_authenticators/base.py", line 182, in refresh_credentials
self._creds = await self._do_refresh_credentials()
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/root/.venv/lib/python3.12/site-packages/flyte/remote/_client/auth/_authenticators/client_credentials.py", line 60, in _do_refresh_credentials
token, refresh_token, expires_in = await token_client.get_token(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/root/.venv/lib/python3.12/site-packages/flyte/remote/_client/auth/_token_client.py", line 152, in get_token
response = await http_session.post(token_endpoint, data=body, headers=headers)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/root/.venv/lib/python3.12/site-packages/httpx/_client.py", line 1859, in post
return await self.request(
^^^^^^^^^^^^^^^^^^^
File "/root/.venv/lib/python3.12/site-packages/httpx/_client.py", line 1540, in request
return await self.send(request, auth=auth, follow_redirects=follow_redirects)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/root/.venv/lib/python3.12/site-packages/httpx/_client.py", line 1629, in send
response = await self._send_handling_auth(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/root/.venv/lib/python3.12/site-packages/httpx/_client.py", line 1657, in _send_handling_auth
response = await self._send_handling_redirects(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/root/.venv/lib/python3.12/site-packages/httpx/_client.py", line 1694, in _send_handling_redirects
response = await self._send_single_request(request)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/root/.venv/lib/python3.12/site-packages/httpx/_client.py", line 1730, in _send_single_request
response = await transport.handle_async_request(request)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/root/.venv/lib/python3.12/site-packages/httpx/_transports/default.py", line 393, in handle_async_request
with map_httpcore_exceptions():
^^^^^^^^^^^^^^^^^^^^^^^^^
File "/root/.local/share/uv/python/cpython-3.12.13-linux-x86_64-gnu/lib/python3.12/contextlib.py", line 158, in __exit__
self.gen.throw(value)
File "/root/.venv/lib/python3.12/site-packages/httpx/_transports/default.py", line 118, in map_httpcore_exceptions
raise mapped_exc(message) from exc
httpx.ConnectTimeout
Root cause
The retry path in _bg_run only catches flyte.errors.SlowDownError. Everything else falls through to the outer except Exception, which marks the action terminally failed. The exception classifier inside _bg_launch only handles connectrpc.errors.ConnectError — so a bare httpx.ConnectTimeout (or any other httpx.TransportError) raised from the auth interceptor duri…
flyteorg/flyte-sdkacoustic-carpenter-78188
05/19/2026, 9:18 PMacoustic-carpenter-78188
05/19/2026, 10:14 PM