<#1090 Catch and retry refresh credential errors> ...
# flyte-github
a
#1090 Catch and retry refresh credential errors Pull request opened by wild-endeavor Summary In
_token_client.py
the line:
Copy code
response = await http_session.post(token_endpoint, data=body, headers=headers)
can throw an
httpx
exception - it never gets to the error categorization block we have below starting on line 154. This bare exception bubbles all the way up to
_bg_run
from
_bg_launch
. Change is to catch these exceptions and convert them to one of our
SlowDown
errors so it gets properly retried. Alternatives: • Why not re-raise a
SlowDownError
directly? Mixes up errors from different layers, feels really bad. • Why not catch inside
get_token
and directly re-raise
AuthenticationPending
?
get_token
is called by both device code and client credentials - the latter doesn't handle the pending exception at all. Can add a new one, but using the higher level retry mechanism feels a bit more right. Separately, • Also updating the error message to be better, seeing "0/10" is weird. Stack Trace from user
Copy code
[1x] Controller failed, system retries 0 / 10 crossed threshold, for action rii0qtww9o8jflxw8k448qvv:
Traceback (most recent call last):
  File "/root/.venv/lib/python3.12/site-packages/httpx/_transports/default.py", line 101, in map_httpcore_exceptions
    yield
  File "/root/.venv/lib/python3.12/site-packages/httpx/_transports/default.py", line 394, in handle_async_request
    resp = await self._pool.handle_async_request(req)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/root/.venv/lib/python3.12/site-packages/httpcore/_async/connection_pool.py", line 256, in handle_async_request
    raise exc from None
  File "/root/.venv/lib/python3.12/site-packages/httpcore/_async/connection_pool.py", line 236, in handle_async_request
    response = await connection.handle_async_request(
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/root/.venv/lib/python3.12/site-packages/httpcore/_async/connection.py", line 101, in handle_async_request
    raise exc
  File "/root/.venv/lib/python3.12/site-packages/httpcore/_async/connection.py", line 78, in handle_async_request
    stream = await self._connect(request)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/root/.venv/lib/python3.12/site-packages/httpcore/_async/connection.py", line 124, in _connect
    stream = await self._network_backend.connect_tcp(**kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/root/.venv/lib/python3.12/site-packages/httpcore/_backends/auto.py", line 31, in connect_tcp
    return await self._backend.connect_tcp(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/root/.venv/lib/python3.12/site-packages/httpcore/_backends/anyio.py", line 113, in connect_tcp
    with map_exceptions(exc_map):
         ^^^^^^^^^^^^^^^^^^^^^^^
  File "/root/.local/share/uv/python/cpython-3.12.13-linux-x86_64-gnu/lib/python3.12/contextlib.py", line 158, in __exit__
    self.gen.throw(value)
  File "/root/.venv/lib/python3.12/site-packages/httpcore/_exceptions.py", line 14, in map_exceptions
    raise to_exc(exc) from exc
httpcore.ConnectTimeout

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/root/.venv/lib/python3.12/site-packages/flyte/_internal/controllers/remote/_core.py", line 465, in _bg_run
    await self._bg_process(action)
  File "/root/.venv/lib/python3.12/site-packages/flyte/_internal/controllers/remote/_core.py", line 437, in _bg_process
    await self._bg_launch(action)
  File "/root/.venv/lib/python3.12/site-packages/flyte/_internal/controllers/remote/_core.py", line 381, in _bg_launch
    await self._actions_service.Enqueue(
  File "/root/.venv/lib/python3.12/site-packages/grpc/aio/_interceptor.py", line 471, in __await__
    call = yield from self._interceptors_task.__await__()
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/root/.venv/lib/python3.12/site-packages/grpc/aio/_interceptor.py", line 701, in _invoke
    return await _run_interceptor(
           ^^^^^^^^^^^^^^^^^^^^^^^
  File "/root/.venv/lib/python3.12/site-packages/grpc/aio/_interceptor.py", line 675, in _run_interceptor
    call_or_response = await interceptors[0].intercept_unary_unary(
                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/root/.venv/lib/python3.12/site-packages/flyte/remote/_client/auth/_grpc_utils/default_metadata_interceptor.py", line 103, in intercept_unary_unary
    return await (await continuation(updated_call_details, request))
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/root/.venv/lib/python3.12/site-packages/grpc/aio/_interceptor.py", line 675, in _run_interceptor
    call_or_response = await interceptors[0].intercept_unary_unary(
                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/root/.venv/lib/python3.12/site-packages/flyte/remote/_client/auth/_grpc_utils/auth_interceptor.py", line 79, in intercept_unary_unary
    await self.authenticator.refresh_credentials(creds_id=creds_id)
  File "/root/.venv/lib/python3.12/site-packages/flyte/remote/_client/auth/_authenticators/base.py", line 182, in refresh_credentials
    self._creds = await self._do_refresh_credentials()
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/root/.venv/lib/python3.12/site-packages/flyte/remote/_client/auth/_authenticators/client_credentials.py", line 60, in _do_refresh_credentials
    token, refresh_token, expires_in = await token_client.get_token(
                                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/root/.venv/lib/python3.12/site-packages/flyte/remote/_client/auth/_token_client.py", line 152, in get_token
    response = await http_session.post(token_endpoint, data=body, headers=headers)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/root/.venv/lib/python3.12/site-packages/httpx/_client.py", line 1859, in post
    return await self.request(
           ^^^^^^^^^^^^^^^^^^^
  File "/root/.venv/lib/python3.12/site-packages/httpx/_client.py", line 1540, in request
    return await self.send(request, auth=auth, follow_redirects=follow_redirects)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/root/.venv/lib/python3.12/site-packages/httpx/_client.py", line 1629, in send
    response = await self._send_handling_auth(
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/root/.venv/lib/python3.12/site-packages/httpx/_client.py", line 1657, in _send_handling_auth
    response = await self._send_handling_redirects(
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/root/.venv/lib/python3.12/site-packages/httpx/_client.py", line 1694, in _send_handling_redirects
    response = await self._send_single_request(request)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/root/.venv/lib/python3.12/site-packages/httpx/_client.py", line 1730, in _send_single_request
    response = await transport.handle_async_request(request)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/root/.venv/lib/python3.12/site-packages/httpx/_transports/default.py", line 393, in handle_async_request
    with map_httpcore_exceptions():
         ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/root/.local/share/uv/python/cpython-3.12.13-linux-x86_64-gnu/lib/python3.12/contextlib.py", line 158, in __exit__
    self.gen.throw(value)
  File "/root/.venv/lib/python3.12/site-packages/httpx/_transports/default.py", line 118, in map_httpcore_exceptions
    raise mapped_exc(message) from exc
httpx.ConnectTimeout
Root cause The retry path in
_bg_run
only catches
flyte.errors.SlowDownError
. Everything else falls through to the outer
except Exception
, which marks the action terminally failed. The exception classifier inside
_bg_launch
only handles
connectrpc.errors.ConnectError
— so a bare
httpx.ConnectTimeout
(or any other
httpx.TransportError
) raised from the auth interceptor duri… flyteorg/flyte-sdk