Hi all, Some retries request isn't honored by Flyte. • Task signature:

@task(
    secret_requests=[
        webit_token,
    ],
    requests=Resources(mem="1G"),
    limits=Resources(mem="4G"),
    timeout=QUERY_TASK_TIMEOUT_SECONDS,
    retries=4,
)
def query_task_async(
    inputs_df: DataFrame
) -> list[WebPageLocation]:

• Here is the error raised by our code. This is a partial stack trace of our code:

...
      File "/home/.local/lib/python3.11/site-packages/nimble/modules/auth/client.py", line 120, in verify_token
        return self.login(
               ^^^^^^^^^^^
      File "/home/.local/lib/python3.11/site-packages/tenacity/__init__.py", line 336, in wrapped_f
        return copy(f, *args, **kw)
               ^^^^^^^^^^^^^^^^^^^^
      File "/home/.local/lib/python3.11/site-packages/tenacity/__init__.py", line 475, in __call__
        do = self.iter(retry_state=retry_state)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
      File "/home/.local/lib/python3.11/site-packages/tenacity/__init__.py", line 376, in iter
        result = action(retry_state)
                 ^^^^^^^^^^^^^^^^^^^
      File "/home/.local/lib/python3.11/site-packages/tenacity/__init__.py", line 419, in exc_check
        raise retry_exc from fut.exception()
    tenacity.RetryError: RetryError[<Future at 0x7ff729397c10 state=finished raised ReadTimeout>]

• Task metadata:

{
  "tags": {},
  "cacheIgnoreInputVars": [],
  "runtime": {
    "type": "FLYTE_SDK",
    "version": "1.14.2",
    "flavor": "python"
  },
  "timeout": {
    "seconds": {
      "low": 900,
      "high": 0,
      "unsigned": false
    }
  },
  "retries": {
    "retries": 4
  }
}

• See the image attached, that shows a single attempt Thanks for your help!

In my experience Flyte retries are cooperative and you need to raise a Flyte specific retry error: https://docs.flyte.org/en/latest/user_guide/flyte_fundamentals/optimizing_tasks.html#configuring-retries