@ancient-wolf-19325 [9]: Trace: Traceback (most recent call last): File "/opt/venv/lib/python3.10/site-packages/s3fs/core.py", line 729, in _lsdir async for c in self._iterdir( File "/opt/venv/lib/python3.10/site-packages/s3fs/core.py", line 779, in _iterdir async for i in it: File "/opt/venv/lib/python3.10/site-packages/aiobotocore/paginate.py", line 30, in anext response = await self._make_request(current_kwargs) File "/opt/venv/lib/python3.10/site-packages/aiobotocore/client.py", line 405, in _make_api_call raise error_class(parsed_response, operation_name) botocore.exceptions.ClientError: An error occurred (SlowDown) when calling the ListObjectsV2 operation (reached max retries: 4): Please reduce your request rate. The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/opt/venv/lib/python3.10/site-packages/flytekit/core/base_task.py", line 800, in dispatch_execute literals_map, native_outputs_as_map = run_sync( File "/opt/venv/lib/python3.10/site-packages/flytekit/utils/asyn.py", line 93, in run_sync return self._runner_map[name].run(coro) File "/opt/venv/lib/python3.10/site-packages/flytekit/utils/asyn.py", line 72, in run res = fut.result(None) File "/usr/local/lib/python3.10/concurrent/futures/_base.py", line 458, in result return self.__get_result() File "/usr/local/lib/python3.10/concurrent/futures/_base.py", line 403, in __get_result raise self._exception File "/opt/venv/lib/python3.10/site-packages/flytekit/core/base_task.py", line 655, in _output_to_literal_map raise e File "/opt/venv/lib/python3.10/site-packages/flytekit/core/type_engine.py", line 1397, in async_to_literal lv = await transformer.async_to_literal(ctx, python_val, python_type, expected) File "/opt/venv/lib/python3.10/site-packages/flytekit/core/type_engine.py", line 1677, in async_to_literal lit_list = await _run_coros_in_chunks(lit_list) File "/opt/venv/lib/python3.10/site-packages/fsspec/asyn.py", line 268, in _run_coros_in_chunks result, k = await done.pop() File "/opt/venv/lib/python3.10/site-packages/fsspec/asyn.py", line 245, in _run_coro return await asyncio.wait_for(coro, timeout=timeout), i File "/usr/local/lib/python3.10/asyncio/tasks.py", line 408, in wait_for return await fut File "/opt/venv/lib/python3.10/site-packages/flytekit/core/type_engine.py", line 1397, in async_to_literal lv = await transformer.async_to_literal(ctx, python_val, python_type, expected) File "/opt/venv/lib/python3.10/site-packages/flytekit/core/type_engine.py", line 2150, in async_to_literal await _run_coros_in_chunks([c for c in lit_map.values()]) File "/opt/venv/lib/python3.10/site-packages/fsspec/asyn.py", line 268, in _run_coros_in_chunks result, k = await done.pop() File "/opt/venv/lib/python3.10/site-packages/fsspec/asyn.py", line 245, in _run_coro return await asyncio.wait_for(coro, timeout=timeout), i File "/usr/local/lib/python3.10/asyncio/tasks.py", line 408, in wait_for return await fut File "/opt/venv/lib/python3.10/site-packages/flytekit/core/type_engine.py", line 1397, in async_to_literal lv = await transformer.async_to_literal(ctx, python_val, python_type, expected) File "/opt/venv/lib/python3.10/site-packages/flytekit/types/pickle/pickle.py", line 94, in async_to_literal remote_path = await FlytePickle.to_pickle(ctx, python_val) File "/opt/venv/lib/python3.10/site-packages/flytekit/types/pickle/pickle.py", line 50, in to_pickle return await ctx.file_access.async_put_raw_data(uri) File "/opt/venv/lib/python3.10/site-packages/flytekit/core/data_persistence.py", line 423, in async_put_raw_data r = await self._put(from_path, to_path, **kwargs) File "/opt/venv/lib/python3.10/site-packages/flytekit/core/data_persistence.py", line 357, in _put dst = await file_system._put(from_path, to_path, recursive=recursive, **kwargs) # pylint: disable=W0212 File "/opt/venv/lib/python3.10/site-packages/fsspec/asyn.py", line 565, in _put trailing_sep(rpath) or await self._isdir(rpath) File "/opt/venv/lib/python3.10/site-packages/s3fs/core.py", line 1498, in _isdir return bool(await self._lsdir(path)) File "/opt/venv/lib/python3.10/site-packages/s3fs/core.py", line 742, in _lsdir raise translate_boto_error(e) OSError: [Errno 16] Please reduce your request rate. Message: OSError: [Errno 16] Please reduce your request rate.

It seems likely that the error you're encountering is related to the rate limit being exceeded when making requests to S3. The

SlowDown

error from the

ListObjectsV2

operation indicates that the request rate is too high, and AWS is asking to reduce it. This is a common issue when there are too many requests being made in a short period of time. To address this, you might consider implementing retries with exponential backoff to handle the rate limiting gracefully. Additionally, reviewing the number of concurrent requests being made and optimizing them could help reduce the frequency of this error. Would you like more details on how to implement these changes in your Flyte project? Sources: - Flyte Community Discussion - Flyte Community Discussion 👍 0 button 👎 0 button 💡 Hint: Mention @runllm in the thread for followups.