gifted-house-14547
01/12/2023, 4:20 AM"Too Many Requests. Exceeded Rate Limit"
Errors in flyte.
Here is the specific error codes from flyte admin below.
{
"json": {
"exec_id": "ff124e27fd7af005d000",
"node": "n0",
"src": "task_execution_manager.go:219"
},
"level": "debug",
"msg": "Successfully recorded task execution event [task_id:<resource_type:TASK project:\"raindrops\" domain:\"development\" name:\"raindrops.flyte.fred.load_fred\" version:\"c0e25bcbd3a23687a1c69330499dc8c0f91d3678\" > parent_node_execution_id:<node_id:\"n0\" execution_id:<project:\"raindrops\" domain:\"development\" name:\"ff124e27fd7af005d000\" > > retry_attempt:1 phase:FAILED producer_id:\"propeller\" logs:<uri:\"<https://logs.qraft.ai/app/discover#/?_g=(time:(from:now-1w,to:now))>&_a=(columns:!(log),filters:!((query:(match_phrase:(kubernetes.namespace_name:flyte-development))),(query:(match_phrase:(kubernetes.pod_name:ff124e27fd7af005d000-n0-1)))),sort:!(!('@timestamp',asc)))\" name:\"Kubernetes Logs (User)\" message_format:JSON > occurred_at:<seconds:1673475022 nanos:900572625 > input_uri:\"<s3://my-s3-bucket/metadata/propeller/raindrops-development-ff124e27fd7af005d000/n0/data/inputs.pb>\" error:<code:\"USER:Unknown\" message:\"Traceback (most recent call last):\\n\\n File \\\"/usr/local/lib/python3.9/site-packages/flytekit/exceptions/scopes.py\\\", line 203, in user_entry_point\\n return wrapped(*args, **kwargs)\\n File \\\"/root/raindrops/flyte/fred.py\\\", line 12, in load_fred\\n load_fred(from_last_date)\\n File \\\"/root/raindrops/source/fred/fred.py\\\", line 135, in load_fred\\n df = api.get_series_all_release(identifier, realtime_start)\\n File \\\"/root/raindrops/source/fred/fred.py\\\", line 54, in get_series_all_release\\n vintage_dates = pd.to_datetime(self._get_series_vintage_dates(series_id))\\n File \\\"/root/raindrops/util.py\\\", line 18, in wrapped\\n return on_exception(expo, Exception, max_tries=5, on_backoff=backoff)(func)(*args, **kwargs)\\n File \\\"/usr/local/lib/python3.9/site-packages/backoff/_sync.py\\\", line 105, in retry\\n ret = target(*args, **kwargs)\\n File \\\"/usr/local/lib/python3.9/site-packages/fredapi/fred.py\\\", line 274, in get_series_vintage_dates\\n root = self.__fetch_data(url)\\n File \\\"/usr/local/lib/python3.9/site-packages/fredapi/fred.py\\\", line 68, in __fetch_data\\n raise ValueError(root.get('message'))\\n\\nMessage:\\n\\n Too Many Requests. Exceeded Rate Limit\\n\\nUser error.\" kind:USER > task_type:\"python-task\" metadata:<generated_name:\"ff124e27fd7af005d000-n0-1\" plugin_identifier:\"container\" > event_version:1 ]",
"ts": "2023-01-11T22:10:22Z"
}
When I execute flyte codes in a remote server, I have never called the API, "fred" above more than 1, but it returns too many requests and the status of workflow become failture.
Moreover, we loaded fred api data into snowflake from the flyte execution.
I can check the data was loaded into snowflake 7 times even if the flyte execution failed.
(Flyte workflow executed only once.)
The logs starts with "PUT" and "SELECT" is the situation when Too Many Requests. Exceeded Rate Limit
happened.
Does anyone have experienced the same problem?gifted-house-14547
01/12/2023, 4:32 AMtall-lock-23197
gifted-house-14547
01/12/2023, 7:07 AMgifted-house-14547
01/12/2023, 7:08 AMtall-lock-23197
freezing-airport-6809
freezing-airport-6809
gifted-house-14547
01/12/2023, 10:08 PM