Hello, Flyte Team! I would like to ask *`"Too Man...
# flyte-support
g
Hello, Flyte Team! I would like to ask
"Too Many Requests.  Exceeded Rate Limit"
Errors in flyte. Here is the specific error codes from flyte admin below.
Copy code
{
  "json": {
    "exec_id": "ff124e27fd7af005d000",
    "node": "n0",
    "src": "task_execution_manager.go:219"
  },
  "level": "debug",
  "msg": "Successfully recorded task execution event [task_id:<resource_type:TASK project:\"raindrops\" domain:\"development\" name:\"raindrops.flyte.fred.load_fred\" version:\"c0e25bcbd3a23687a1c69330499dc8c0f91d3678\" > parent_node_execution_id:<node_id:\"n0\" execution_id:<project:\"raindrops\" domain:\"development\" name:\"ff124e27fd7af005d000\" > > retry_attempt:1 phase:FAILED producer_id:\"propeller\" logs:<uri:\"<https://logs.qraft.ai/app/discover#/?_g=(time:(from:now-1w,to:now))>&_a=(columns:!(log),filters:!((query:(match_phrase:(kubernetes.namespace_name:flyte-development))),(query:(match_phrase:(kubernetes.pod_name:ff124e27fd7af005d000-n0-1)))),sort:!(!('@timestamp',asc)))\" name:\"Kubernetes Logs (User)\" message_format:JSON > occurred_at:<seconds:1673475022 nanos:900572625 > input_uri:\"<s3://my-s3-bucket/metadata/propeller/raindrops-development-ff124e27fd7af005d000/n0/data/inputs.pb>\" error:<code:\"USER:Unknown\" message:\"Traceback (most recent call last):\\n\\n      File \\\"/usr/local/lib/python3.9/site-packages/flytekit/exceptions/scopes.py\\\", line 203, in user_entry_point\\n        return wrapped(*args, **kwargs)\\n      File \\\"/root/raindrops/flyte/fred.py\\\", line 12, in load_fred\\n        load_fred(from_last_date)\\n      File \\\"/root/raindrops/source/fred/fred.py\\\", line 135, in load_fred\\n        df = api.get_series_all_release(identifier, realtime_start)\\n      File \\\"/root/raindrops/source/fred/fred.py\\\", line 54, in get_series_all_release\\n        vintage_dates = pd.to_datetime(self._get_series_vintage_dates(series_id))\\n      File \\\"/root/raindrops/util.py\\\", line 18, in wrapped\\n        return on_exception(expo, Exception, max_tries=5, on_backoff=backoff)(func)(*args, **kwargs)\\n      File \\\"/usr/local/lib/python3.9/site-packages/backoff/_sync.py\\\", line 105, in retry\\n        ret = target(*args, **kwargs)\\n      File \\\"/usr/local/lib/python3.9/site-packages/fredapi/fred.py\\\", line 274, in get_series_vintage_dates\\n        root = self.__fetch_data(url)\\n      File \\\"/usr/local/lib/python3.9/site-packages/fredapi/fred.py\\\", line 68, in __fetch_data\\n        raise ValueError(root.get('message'))\\n\\nMessage:\\n\\n    Too Many Requests.  Exceeded Rate Limit\\n\\nUser error.\" kind:USER > task_type:\"python-task\" metadata:<generated_name:\"ff124e27fd7af005d000-n0-1\" plugin_identifier:\"container\" > event_version:1 ]",
  "ts": "2023-01-11T22:10:22Z"
}
When I execute flyte codes in a remote server, I have never called the API, "fred" above more than 1, but it returns too many requests and the status of workflow become failture. Moreover, we loaded fred api data into snowflake from the flyte execution. I can check the data was loaded into snowflake 7 times even if the flyte execution failed. (Flyte workflow executed only once.) The logs starts with "PUT" and "SELECT" is the situation when
Too Many Requests.  Exceeded Rate Limit
happened. Does anyone have experienced the same problem?
Additionally, there are no logs why requests go multiple times to snowflake. There were just loaded data in snowflake
t
Does the code work locally?
g
@tall-lock-23197 I solved this issue! there wer codes error ;( Thank you 🙂
Is there any solution to debug flyte codes?
t
Um, it's best to run the code locally first to catch user-introduced bugs. Else, to debug system bugs, it got to be going through the logs.
❤️ 1
f
Hmm, is this s3 that’s throttling
This can happen, if you have too many tasks writing to the same key
g
@freezing-airport-6809 I think I should check s3's throttling. My team treats tick data (time series), so that may cause this issue
161 Views