salmon-refrigerator-32115
10/16/2023, 7:06 PMWorkflow[customer-ds:development:<http://ml_project_1.test_env_vars.wf|ml_project_1.test_env_vars.wf>] failed. RuntimeExecutionError: max number of system retry attempts [51/50] exhausted. Last known status message: failed at Node[start-node]. CausedByError: Failed to store workflow inputs (as start node), caused by: Failed to write data [0b] to path [metadata/propeller/customer-ds-development-f829d772e914540dfa24/start-node/data/0/outputs.pb].: PutObject, putting object: AccessDenied: Access Denied
status code: 403, request id: TBZ9Z342JT9MVTJP, host id: OCexqZHh7PUeIh7kQWMjmZ3gAcE98R1OqtpGim/Awc66m5UgH+JdnlMmxmgRi70ZbRfCKITm7RG+aXq/AKy5QQ==
From propeller logs:
│ {"json":{"exec_id":"aggb47m8lth9qgr8m4vr","node":"start-node","ns":"customer-ds-development","res_ver":"991597274","routine":"worker-28","wf":"customer-ds:development:<http://ml_project_1.test_env_vars.wf|ml_project_1.test_env_vars.wf>"},"level":"error","msg":"Failed to wr │
│ {"json":{"exec_id":"aggb47m8lth9qgr8m4vr","node":"start-node","ns":"customer-ds-development","res_ver":"991597274","routine":"worker-28","wf":"customer-ds:development:<http://ml_project_1.test_env_vars.wf|ml_project_1.test_env_vars.wf>"},"level":"error","msg":"Failed to wr │
│ {"json":{"exec_id":"aggb47m8lth9qgr8m4vr","ns":"customer-ds-development","res_ver":"991597274","routine":"worker-28","wf":"customer-ds:development:<http://ml_project_1.test_env_vars.wf|ml_project_1.test_env_vars.wf>"},"level":"error","msg":"Error when trying to reconcile w │
│ E1016 18:56:14.532983 1 workers.go:102] error syncing 'customer-ds-development/aggb47m8lth9qgr8m4vr': failed at Node[start-node]. CausedByError: Failed to store workflow inputs (as start node), caused by: Failed to write data [0 │
│ status code: 403, request id: W93BW8RFPXS6Y2CF, host id: v0uNuIeisj96rMbYNFL5W1KpIikl6BixP2rEyDOqTHLx0BFmSSt9bAhJewmTYSNPUMuWyNlypS8= │
│ {"json":{"exec_id":"aggb47m8lth9qgr8m4vr","node":"start-node","ns":"customer-ds-development","res_ver":"991597769","routine":"worker-32","wf":"customer-ds:development:<http://ml_project_1.test_env_vars.wf|ml_project_1.test_env_vars.wf>"},"level":"error","msg":"Failed to wr │
│ {"json":{"exec_id":"aggb47m8lth9qgr8m4vr","node":"start-node","ns":"customer-ds-development","res_ver":"991597769","routine":"worker-32","wf":"customer-ds:development:<http://ml_project_1.test_env_vars.wf|ml_project_1.test_env_vars.wf>"},"level":"error","msg":"Failed to wr │
│ {"json":{"exec_id":"aggb47m8lth9qgr8m4vr","ns":"customer-ds-development","res_ver":"991597769","routine":"worker-32","wf":"customer-ds:development:<http://ml_project_1.test_env_vars.wf|ml_project_1.test_env_vars.wf>"},"level":"error","msg":"Error when trying to reconcile w │
│ E1016 18:56:24.587487 1 workers.go:102] error syncing 'customer-ds-development/aggb47m8lth9qgr8m4vr': failed at Node[start-node]. CausedByError: Failed to store workflow inputs (as start node), caused by: Failed to write data [0 │
│ status code: 403, request id: 29EX1TAB8BQWV0W6, host id: T/Ifpg+KMsXYjyoxjbL4Y4n01aBJXr6JMqwUncTkvimyRbWJmZwO3CqNWNxXPzcdj527sqJSp2yVPz/mJ1d1TN+iJgxAum6O
Note that each host id is different.
When submitting the local workflow to remote via pyflyte run --remote, the flyte server is able to write to the same S3 bucket at
/metadata/customer-ds/development/f829d772e914540dfa24/
Why cannot the propeller write to /metadata/propeller/customer-ds-development-f829d772e914540dfa24/start-node/data/0/outputs.pb?
Could someone shed some light?
Thank you very much!tall-lock-23197
salmon-refrigerator-32115
11/07/2023, 5:12 PM