#1760 [BUG] FlytePropeller failed to get permissions to S3 on EKS, error is silent in the background Issue created by kumare3 Describe the bug In case where flytepropeller on EKS, is unable to get permissions to write to S3, it keeps on trying, upto system retries, without visibility in the UI. The workflow remains in

UNKNOWN

status. Ideal, would be if the workflow gets information about the system error being experienced.

{"json":{"exec_id":"uaben0erib","ns":"flytesnacks-development","res_ver":"52746460","routine":"worker-8","src":"executor.go:355","wf":"flytesnacks:development:core.control_flow.high_cpu_wf.single_integer_map_task"},"level":"info","msg":"Handling Workflow [uaben0erib], id: [project:\"flytesnacks\" domain:\"development\" name:\"uaben0erib\" ], p [Ready]","ts":"2021-10-29T04:17:44Z"}
{"json":{"exec_id":"uaben0erib","ns":"flytesnacks-development","res_ver":"52746460","routine":"worker-8","src":"executor.go:122","wf":"flytesnacks:development:core.control_flow.high_cpu_wf.single_integer_map_task"},"level":"info","msg":"Setting the MetadataDir for StartNode [s3://..../metadata/propeller/flytesnacks-development-uaben0erib/start-node/data]","ts":"2021-10-29T04:17:44Z"}
{"json":{"exec_id":"uaben0erib","node":"start-node","ns":"flytesnacks-development","res_ver":"52746460","routine":"worker-8","src":"protobuf_store.go:77","wf":"flytesnacks:development:core.control_flow.high_cpu_wf.single_integer_map_task"},"level":"error","msg":"Failed to write to the raw store [s3://.../metadata/propeller/flytesnacks-development-uaben0erib/start-node/data/0/outputs.pb] Error: Failed to write data [28b] to path [metadata/propeller/flytesnacks-development-uaben0erib/start-node/data/0/outputs.pb].: PutObject, putting object: WebIdentityErr: failed fetching WebIdentity token: \ncaused by: WebIdentityErr: unable to read file at /var/run/secrets/eks.amazonaws.com/serviceaccount/token\ncaused by: open /var/run/secrets/eks.amazonaws.com/serviceaccount/token: permission denied","ts":"2021-10-29T04:17:44Z"}

Expected behavior The UI should show the error that is experienced by the system Additional context to reproduce No response Screenshots No response Are you sure this issue hasn't been raised already? ☑︎ Yes Have you read the Code of Conduct? ☑︎ Yes flyteorg/flyte