Laura Lin
10/18/2022, 11:30 PMlytekit.exceptions.user.FlyteAssertion: Failed to put data from /tmp/flyte-gnaguqof/sandbox/local_flytekit/engine_dir to <s3://flyte-bucket/metadata/propeller/flytetester-development-a9xg68jktjbhd7nz7sc7/n0/data/0> (recursive=True).
Original exception: Called process exited with error code: 1. Stderr dump:
b'upload failed: ../../tmp/flyte-gnaguqof/sandbox/local_flytekit/engine_dir/error.pb to <s3://flyte-bucket/metadata/propeller/flytetester-development-a9xg68jktjbhd7nz7sc7/n0/data/0/error.pb> An error occurred (AccessDenied) when calling the PutObject operation: Access Denied\n'
the flyte-user-role has AmazonS3FullAccess, I verified that the failing pod has the env var AWS_ROLE_ARN
set to the flyte-user-role. And when I look inside the bucket, I can see that there's a <s3://flyfte-bucket/metadata/propeller/flytetester-development-a9xg68jktjbhd7nz7sc7/n0/data/inputs.pb>
So something is working correctly to put in objects but then fails? Not using minio either and docker image has awscli==1.25.94
flyte-user-role
has never been accessed.Ketan (kumare3)
Laura Lin
10/19/2022, 12:12 AMKetan (kumare3)
Laura Lin
10/19/2022, 12:13 AMKetan (kumare3)
pyflyte run --service-account
Laura Lin
10/19/2022, 12:14 AMflytectl register
?Ketan (kumare3)
Laura Lin
10/19/2022, 12:16 AMKetan (kumare3)
Laura Lin
10/19/2022, 12:21 AMKetan (kumare3)
Laura Lin
10/19/2022, 12:23 AMKetan (kumare3)
Hiromu Hota
01/06/2023, 6:17 PM$ kubectl annotate serviceaccount -n flytesnacks-development default <http://eks.amazonaws.com/role-arn=arn:aws:iam::xxxx:role/flyte-user-role|eks.amazonaws.com/role-arn=arn:aws:iam::xxxx:role/flyte-user-role>
Then a worker pod is spawn with AWS env vars like this:
$ kubectl describe pod f81f7df8fd03f4f84b45-n0-0 -n flytesnacks-development | grep AWS
AWS_STS_REGIONAL_ENDPOINTS: regional
AWS_DEFAULT_REGION: xxxxxx
AWS_REGION: xxxxxx
AWS_ROLE_ARN: arn:aws:iam::xxxxx:role/flyte-user-role
AWS_WEB_IDENTITY_TOKEN_FILE: /var/run/secrets/eks.amazonaws.com/serviceaccount/token
Ketan (kumare3)