seunggs
09/16/2022, 5:33 PMflyte-executor
with the flyte-user-role
(which has full s3 access) attached as an annotation and running Flyte executions with this service account, but it’s giving me PutObject access denied error. This service account is in the project+domain namespace. What am I doing wrong?Yee
aws sts get-caller-identity
seunggs
09/16/2022, 9:24 PMAmazonS3FullAccess
policy attached to itfatal error: An error occurred (403) when calling the HeadObject operation: Forbidden
sleep infinity
to the args in the yaml and exec
into it to run aws sts get-caller-identity
? Let me know when you got a minute - thanks!Samhita Alla
Yee
seunggs
09/20/2022, 7:54 PMYee
seunggs
09/20/2022, 8:03 PMYee
seunggs
09/20/2022, 8:05 PMapiVersion: v1
imagePullSecrets:
- name: gcr-json-key
kind: ServiceAccount
metadata:
annotations:
<http://eks.amazonaws.com/role-arn|eks.amazonaws.com/role-arn>: arn:aws:iam::xxx:role/flyte-user-role
labels:
<http://app.kubernetes.io/managed-by|app.kubernetes.io/managed-by>: pulumi
name: flyte-executor
namespace: shelly-robotics-bipedal-robot-development
resourceVersion: "57747250"
uid: 9db3e9da-cf32-4a78-8b06-81d83b66c611
secrets:
- name: flyte-executor-token-l6rkj
Yee
get pod <pod name> -o yaml
and grep for “iam”seunggs
09/20/2022, 8:35 PMget pod <pod name> -o yaml | grep 'iam'
returns thisvalue: arn:aws:iam::xxx:role/flyte-user-role
name: aws-iam-token
- name: aws-iam-token
Yee
seunggs
09/20/2022, 8:38 PMaws sts assume-role …
locally, I get this error: An error occurred (AccessDenied) when calling the AssumeRole operation
Yee
seunggs
09/20/2022, 8:47 PM{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "",
"Effect": "Allow",
"Principal": {
"Federated": "arn:aws:iam::xxx:oidc-provider/oidc.eks.us-west-1.amazonaws.com/id/2A6739B7813451087E3258C60BC37CF4"
},
"Action": "sts:AssumeRoleWithWebIdentity",
"Condition": {
"StringEquals": {
"<http://oidc.eks.us-west-1.amazonaws.com/id/2A6739B7813451087E3258C60BC37CF4:aud|oidc.eks.us-west-1.amazonaws.com/id/2A6739B7813451087E3258C60BC37CF4:aud>": "<http://sts.amazonaws.com|sts.amazonaws.com>"
}
}
},
{
"Sid": "",
"Effect": "Allow",
"Principal": {
"Service": "<http://ec2.amazonaws.com|ec2.amazonaws.com>"
},
"Action": "sts:AssumeRole"
}
]
}
Yee
seunggs
09/20/2022, 8:51 PMYee
seunggs
09/20/2022, 8:54 PMsleep infinity
?aws sts get-caller-identity
and here’s the response:{
"UserId": "xxx:botocore-session-1663707573",
"Account": "xxx",
"Arn": "arn:aws:sts::xxx:assumed-role/flyte-user-role/botocore-session-1663707573"
}
Yee
seunggs
09/20/2022, 9:01 PMYee
cat > abc
hello
^C
seunggs
09/20/2022, 9:02 PMaws s3 cp abc <s3://sidetrek-flyte-cluster-flyte-bucket/metadata/propeller/shelly-robotics-bipedal-robot-development-an6gvhl5dn8vr44nn9ds/n0/data/0/abc.txt>
Yee
seunggs
09/20/2022, 9:05 PMSecurity Context
section - leaving IAM Role field emptyYee
seunggs
09/20/2022, 9:24 PMYee
seunggs
09/20/2022, 9:24 PMseunggs
09/20/2022, 9:25 PM>>> import boto3
>>> boto3.client('sts').get_caller_identity().get('Account')
Yee
seunggs
09/20/2022, 9:26 PM{
"asctime": "2022-09-22 00:01:44,624",
"name": "flytekit",
"levelname": "ERROR",
"message": "Exception when trying to execute ['aws', '--endpoint-url', '<http://minio.flyte:9000>', 's3', 'cp', '--recursive', '--acl', 'bucket-owner-full-control', '/tmp/flyte-oz6o659c/sandbox/local_flytekit/engine_dir', '<s3://sidetrek-flyte-cluster-flyte-bucket/metadata/propeller/shelly-robotics-bipedal-robot-development-aq44hcpb7rdhwxpw22k9/n0/data/3>'], reason: Called process exited with error code: 1. Stderr dump:\n\nb'upload failed: ../tmp/flyte-oz6o659c/sandbox/local_flytekit/engine_dir/error.pb to <s3://sidetrek-flyte-cluster-flyte-bucket/metadata/propeller/shelly-robotics-bipedal-robot-development-aq44hcpb7rdhwxpw22k9/n0/data/3/error.pb> An error occurred (AccessDenied) when calling the PutObject operation: Access Denied.\\n'"
}
Samhita Alla
seunggs
09/23/2022, 8:00 PM