sticky-angle-28419
09/16/2022, 5:33 PMflyte-executor
with the flyte-user-role
(which has full s3 access) attached as an annotation and running Flyte executions with this service account, but it’s giving me PutObject access denied error. This service account is in the project+domain namespace. What am I doing wrong?thankful-minister-83577
thankful-minister-83577
aws sts get-caller-identity
sticky-angle-28419
09/16/2022, 9:24 PMAmazonS3FullAccess
policy attached to itsticky-angle-28419
09/16/2022, 9:24 PMsticky-angle-28419
09/16/2022, 9:34 PMsticky-angle-28419
09/16/2022, 9:48 PMfatal error: An error occurred (403) when calling the HeadObject operation: Forbidden
sticky-angle-28419
09/17/2022, 1:27 AMsleep infinity
to the args in the yaml and exec
into it to run aws sts get-caller-identity
? Let me know when you got a minute - thanks!tall-lock-23197
thankful-minister-83577
sticky-angle-28419
09/20/2022, 7:54 PMsticky-angle-28419
09/20/2022, 7:54 PMthankful-minister-83577
thankful-minister-83577
thankful-minister-83577
sticky-angle-28419
09/20/2022, 8:03 PMsticky-angle-28419
09/20/2022, 8:03 PMthankful-minister-83577
thankful-minister-83577
thankful-minister-83577
thankful-minister-83577
sticky-angle-28419
09/20/2022, 8:05 PMsticky-angle-28419
09/20/2022, 8:06 PMapiVersion: v1
imagePullSecrets:
- name: gcr-json-key
kind: ServiceAccount
metadata:
annotations:
<http://eks.amazonaws.com/role-arn|eks.amazonaws.com/role-arn>: arn:aws:iam::xxx:role/flyte-user-role
labels:
<http://app.kubernetes.io/managed-by|app.kubernetes.io/managed-by>: pulumi
name: flyte-executor
namespace: shelly-robotics-bipedal-robot-development
resourceVersion: "57747250"
uid: 9db3e9da-cf32-4a78-8b06-81d83b66c611
secrets:
- name: flyte-executor-token-l6rkj
sticky-angle-28419
09/20/2022, 8:06 PMthankful-minister-83577
get pod <pod name> -o yaml
and grep for “iam”thankful-minister-83577
thankful-minister-83577
sticky-angle-28419
09/20/2022, 8:35 PMget pod <pod name> -o yaml | grep 'iam'
returns thissticky-angle-28419
09/20/2022, 8:35 PMvalue: arn:aws:iam::xxx:role/flyte-user-role
name: aws-iam-token
- name: aws-iam-token
sticky-angle-28419
09/20/2022, 8:35 PMthankful-minister-83577
thankful-minister-83577
sticky-angle-28419
09/20/2022, 8:38 PMaws sts assume-role …
locally, I get this error: An error occurred (AccessDenied) when calling the AssumeRole operation
thankful-minister-83577
thankful-minister-83577
thankful-minister-83577
sticky-angle-28419
09/20/2022, 8:47 PMsticky-angle-28419
09/20/2022, 8:47 PM{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "",
"Effect": "Allow",
"Principal": {
"Federated": "arn:aws:iam::xxx:oidc-provider/oidc.eks.us-west-1.amazonaws.com/id/2A6739B7813451087E3258C60BC37CF4"
},
"Action": "sts:AssumeRoleWithWebIdentity",
"Condition": {
"StringEquals": {
"<http://oidc.eks.us-west-1.amazonaws.com/id/2A6739B7813451087E3258C60BC37CF4:aud|oidc.eks.us-west-1.amazonaws.com/id/2A6739B7813451087E3258C60BC37CF4:aud>": "<http://sts.amazonaws.com|sts.amazonaws.com>"
}
}
},
{
"Sid": "",
"Effect": "Allow",
"Principal": {
"Service": "<http://ec2.amazonaws.com|ec2.amazonaws.com>"
},
"Action": "sts:AssumeRole"
}
]
}
sticky-angle-28419
09/20/2022, 8:47 PMthankful-minister-83577
sticky-angle-28419
09/20/2022, 8:51 PMsticky-angle-28419
09/20/2022, 8:52 PMthankful-minister-83577
thankful-minister-83577
sticky-angle-28419
09/20/2022, 8:54 PMsleep infinity
?sticky-angle-28419
09/20/2022, 9:00 PMaws sts get-caller-identity
and here’s the response:sticky-angle-28419
09/20/2022, 9:00 PM{
"UserId": "xxx:botocore-session-1663707573",
"Account": "xxx",
"Arn": "arn:aws:sts::xxx:assumed-role/flyte-user-role/botocore-session-1663707573"
}
thankful-minister-83577
sticky-angle-28419
09/20/2022, 9:01 PMsticky-angle-28419
09/20/2022, 9:01 PMthankful-minister-83577
thankful-minister-83577
thankful-minister-83577
cat > abc
hello
^C
thankful-minister-83577
sticky-angle-28419
09/20/2022, 9:02 PMsticky-angle-28419
09/20/2022, 9:04 PMsticky-angle-28419
09/20/2022, 9:04 PMaws s3 cp abc <s3://sidetrek-flyte-cluster-flyte-bucket/metadata/propeller/shelly-robotics-bipedal-robot-development-an6gvhl5dn8vr44nn9ds/n0/data/0/abc.txt>
thankful-minister-83577
sticky-angle-28419
09/20/2022, 9:05 PMsticky-angle-28419
09/20/2022, 9:05 PMsticky-angle-28419
09/20/2022, 9:05 PMsticky-angle-28419
09/20/2022, 9:06 PMSecurity Context
section - leaving IAM Role field emptysticky-angle-28419
09/20/2022, 9:06 PMsticky-angle-28419
09/20/2022, 9:06 PMthankful-minister-83577
sticky-angle-28419
09/20/2022, 9:24 PMthankful-minister-83577
sticky-angle-28419
09/20/2022, 9:24 PMthankful-minister-83577
sticky-angle-28419
09/20/2022, 9:25 PMsticky-angle-28419
09/20/2022, 9:25 PM>>> import boto3
>>> boto3.client('sts').get_caller_identity().get('Account')
thankful-minister-83577
thankful-minister-83577
sticky-angle-28419
09/20/2022, 9:26 PMsticky-angle-28419
09/22/2022, 12:11 AMsticky-angle-28419
09/22/2022, 12:11 AM{
"asctime": "2022-09-22 00:01:44,624",
"name": "flytekit",
"levelname": "ERROR",
"message": "Exception when trying to execute ['aws', '--endpoint-url', '<http://minio.flyte:9000>', 's3', 'cp', '--recursive', '--acl', 'bucket-owner-full-control', '/tmp/flyte-oz6o659c/sandbox/local_flytekit/engine_dir', '<s3://sidetrek-flyte-cluster-flyte-bucket/metadata/propeller/shelly-robotics-bipedal-robot-development-aq44hcpb7rdhwxpw22k9/n0/data/3>'], reason: Called process exited with error code: 1. Stderr dump:\n\nb'upload failed: ../tmp/flyte-oz6o659c/sandbox/local_flytekit/engine_dir/error.pb to <s3://sidetrek-flyte-cluster-flyte-bucket/metadata/propeller/shelly-robotics-bipedal-robot-development-aq44hcpb7rdhwxpw22k9/n0/data/3/error.pb> An error occurred (AccessDenied) when calling the PutObject operation: Access Denied.\\n'"
}
sticky-angle-28419
09/22/2022, 12:19 AMtall-lock-23197
sticky-angle-28419
09/23/2022, 8:00 PM