seunggs
09/16/2022, 5:33 PMflyte-executor
with the flyte-user-role
(which has full s3 access) attached as an annotation and running Flyte executions with this service account, but it’s giving me PutObject access denied error. This service account is in the project+domain namespace. What am I doing wrong?Yee
Yee
aws sts get-caller-identity
seunggs
09/16/2022, 9:24 PMAmazonS3FullAccess
policy attached to itseunggs
09/16/2022, 9:24 PMseunggs
09/16/2022, 9:34 PMseunggs
09/16/2022, 9:48 PMfatal error: An error occurred (403) when calling the HeadObject operation: Forbidden
seunggs
09/17/2022, 1:27 AMsleep infinity
to the args in the yaml and exec
into it to run aws sts get-caller-identity
? Let me know when you got a minute - thanks!Samhita Alla
Yee
seunggs
09/20/2022, 7:54 PMseunggs
09/20/2022, 7:54 PMYee
Yee
Yee
seunggs
09/20/2022, 8:03 PMseunggs
09/20/2022, 8:03 PMYee
Yee
Yee
Yee
seunggs
09/20/2022, 8:05 PMseunggs
09/20/2022, 8:06 PMapiVersion: v1
imagePullSecrets:
- name: gcr-json-key
kind: ServiceAccount
metadata:
annotations:
<http://eks.amazonaws.com/role-arn|eks.amazonaws.com/role-arn>: arn:aws:iam::xxx:role/flyte-user-role
labels:
<http://app.kubernetes.io/managed-by|app.kubernetes.io/managed-by>: pulumi
name: flyte-executor
namespace: shelly-robotics-bipedal-robot-development
resourceVersion: "57747250"
uid: 9db3e9da-cf32-4a78-8b06-81d83b66c611
secrets:
- name: flyte-executor-token-l6rkj
seunggs
09/20/2022, 8:06 PMYee
get pod <pod name> -o yaml
and grep for “iam”Yee
Yee
seunggs
09/20/2022, 8:35 PMget pod <pod name> -o yaml | grep 'iam'
returns thisseunggs
09/20/2022, 8:35 PMvalue: arn:aws:iam::xxx:role/flyte-user-role
name: aws-iam-token
- name: aws-iam-token
seunggs
09/20/2022, 8:35 PMYee
Yee
seunggs
09/20/2022, 8:38 PMaws sts assume-role …
locally, I get this error: An error occurred (AccessDenied) when calling the AssumeRole operation
Yee
Yee
Yee
seunggs
09/20/2022, 8:47 PMseunggs
09/20/2022, 8:47 PM{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "",
"Effect": "Allow",
"Principal": {
"Federated": "arn:aws:iam::xxx:oidc-provider/oidc.eks.us-west-1.amazonaws.com/id/2A6739B7813451087E3258C60BC37CF4"
},
"Action": "sts:AssumeRoleWithWebIdentity",
"Condition": {
"StringEquals": {
"<http://oidc.eks.us-west-1.amazonaws.com/id/2A6739B7813451087E3258C60BC37CF4:aud|oidc.eks.us-west-1.amazonaws.com/id/2A6739B7813451087E3258C60BC37CF4:aud>": "<http://sts.amazonaws.com|sts.amazonaws.com>"
}
}
},
{
"Sid": "",
"Effect": "Allow",
"Principal": {
"Service": "<http://ec2.amazonaws.com|ec2.amazonaws.com>"
},
"Action": "sts:AssumeRole"
}
]
}
seunggs
09/20/2022, 8:47 PMYee
seunggs
09/20/2022, 8:51 PMseunggs
09/20/2022, 8:52 PMYee
Yee
seunggs
09/20/2022, 8:54 PMsleep infinity
?seunggs
09/20/2022, 9:00 PMaws sts get-caller-identity
and here’s the response:seunggs
09/20/2022, 9:00 PM{
"UserId": "xxx:botocore-session-1663707573",
"Account": "xxx",
"Arn": "arn:aws:sts::xxx:assumed-role/flyte-user-role/botocore-session-1663707573"
}
Yee
seunggs
09/20/2022, 9:01 PMseunggs
09/20/2022, 9:01 PMYee
Yee
Yee
cat > abc
hello
^C
Yee
seunggs
09/20/2022, 9:02 PMseunggs
09/20/2022, 9:04 PMseunggs
09/20/2022, 9:04 PMaws s3 cp abc <s3://sidetrek-flyte-cluster-flyte-bucket/metadata/propeller/shelly-robotics-bipedal-robot-development-an6gvhl5dn8vr44nn9ds/n0/data/0/abc.txt>
Yee
seunggs
09/20/2022, 9:05 PMseunggs
09/20/2022, 9:05 PMseunggs
09/20/2022, 9:05 PMseunggs
09/20/2022, 9:06 PMSecurity Context
section - leaving IAM Role field emptyseunggs
09/20/2022, 9:06 PMseunggs
09/20/2022, 9:06 PMYee
seunggs
09/20/2022, 9:24 PMYee
seunggs
09/20/2022, 9:24 PMseunggs
09/20/2022, 9:25 PMseunggs
09/20/2022, 9:25 PM>>> import boto3
>>> boto3.client('sts').get_caller_identity().get('Account')
Yee
Yee
seunggs
09/20/2022, 9:26 PMseunggs
09/22/2022, 12:11 AMseunggs
09/22/2022, 12:11 AM{
"asctime": "2022-09-22 00:01:44,624",
"name": "flytekit",
"levelname": "ERROR",
"message": "Exception when trying to execute ['aws', '--endpoint-url', '<http://minio.flyte:9000>', 's3', 'cp', '--recursive', '--acl', 'bucket-owner-full-control', '/tmp/flyte-oz6o659c/sandbox/local_flytekit/engine_dir', '<s3://sidetrek-flyte-cluster-flyte-bucket/metadata/propeller/shelly-robotics-bipedal-robot-development-aq44hcpb7rdhwxpw22k9/n0/data/3>'], reason: Called process exited with error code: 1. Stderr dump:\n\nb'upload failed: ../tmp/flyte-oz6o659c/sandbox/local_flytekit/engine_dir/error.pb to <s3://sidetrek-flyte-cluster-flyte-bucket/metadata/propeller/shelly-robotics-bipedal-robot-development-aq44hcpb7rdhwxpw22k9/n0/data/3/error.pb> An error occurred (AccessDenied) when calling the PutObject operation: Access Denied.\\n'"
}
seunggs
09/22/2022, 12:19 AMSamhita Alla
seunggs
09/23/2022, 8:00 PM