Robin Eklund
11/16/2022, 4:14 PMflytekit.exceptions.scopes.FlyteScopedUserException: An error occurred (ExpiredToken) when calling the AssumeRole operation: The security token included in the request is expired
and after a while it also prints this error:
Called process exited with error code: 1. Stderr dump:\n\nb'upload failed: ../../tmp/flyte-kvw3xxto/sandbox/local_flytekit/engine_dir/error.pb to s3://<s3_bucket>/metadata/propeller/<project_name>-development-<execution_id>/n1/data/0/error.pb An error occurred (AccessDenied) when calling the PutObject operation: Access Denied
I trigger this workflow with a Launch Plan which specify a specific service account (not default). It was created like this:
security_context = SecurityContext(
run_as=Identity(
iam_role=None,
k8s_service_account=f"my-aws-role",
),
)
LaunchPlan.get_or_create(
name="my_lp",
workflow=my_wf,
security_context=security_context,
)
Anyone who faced the same error before?Samhita Alla
Prafulla Mahindrakar
11/17/2022, 6:05 AMkubectl get sa -n /<project_name>-development my-aws-role -o yaml
Robin Eklund
11/17/2022, 6:54 AMapiVersion: v1
automountServiceAccountToken: true
kind: ServiceAccount
metadata:
annotations:
<http://eks.amazonaws.com/role-arn|eks.amazonaws.com/role-arn>: arn:aws:iam::<aws_account_id>:role/eks/MyAwsRole
creationTimestamp: "2022-10-28T11:24:50Z"
managedFields:
- apiVersion: v1
fieldsType: FieldsV1
fieldsV1:
f:automountServiceAccountToken: {}
f:metadata:
f:annotations:
.: {}
f:<http://eks.amazonaws.com/role-arn|eks.amazonaws.com/role-arn>: {}
manager: HashiCorp
operation: Update
time: "2022-10-28T11:24:50Z"
- apiVersion: v1
fieldsType: FieldsV1
fieldsV1:
f:secrets:
.: {}
k:{"name":"my-aws-role-token-pvz2l"}: {}
manager: kube-controller-manager
operation: Update
time: "2022-10-28T11:24:50Z"
name: my-aws-role
namespace: <project_name>-development
resourceVersion: "20306065"
uid: <UUID>
secrets:
- name: my-flyte-role-token-pvz2l
Does it look OK for you, or what should i verify here?
I think the error about the put request comes due to the session token expires?s3:GetBucketLocation
s3:GetObject
s3:ListBucket
s3:ListBucketMultipartUploads
s3:ListMultipartUploadParts
s3:AbortMultipartUpload
s3:PutObject
s3:DeleteObject
s3:ListAllMyBuckets
Prafulla Mahindrakar
11/17/2022, 7:12 AMkubectl get pod -n <project_name>-development <execuitid>-* -o yaml
and check serviceAccount field.
Or else we have dig more into why there a reduced session token being given from STS.Robin Eklund
11/17/2022, 7:15 AMaws sts get-caller-identity
inside a task - and the right role is being usedAWS_ROLE_ARN
value?Prafulla Mahindrakar
11/17/2022, 7:25 AMs3://<s3_bucket>/metadata/propeller/
and only for >5 min it fails to uploadRobin Eklund
11/17/2022, 7:39 AMPrafulla Mahindrakar
11/17/2022, 7:45 AMRobin Eklund
11/17/2022, 8:52 AMAndrew Korzhuev
11/17/2022, 11:48 AMPrafulla Mahindrakar
11/17/2022, 12:21 PM