echoing-carpenter-92090
06/14/2023, 6:23 PMpyflyte
executions are failing.
Failed with Exception Code: SYSTEM:Unknown
RPC Failed, with Status: StatusCode.INTERNAL
details: failed to create a signed url. Error: WebIdentityErr: failed to retrieve credentials
caused by: AccessDenied: Not authorized to perform sts:AssumeRoleWithWebIdentity
status code: 403, request id: 5efc9c88-fdcb-42ab-bea8-8de7a79101e9
Debug string UNKNOWN:Error received from peer {grpc_message:"failed to create a signed url. Error: WebIdentityErr: failed to retrieve credentials\ncaused by: AccessDenied: Not authorized to perform sts:AssumeRoleWithWebIdentity\n\tstatus code: 403, request id: 5efc9c88-fdcb-42ab-bea8-8de7a79101e9", grpc_status:13, created_time:"2023-06-14T11:16:05.730571-07:00"}
average-finland-92144
06/14/2023, 6:30 PMaverage-finland-92144
06/14/2023, 6:30 PMaws iam get-role --role-name flyte-system-role --query Role.AssumeRolePolicyDocument
average-finland-92144
06/14/2023, 6:31 PMkubectl describe sa flyte-backend-flyte-binary -n flyte
echoing-carpenter-92090
06/14/2023, 6:31 PMAn error occurred (NoSuchEntity) when calling the GetRole operation: The role with name flyte-system-role cannot be found.
echoing-carpenter-92090
06/14/2023, 6:31 PMName: flyte-backend-flyte-binary
Namespace: flyte
Labels: <http://app.kubernetes.io/instance=flyte-backend|app.kubernetes.io/instance=flyte-backend>
<http://app.kubernetes.io/managed-by=Helm|app.kubernetes.io/managed-by=Helm>
<http://app.kubernetes.io/name=flyte-binary|app.kubernetes.io/name=flyte-binary>
<http://app.kubernetes.io/version=1.16.0|app.kubernetes.io/version=1.16.0>
<http://helm.sh/chart=flyte-binary-v1.6.2|helm.sh/chart=flyte-binary-v1.6.2>
Annotations: <http://eks.amazonaws.com/role-arn|eks.amazonaws.com/role-arn>: arn:aws:iam::xxxxxx:role/eksctl-flyte-cluster-cluster-ServiceRole-1M4CS3AC5LGV8
<http://meta.helm.sh/release-name|meta.helm.sh/release-name>: flyte-backend
<http://meta.helm.sh/release-namespace|meta.helm.sh/release-namespace>: flyte
Image pull secrets: <none>
Mountable secrets: <none>
Tokens: <none>
Events: <none>
average-finland-92144
06/14/2023, 6:33 PMaws iam get-role --role-name eksctl-flyte-cluster-cluster-ServiceRole-1M4CS3AC5LGV8 --query Role.AssumeRolePolicyDocument
echoing-carpenter-92090
06/14/2023, 6:33 PM{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Principal": {
"Service": "<http://eks.amazonaws.com|eks.amazonaws.com>"
},
"Action": "sts:AssumeRole"
}
]
}
average-finland-92144
06/14/2023, 6:34 PMaverage-finland-92144
06/14/2023, 6:35 PMechoing-carpenter-92090
06/14/2023, 6:37 PM2023-06-14 11:36:30 [ℹ] IAM Open ID Connect provider is already associated with cluster "flyte-cluster" in "us-west-2"
echoing-carpenter-92090
06/14/2023, 6:38 PMaverage-finland-92144
06/14/2023, 6:45 PMechoing-carpenter-92090
06/14/2023, 6:52 PMaverage-finland-92144
06/14/2023, 6:57 PMechoing-carpenter-92090
06/14/2023, 7:00 PMserviceAccount:
create: true
annotations:
<http://eks.amazonaws.com/role-arn|eks.amazonaws.com/role-arn>: "arn:aws:iam::xxxxxxx:role/flyte-system-role"
average-finland-92144
06/14/2023, 7:06 PMaverage-finland-92144
06/14/2023, 7:06 PMaws iam get-role --role-name <your-role-name> --query Role.AssumeRolePolicyDocument
echoing-carpenter-92090
06/14/2023, 7:07 PM{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Principal": {
"Federated": "arn:aws:iam::xxxxxxxx:oidc-provider/oidc.eks.us-west-2.amazonaws.com/id/<hash>"
},
"Action": "sts:AssumeRoleWithWebIdentity",
"Condition": {
"StringEquals": {
"<http://oidc.eks.us-west-2.amazonaws.com/id/<hash>:aud|oidc.eks.us-west-2.amazonaws.com/id/<hash>:aud>": "<http://sts.amazonaws.com|sts.amazonaws.com>",
"<http://oidc.eks.us-west-2.amazonaws.com/id/<hash>:sub|oidc.eks.us-west-2.amazonaws.com/id/<hash>:sub>": "system:serviceaccount:flyte:flyte-backend-flyte-binary"
}
}
}
]
}
echoing-carpenter-92090
06/14/2023, 7:09 PMaverage-finland-92144
06/14/2023, 7:14 PMauth
right?echoing-carpenter-92090
06/14/2023, 7:15 PMaverage-finland-92144
06/14/2023, 7:16 PM$HOME/.flyte/config.yaml
echoing-carpenter-92090
06/14/2023, 7:18 PMadmin:
# For GRPC endpoints you might want to use dns:///flyte.myexample.com
endpoint: dns:///<domain-name>.com
authType: Pkce
insecure: false
logger:
show-source: true
level: 0
echoing-carpenter-92090
06/14/2023, 7:19 PMechoing-carpenter-92090
06/14/2023, 7:35 PMechoing-carpenter-92090
06/14/2023, 7:38 PMaverage-finland-92144
06/14/2023, 7:55 PMastonishing-boots-31421
06/15/2023, 4:17 PMechoing-carpenter-92090
06/15/2023, 4:27 PMcustomData.<env>.defaultIamRole
field from the values.yaml and the executions were still working fine. I needed to uncomment those fields and redeploy.
cluster_resources:
customData:
# - production:
# - defaultIamRole:
# value: arn:aws:iam::<AWS-ACCOUNT-ID>:role/flyte-system-role
# - staging:
# - defaultIamRole:
# value: arn:aws:iam::<AWS-ACCOUNT-ID>:role/flyte-system-role
# - development:
# - defaultIamRole:
# value: arn:aws:iam::<AWS-ACCOUNT-ID>:role/flyte-system-role
echoing-carpenter-92090
06/15/2023, 4:28 PMastonishing-boots-31421
06/15/2023, 4:36 PMvalues.yaml
?echoing-carpenter-92090
06/15/2023, 4:43 PMinline.cluster_resources