Hampus Rosvall
04/25/2022, 11:29 AMeks-flyte-user-rule
assume the IAM Role you provide in the launch plan, or is it assumed by some service account that gets created?
Environment:
...
AWS_ROLE_ARN: arn:aws:iam::<account_id>:role/eks-flyte-user-role
Prafulla Mahindrakar
04/25/2022, 12:36 PMHampus Rosvall
04/25/2022, 12:45 PMauth_role = AuthRole(assumable_iam_role="my:iam:role")
launch_plan.LaunchPlan.get_or_create(
workflow=wf,
name="your_lp_name_3",
auth_role=auth_role,
)
In the above example, is the IAM role assumed by the flyte-user-role
i.e., I need additional trust relationship, or is the flyte-user-role
overriden with my:iam:role
?Prafulla Mahindrakar
04/25/2022, 12:51 PMAdvanced Options
or you can provide this at project-domain level using flytectl if you are using the latest admin and flytectl (docs are in review for this https://github.com/flyteorg/flytectl/pull/316)my:iam:role
will override the flyte-user-roleHampus Rosvall
04/25/2022, 1:10 PMsecurity_context:
run_as:
k8s_service_account: default
Prafulla Mahindrakar
04/25/2022, 1:10 PMHampus Rosvall
04/25/2022, 1:31 PMPrafulla Mahindrakar
04/25/2022, 1:33 PMHampus Rosvall
04/25/2022, 1:35 PM~ flytectl get launchplan -p hackday -d development --latest -o yaml
- closure:
createdAt: "2022-04-25T09:23:55.415089Z"
expectedInputs: {}
expectedOutputs: {}
updatedAt: "2022-04-25T09:23:55.415089Z"
id:
domain: development
name: flyte.workflows.workflow.my_wf
project: hackday
resourceType: LAUNCH_PLAN
version: v0.0.5
spec:
annotations: {}
defaultInputs: {}
entityMetadata: {}
fixedInputs: {}
labels: {}
rawOutputDataConfig: {}
workflowId:
domain: development
name: flyte.workflows.workflow.my_wf
project: hackday
resourceType: WORKFLOW
version: v0.0.5
- closure:
createdAt: "2022-04-08T13:17:18.634988Z"
expectedInputs: {}
expectedOutputs: {}
updatedAt: "2022-04-08T13:17:18.634988Z"
id:
domain: development
name: flyte.workflows.workflow.my_wf
project: hackday
resourceType: LAUNCH_PLAN
version: v0.0.4
spec:
annotations: {}
authRole: {}
defaultInputs: {}
entityMetadata: {}
fixedInputs: {}
labels: {}
rawOutputDataConfig: {}
workflowId:
domain: development
name: flyte.workflows.workflow.my_wf
project: hackday
resourceType: WORKFLOW
version: v0.0.4
- closure:
createdAt: "2022-04-08T09:47:03.133154Z"
expectedInputs: {}
expectedOutputs: {}
updatedAt: "2022-04-08T09:47:03.133154Z"
id:
domain: development
name: flyte.workflows.workflow.my_wf
project: hackday
resourceType: LAUNCH_PLAN
version: v0.0.3
spec:
annotations: {}
authRole: {}
defaultInputs: {}
entityMetadata: {}
fixedInputs: {}
labels: {}
rawOutputDataConfig: {}
workflowId:
domain: development
name: flyte.workflows.workflow.my_wf
project: hackday
resourceType: WORKFLOW
version: v0.0.3
- closure:
createdAt: "2022-04-08T09:40:20.073883Z"
expectedInputs: {}
expectedOutputs: {}
updatedAt: "2022-04-08T09:40:20.073883Z"
id:
domain: development
name: flyte.workflows.workflow.my_wf
project: hackday
resourceType: LAUNCH_PLAN
version: v0.0.2
spec:
annotations: {}
authRole: {}
defaultInputs: {}
entityMetadata: {}
fixedInputs: {}
labels: {}
rawOutputDataConfig: {}
workflowId:
domain: development
name: flyte.workflows.workflow.my_wf
project: hackday
resourceType: WORKFLOW
version: v0.0.2
- closure:
createdAt: "2022-04-08T09:19:29.014621Z"
expectedInputs: {}
expectedOutputs:
variables:
o0:
description: o0
type:
simple: STRING
updatedAt: "2022-04-08T09:19:29.014621Z"
id:
domain: development
name: flyte.workflows.workflow.my_wf
project: hackday
resourceType: LAUNCH_PLAN
version: v0.0.1
spec:
annotations: {}
authRole: {}
defaultInputs: {}
entityMetadata: {}
fixedInputs: {}
labels: {}
rawOutputDataConfig: {}
workflowId:
domain: development
name: flyte.workflows.workflow.my_wf
project: hackday
resourceType: WORKFLOW
version: v0.0.1
Prafulla Mahindrakar
04/25/2022, 1:39 PMHampus Rosvall
04/25/2022, 1:59 PMPrafulla Mahindrakar
04/25/2022, 2:02 PMHampus Rosvall
04/25/2022, 2:47 PMflytectl
it gets picked up in the Pod annotations, but it seems like it is using the `flyte-user-role`set in the environment variables. E.g., I can pass iamRoleArn: asdf
and I am still able to interact with AWS resources as per my flyte-user-role
IAM policy.katrina
flytectl get execution -p flytesnacks -d development <name> -o yaml
Hampus Rosvall
04/25/2022, 5:04 PMkatrina
Prafulla Mahindrakar
04/25/2022, 5:14 PM<http://ghcr.io/flyteorg/flyteadmin-release:v1.0.0-b1|ghcr.io/flyteorg/flyteadmin-release:v1.0.0-b1>
With an unknown IamRole
k get pods -n flytesnacks-development avj9vsrl8qxx8bmbz9mv-n0-0 -o yaml
apiVersion: v1
kind: Pod
metadata:
annotations:
<http://cluster-autoscaler.kubernetes.io/safe-to-evict|cluster-autoscaler.kubernetes.io/safe-to-evict>: "false"
<http://eks.amazonaws.com/role-arn|eks.amazonaws.com/role-arn>: abcdefgh
<http://kubernetes.io/psp|kubernetes.io/psp>: eks.privileged
Execution failed rightly for me
Exception: Called process exited with error code: 1. Stderr dump:
b'upload failed: ../tmp/flyte/local_flytekit/ae64be4b354052b1124d2fff213d9654/engine_dir/error.pb to <s3://flyte-demo/metadata/propeller/flytesnacks-development-avj9vsrl8qxx8bmbz9mv/n0/data/0/error.pb> An error occurred (AccessDenied) when calling the PutObject operation: Access Denied\n'
reason: Error
startedAt: "2022-04-25T17:10:00Z"
Hampus Rosvall
04/25/2022, 5:35 PMflytectl get execution -p hackday -d development f0dbbc7271146457a8d5 -o yaml
closure:
createdAt: "2022-04-25T14:12:08.553786193Z"
duration: 46.179998699s
outputs:
uri: s3://<bucket>/metadata/propeller/hackday-development-f0dbbc7271146457a8d5/end-node/data/0/outputs.pb
phase: SUCCEEDED
startedAt: "2022-04-25T14:12:13.666273249Z"
stateChangeDetails:
occurredAt: "2022-04-25T14:12:08.553786193Z"
updatedAt: "2022-04-25T14:12:59.846271699Z"
workflowId:
domain: development
name: flyte.workflows.workflow.my_wf
project: hackday
resourceType: WORKFLOW
version: v0.0.5
id:
domain: development
name: f0dbbc7271146457a8d5
project: hackday
spec:
authRole:
assumableIamRole: arn:aws:iam::asdasd
launchPlan:
domain: development
name: flyte.workflows.workflow.my_wf
project: hackday
resourceType: LAUNCH_PLAN
version: v0.0.5
metadata:
systemMetadata: {}
securityContext:
runAs:
iamRole: arn:aws:iam::asdasd
So it actually looks correct, but I am downloading some data from S3 in the task which runs successfully which makes me think it is using another role, or am I missing something?Prafulla Mahindrakar
04/25/2022, 5:40 PMmemory: 500Mi
Environment:
FLYTE_INTERNAL_CONFIGURATION_PATH: /root/sandbox.config
FLYTE_INTERNAL_IMAGE: <http://ghcr.io/flyteorg/flytecookbook:core-773447b298bfa8ecfc2b25983ce1ed2d33753d01|ghcr.io/flyteorg/flytecookbook:core-773447b298bfa8ecfc2b25983ce1ed2d33753d01>
FLYTE_INTERNAL_EXECUTION_WORKFLOW: flytesnacks:development:core.basic.lp.go_greet
FLYTE_INTERNAL_EXECUTION_ID: avj9vsrl8qxx8bmbz9mv
FLYTE_INTERNAL_EXECUTION_PROJECT: flytesnacks
FLYTE_INTERNAL_EXECUTION_DOMAIN: development
FLYTE_ATTEMPT_NUMBER: 0
FLYTE_INTERNAL_TASK_PROJECT: flytesnacks
FLYTE_INTERNAL_TASK_DOMAIN: development
FLYTE_INTERNAL_TASK_NAME: core.basic.lp.greet
FLYTE_INTERNAL_TASK_VERSION: 773447b298bfa8ecfc2b25983ce1ed2d33753d01
FLYTE_INTERNAL_PROJECT: flytesnacks
FLYTE_INTERNAL_DOMAIN: development
FLYTE_INTERNAL_NAME: core.basic.lp.greet
FLYTE_INTERNAL_VERSION: 773447b298bfa8ecfc2b25983ce1ed2d33753d01
Mounts:
/var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-nvnx4 (ro)
flyteadmin:
roleNameKey: "<http://eks.amazonaws.com/role-arn|eks.amazonaws.com/role-arn>"
profilerPort: 10254
eventVersion: 2
metricsScope: "flyte:"
metadataStoragePrefix:
Hampus Rosvall
04/25/2022, 5:45 PMAWS_
env vars are set by the Service Account on EKS (https://docs.aws.amazon.com/eks/latest/userguide/specify-service-account-role.html)katrina
Mücahit
04/25/2022, 6:46 PMHampus Rosvall
04/25/2022, 7:07 PMkatrina
iamRole: arn:aws:iam::asdasd
was not correct or expected?Hampus Rosvall
04/25/2022, 7:59 PMkatrina
Hampus Rosvall
04/25/2022, 8:31 PMkatrina
Prafulla Mahindrakar
04/26/2022, 2:20 PMkubectl exec -it test-praf -n flytesnacks-development -- /bin/bash
root@test-praf:~#
root@test-praf:~#
root@test-praf:~#
root@test-praf:~# touch a
root@test-praf:~# aws s3 cp a <s3://flyte-demo/metadata/propeller/flytesnacks-development-akqfpd7n4b8lh78m9c5r/n0/data/0/a>
upload failed: ./a to <s3://flyte-demo/metadata/propeller/flytesnacks-development-akqfpd7n4b8lh78m9c5r/n0/data/0/a> An error occurred (AccessDenied) when calling the PutObject operation: Access Denied
The pod has the following
k get pod -n flytesnacks-development test-praf -o yaml
apiVersion: v1
kind: Pod
metadata:
annotations:
<http://iam.amazonaws.com/role|iam.amazonaws.com/role>: arn:aws:iam::590375264460:role/eksctl-flyte-demo-2-addon-iamserviceaccount-Role1-11QUDNRU7X84P
....
name: test-praf
namespace: flytesnacks-development
....
serviceAccount: default
serviceAccountName: default
Now if i use it with service account annotated with IAM role which has permissions 1NRJQGB2NSHL9 , then AWS_ROLE_ARN is exported with this annotated role and it doesn’t matter what additional annotation are on the pod related to role which are simply ignored
k get pod -n flytesnacks-development test-praf -o yaml
apiVersion: v1
kind: Pod
metadata:
annotations:
<http://iam.amazonaws.com/role|iam.amazonaws.com/role>: arn:aws:iam::590375264460:role/eksctl-flyte-demo-2-addon-iamserviceaccount-Role1-11QUDNRU7X84P
....
- name: AWS_ROLE_ARN
value: arn:aws:iam::590375264460:role/eksctl-flyte-demo-2-addon-iamserviceaccount-Role1-1NRJQGB2NSHL9
- name: AWS_WEB_IDENTITY_TOKEN_FILE
value: /var/run/secrets/eks.amazonaws.com/serviceaccount/token
....
serviceAccount: demo
serviceAccountName: demo
And it allows me to copy to s3 using this annotated rrole.
kubectl exec -it test-praf -n flytesnacks-development -- /bin/bash
root@test-praf:~# touch a
root@test-praf:~# aws s3 cp a <s3://flyte-demo/metadata/propeller/flytesnacks-development-akqfpd7n4b8lh78m9c5r/n0/data/0/a>
upload: ./a to <s3://flyte-demo/metadata/propeller/flytesnacks-development-akqfpd7n4b8lh78m9c5r/n0/data/0/a>
root@test-praf:~# cat ~/.aws/cli/cache/574579698c8d6815d0805a57d875c58fa630e77a.json
{"Credentials": {"AccessKeyId": "...", ..., "Expiration": "2022-04-26T13:52:59Z"}, "SubjectFromWebIdentityToken": "system:serviceaccount:flytesnacks-development:demo", "AssumedRoleUser": {"AssumedRoleId": "AROAYS5I3UDGCIBQWKOCJ:botocore-session-1650977579", "Arn": "arn:aws:sts::590375264460:assumed-role/eksctl-flyte-demo-2-addon-iamserviceaccount-Role1-1NRJQGB2NSHL9/botocore-session-1650977579"}, "Provider": "arn:aws:iam::590375264460:oidc-provider/oidc.eks.us-east-2.amazonaws.com/id/46B254ACC1AC1B23CCCA2973F62AB323", "Audience": "<http://sts.amazonaws.com|sts.amazonaws.com>", "ResponseMetadata": {"RequestId": "b9301edc-ce4f-42ff-a049-4cb09936c5c4", "HTTPStatusCode": 200, "HTTPHeaders": {"x-amzn-requestid": "b9301edc-ce4f-42ff-a049-4cb09936c5c4", "content-type": "text/xml", "content-length": "1975", "date": "Tue, 26 Apr 2022 12:52:59 GMT"}, "RetryAttempts": 0}}