Hi I noticed helm treated our `accountNumber` as a float in Flyte #flyte-deployment

Hi - I noticed helm treated our `accountNumber` as...

Edgar Trujillo

06/21/2022, 3:32 PM

Hi - I noticed helm treated our

accountNumber

as a float in all annotation related lines in values-eks.yaml

Copy code

<http://eks.amazonaws.com/role-arn|eks.amazonaws.com/role-arn>: arn:aws:iam::{{ .Values.userSettings.accountNumber }}:role/iam-role-flyte

I changed the lines to explicitly define the arn and can now schedule executions to run

remotely

But the console shows no actual executions and the role assigned is the

default

rather than 1 of the 2 roles created (

iam-role-flyte

flyte-user-role

) Doing a describe on the flytepropeller shows

Copy code

Environment:
      POD_NAME:                     flytepropeller-7d67df4c85-7sf4v (v1:metadata.name)
      POD_NAMESPACE:                flyte (v1:metadata.namespace)
      AWS_STS_REGIONAL_ENDPOINTS:   regional
      AWS_DEFAULT_REGION:           us-east-1
      AWS_REGION:                   us-east-1
      AWS_ROLE_ARN:                 arn:aws:iam::1.AWS_ACCTe+11:role/iam-role-flyte
      AWS_WEB_IDENTITY_TOKEN_FILE:  /var/run/secrets/eks.amazonaws.com/serviceaccount/token

Any idea where the

AWS_ROLE_ARN

is being mounted from?

Nastya Rusina

06/21/2022, 5:44 PM

@Jason Porter

Edgar Trujillo

06/21/2022, 6:03 PM

I resolved it by manually editing the deployment and specifying the

AWS_ROLE_ARN

env value. I saw a similar botched

AWS_ROLE_ARN

in the DataCatalog deployment and after updating that i now get

Copy code

E0621 17:56:06.090698       1 workers.go:102] error syncing 'flytesnacks-development/amp8cm9q2fbrdn84nvcd': failed at Node[start-node]. CausedByError: Failed to store workflow inputs (as start node), caused by: Failed to write data [61b] to path [metadata/propeller/flytesnacks-development-amp8cm9q2fbrdn84nvcd/start-node/data/0/outputs.pb].: PutObject, putting object: AccessDenied: Access Denied
	status code: 403, request id: HXTEMC45YC2H49GF, host id: zkybrDtNFRW0TBe5hbBwiWjiSwdXQJIWAEG6al3gKOnCojFSEPmmY3/GB1H8QYbDgncp6xRpDfw=

which is strange since these paths exists in the s3 •

metadata/flytesnacks/development/amp8cm9q2fbrdn84nvcd/inputs

•

metadata/flytesnacks/development/amp8cm9q2fbrdn84nvcd/user_input

Edgar Trujillo

06/21/2022, 6:43 PM

I realized the root error was in my

flytepropeller

deployment. While I manually added:

Copy code

- name: AWS_ROLE_ARN
  value: arn:aws:iam::ACCCOUNT:role/iam-role-flyte

I forgot to add:

Copy code

- name: AWS_WEB_IDENTITY_TOKEN_FILE
  value: /var/run/secrets/eks.amazonaws.com/serviceaccount/token

FWIW, I deployed this with

tf

and

helm

onto a private EKS and I think using

tf

may have caused the initial botched

AWS_ROLE_ARN

👍 1

Ketan (kumare3)

06/22/2022, 1:13 AM

Ohh @Edgar Trujillo can you please recommend a doc edit - cc @Marc Paquette

168 Views

Open in Slack

Previous Next