Hey eveyone My team and I have been going through the standa Flyte #announcements

Hey eveyone! My team and I have been going through...

Louis DiNatale

04/18/2022, 2:35 PM

Hey eveyone! My team and I have been going through the standard set up and have run into a snag with the the hello world set up on aws. We are running into access denied to our s3 buckets. We have all the roles set up and have given the workflow the permission through the console. Is there some basic step we probably missed?

Mike Zhong

04/18/2022, 2:36 PM

This is the error we are receiving. Even when specifying a role (which has full s3 permissions at launch time), we are encountering this error

Prafulla Mahindrakar

04/18/2022, 2:49 PM

Can you check what is the service account on the pod launched by flyte for this workflow and see if that sa ha permissions to write . Also are you following this guide for setup https://docs.flyte.org/en/latest/deployment/aws/manual.html#flyte-user-role

Louis DiNatale

04/18/2022, 2:50 PM

Yes we followed this guide.

Louis DiNatale

04/18/2022, 2:50 PM

Will follow up after checking

Ketan (kumare3)

04/18/2022, 3:20 PM

Welcome to the community @Louis DiNatale

🎉 1

Mike Zhong

04/18/2022, 5:44 PM

the pod is using the

flyte-user-role

shown below. In aws console, we have checked this role and can confirm it has full s3 permissions. Do we need to set up any trust permission to allow the node role to assume this role?

Mike Zhong

04/18/2022, 5:46 PM

under

Annotations

, it is showing a different role though

Copy code

Annotations:  <http://cluster-autoscaler.kubernetes.io/safe-to-evict|cluster-autoscaler.kubernetes.io/safe-to-evict>: false
              <http://iam.amazonaws.com/role|iam.amazonaws.com/role>: arn:aws:iam::763216446258:role/Dev-iam-role-flyte
              <http://kubernetes.io/psp|kubernetes.io/psp>: eks.privileged

However, both roles in IAM have full s3 permissions, not sure which one is actually being used

Prafulla Mahindrakar

04/19/2022, 5:22 AM

Assuming you have gone through these steps to add the trust relationship between the oidc provider and the created roles https://docs.flyte.org/en/latest/deployment/aws/manual.html#oidc-provider-for-the-eks-cluster You can find which role is being used by hitting the metadataserver from within the pod . This guide shows how to check the role https://kubernetes-on-aws.readthedocs.io/en/latest/user-guide/iam-roles.html Question : Have you enabled cluster resource manager in the helm chart ? Another way would be to user kubernetes service account. You can associate iam role with a kubernetes service account and configure flyte to use that service account for user pods https://docs.aws.amazon.com/eks/latest/userguide/specify-service-account-role.html You can send this service account from the UI instead of the IAM role and check if that works for you .

Prafulla Mahindrakar

04/19/2022, 5:31 AM

Another thing i noticed is the rolename being used on the pods uses

Copy code

<http://iam.amazonaws.com/role|iam.amazonaws.com/role>

instead of

Copy code

<http://eks.amazonaws.com/role-arn|eks.amazonaws.com/role-arn>

The value seems to be arn and not just the role name . You can modify this behavior by updating the admin config map and adding the roleNameKey

Copy code

flyteadmin:
      roleNameKey: "<http://eks.amazonaws.com/role-arn|eks.amazonaws.com/role-arn>"
      profilerPort: 10254
      eventVersion: 2
      metricsScope: "flyte:"
      metadataStoragePrefix:
        - "metadata"
        - "admin"

Louis DiNatale

04/19/2022, 1:54 PM

Thank you Prafulla, I will go through these steps today and update this thread.

Mike Zhong

04/19/2022, 2:22 PM

Thanks for the information @User, the

AWS_ROLE_ARN

from the screenshot had a trust policy with an OIDC provider for a previous cluster (test deployment). We created a new cluster and created new roles but our IaC must not have updated entirely as the new roles were still trusting the old OIDC. I have made the update and we will be testing shortly, thanks for your help

Prafulla Mahindrakar

04/19/2022, 3:51 PM

Sounds good

Mike Zhong

04/22/2022, 1:13 PM

Thanks for the help, this was resolved, needed a trust policy for the flyte user role and OIDC provider

🎉 2

Ketan (kumare3)

04/22/2022, 1:18 PM

Is there something to add to docs. @Mike Zhong can you suggest a doc edit

Mike Zhong

04/22/2022, 1:19 PM

I think the docs are clear enough as is, our confusion is a result of having deployed multiple flyte + eks stacks in the same AWS account and region

Ketan (kumare3)

04/22/2022, 1:22 PM

But that should work too?

Prafulla Mahindrakar

04/22/2022, 1:50 PM

Yes Its possible to do that too if we keep them independent and having different cluster policy names and oidc provider

162 Views

Open in Slack

Previous Next