https://flyte.org logo
Powered by
l

Louis DiNatale

04/18/2022, 2:35 PM
Hey eveyone! My team and I have been going through the standard set up and have run into a snag with the the hello world set up on aws. We are running into access denied to our s3 buckets. We have all the roles set up and have given the workflow the permission through the console. Is there some basic step we probably missed?
m

Mike Zhong

04/18/2022, 2:36 PM
This is the error we are receiving. Even when specifying a role (which has full s3 permissions at launch time), we are encountering this error
p

Prafulla Mahindrakar

04/18/2022, 2:49 PM
Can you check what is the service account on the pod launched by flyte for this workflow and see if that sa ha permissions to write . Also are you following this guide for setup https://docs.flyte.org/en/latest/deployment/aws/manual.html#flyte-user-role
l

Louis DiNatale

04/18/2022, 2:50 PM
Yes we followed this guide.
Will follow up after checking
k

Ketan (kumare3)

04/18/2022, 3:20 PM
Welcome to the community @Louis DiNatale
🎉 1
m

Mike Zhong

04/18/2022, 5:44 PM
the pod is using the 
flyte-user-role
shown below. In aws console, we have checked this role and can confirm it has full s3 permissions. Do we need to set up any trust permission to allow the node role to assume this role?
under 
Annotations
, it is showing a different role though
Copy code
Annotations:  <http://cluster-autoscaler.kubernetes.io/safe-to-evict|cluster-autoscaler.kubernetes.io/safe-to-evict>: false
              <http://iam.amazonaws.com/role|iam.amazonaws.com/role>: arn:aws:iam::763216446258:role/Dev-iam-role-flyte
              <http://kubernetes.io/psp|kubernetes.io/psp>: eks.privileged
However, both roles in IAM have full s3 permissions, not sure which one is actually being used
p

Prafulla Mahindrakar

04/19/2022, 5:22 AM
Assuming you have gone through these steps to add the trust relationship between the oidc provider and the created roles https://docs.flyte.org/en/latest/deployment/aws/manual.html#oidc-provider-for-the-eks-cluster You can find which role is being used by hitting the metadataserver from within the pod . This guide shows how to check the role https://kubernetes-on-aws.readthedocs.io/en/latest/user-guide/iam-roles.html Question : Have you enabled cluster resource manager in the helm chart ? Another way would be to user kubernetes service account. You can associate iam role with a kubernetes service account and configure flyte to use that service account for user pods https://docs.aws.amazon.com/eks/latest/userguide/specify-service-account-role.html You can send this service account from the UI instead of the IAM role and check if that works for you .
Another thing i noticed is the rolename being used on the pods uses
Copy code
<http://iam.amazonaws.com/role|iam.amazonaws.com/role>
instead of
Copy code
<http://eks.amazonaws.com/role-arn|eks.amazonaws.com/role-arn>
The value seems to be arn and not just the role name . You can modify this behavior by updating the admin config map and adding the roleNameKey
Copy code
flyteadmin:
      roleNameKey: "<http://eks.amazonaws.com/role-arn|eks.amazonaws.com/role-arn>"
      profilerPort: 10254
      eventVersion: 2
      metricsScope: "flyte:"
      metadataStoragePrefix:
        - "metadata"
        - "admin"
l

Louis DiNatale

04/19/2022, 1:54 PM
Thank you Prafulla, I will go through these steps today and update this thread.
m

Mike Zhong

04/19/2022, 2:22 PM
Thanks for the information @User, the 
AWS_ROLE_ARN
from the screenshot had a trust policy with an OIDC provider for a previous cluster (test deployment). We created a new cluster and created new roles but our IaC must not have updated entirely as the new roles were still trusting the old OIDC. I have made the update and we will be testing shortly, thanks for your help
p

Prafulla Mahindrakar

04/19/2022, 3:51 PM
Sounds good
m

Mike Zhong

04/22/2022, 1:13 PM
Thanks for the help, this was resolved, needed a trust policy for the flyte user role and OIDC provider
🎉 2
k

Ketan (kumare3)

04/22/2022, 1:18 PM
Is there something to add to docs. @Mike Zhong can you suggest a doc edit
m

Mike Zhong

04/22/2022, 1:19 PM
I think the docs are clear enough as is, our confusion is a result of having deployed multiple flyte + eks stacks in the same AWS account and region
k

Ketan (kumare3)

04/22/2022, 1:22 PM
But that should work too?
p

Prafulla Mahindrakar

04/22/2022, 1:50 PM
Yes Its possible to do that too if we keep them independent and having different cluster policy names and oidc provider
2 Views
Powered by