Hey eveyone! My team and I have been going through...
# announcements
Hey eveyone! My team and I have been going through the standard set up and have run into a snag with the the hello world set up on aws. We are running into access denied to our s3 buckets. We have all the roles set up and have given the workflow the permission through the console. Is there some basic step we probably missed?
This is the error we are receiving. Even when specifying a role (which has full s3 permissions at launch time), we are encountering this error
Can you check what is the service account on the pod launched by flyte for this workflow and see if that sa ha permissions to write . Also are you following this guide for setup https://docs.flyte.org/en/latest/deployment/aws/manual.html#flyte-user-role
Yes we followed this guide.
Will follow up after checking
Welcome to the community @Louis DiNatale
🎉 1
the pod is using the
shown below. In aws console, we have checked this role and can confirm it has full s3 permissions. Do we need to set up any trust permission to allow the node role to assume this role?
, it is showing a different role though
Copy code
Annotations:  <http://cluster-autoscaler.kubernetes.io/safe-to-evict|cluster-autoscaler.kubernetes.io/safe-to-evict>: false
              <http://iam.amazonaws.com/role|iam.amazonaws.com/role>: arn:aws:iam::763216446258:role/Dev-iam-role-flyte
              <http://kubernetes.io/psp|kubernetes.io/psp>: eks.privileged
However, both roles in IAM have full s3 permissions, not sure which one is actually being used
Assuming you have gone through these steps to add the trust relationship between the oidc provider and the created roles https://docs.flyte.org/en/latest/deployment/aws/manual.html#oidc-provider-for-the-eks-cluster You can find which role is being used by hitting the metadataserver from within the pod . This guide shows how to check the role https://kubernetes-on-aws.readthedocs.io/en/latest/user-guide/iam-roles.html Question : Have you enabled cluster resource manager in the helm chart ? Another way would be to user kubernetes service account. You can associate iam role with a kubernetes service account and configure flyte to use that service account for user pods https://docs.aws.amazon.com/eks/latest/userguide/specify-service-account-role.html You can send this service account from the UI instead of the IAM role and check if that works for you .
Another thing i noticed is the rolename being used on the pods uses
Copy code
instead of
Copy code
The value seems to be arn and not just the role name . You can modify this behavior by updating the admin config map and adding the roleNameKey
Copy code
      roleNameKey: "<http://eks.amazonaws.com/role-arn|eks.amazonaws.com/role-arn>"
      profilerPort: 10254
      eventVersion: 2
      metricsScope: "flyte:"
        - "metadata"
        - "admin"
Thank you Prafulla, I will go through these steps today and update this thread.
Thanks for the information @User, the
from the screenshot had a trust policy with an OIDC provider for a previous cluster (test deployment). We created a new cluster and created new roles but our IaC must not have updated entirely as the new roles were still trusting the old OIDC. I have made the update and we will be testing shortly, thanks for your help
Sounds good
Thanks for the help, this was resolved, needed a trust policy for the flyte user role and OIDC provider
🎉 2
Is there something to add to docs. @Mike Zhong can you suggest a doc edit
I think the docs are clear enough as is, our confusion is a result of having deployed multiple flyte + eks stacks in the same AWS account and region
But that should work too?
Yes Its possible to do that too if we keep them independent and having different cluster policy names and oidc provider