https://flyte.org
Join Slack
Hey eveyone! My team and I have been going through the standard set up and have run into a snag with...
# announcements
n
Hey eveyone! My team and I have been going through the standard set up and have run into a snag with the the hello world set up on aws. We are running into access denied to our s3 buckets. We have all the roles set up and have given the workflow the permission through the console. Is there some basic step we probably missed?
t
This is the error we are receiving. Even when specifying a role (which has full s3 permissions at launch time), we are encountering this error
i
Can you check what is the service account on the pod launched by flyte for this workflow and see if that sa ha permissions to write . Also are you following this guide for setup https://docs.flyte.org/en/latest/deployment/aws/manual.html#flyte-user-role
n
Yes we followed this guide.
Will follow up after checking
f
Welcome to the community @nice-zebra-99977
🎉 1
t
the pod is using the 
flyte-user-role
shown below. In aws console, we have checked this role and can confirm it has full s3 permissions. Do we need to set up any trust permission to allow the node role to assume this role?
under 
Annotations
, it is showing a different role though
Copy code
Annotations:  <http://cluster-autoscaler.kubernetes.io/safe-to-evict|cluster-autoscaler.kubernetes.io/safe-to-evict>: false
              <http://iam.amazonaws.com/role|iam.amazonaws.com/role>: arn:aws:iam::763216446258:role/Dev-iam-role-flyte
              <http://kubernetes.io/psp|kubernetes.io/psp>: eks.privileged
However, both roles in IAM have full s3 permissions, not sure which one is actually being used
i
Assuming you have gone through these steps to add the trust relationship between the oidc provider and the created roles https://docs.flyte.org/en/latest/deployment/aws/manual.html#oidc-provider-for-the-eks-cluster You can find which role is being used by hitting the metadataserver from within the pod . This guide shows how to check the role https://kubernetes-on-aws.readthedocs.io/en/latest/user-guide/iam-roles.html Question : Have you enabled cluster resource manager in the helm chart ? Another way would be to user kubernetes service account. You can associate iam role with a kubernetes service account and configure flyte to use that service account for user pods https://docs.aws.amazon.com/eks/latest/userguide/specify-service-account-role.html You can send this service account from the UI instead of the IAM role and check if that works for you .
Another thing i noticed is the rolename being used on the pods uses
Copy code
<http://iam.amazonaws.com/role|iam.amazonaws.com/role>
instead of
Copy code
<http://eks.amazonaws.com/role-arn|eks.amazonaws.com/role-arn>
The value seems to be arn and not just the role name . You can modify this behavior by updating the admin config map and adding the roleNameKey
Copy code
flyteadmin:
      roleNameKey: "<http://eks.amazonaws.com/role-arn|eks.amazonaws.com/role-arn>"
      profilerPort: 10254
      eventVersion: 2
      metricsScope: "flyte:"
      metadataStoragePrefix:
        - "metadata"
        - "admin"
n
Thank you Prafulla, I will go through these steps today and update this thread.
t
Thanks for the information @User, the 
AWS_ROLE_ARN
from the screenshot had a trust policy with an OIDC provider for a previous cluster (test deployment). We created a new cluster and created new roles but our IaC must not have updated entirely as the new roles were still trusting the old OIDC. I have made the update and we will be testing shortly, thanks for your help
i
Sounds good
t
Thanks for the help, this was resolved, needed a trust policy for the flyte user role and OIDC provider
🎉 2
f
Is there something to add to docs. @thousands-area-8239 can you suggest a doc edit
t
I think the docs are clear enough as is, our confusion is a result of having deployed multiple flyte + eks stacks in the same AWS account and region
f
But that should work too?
i
Yes Its possible to do that too if we keep them independent and having different cluster policy names and oidc provider
162 Views
Open in Slack
PreviousNext
Powered by

Flyte

Flyte enables production-grade orchestration for machine learning workflows and data processing created to accelerate local workflows to production.

Powered by