https://flyte.org logo
r

Roberto Ruiz

05/18/2022, 8:27 PM
Hi, I have a flyte cluster on AWS with the manual setup. I'm having trouble registering workflows, for example trying to register the examples with
flytectl register examples -d development -p flytesnacks
I'm getting:
Copy code
Error: example 0xc0006ae770 failed to register rpc error: code = Internal desc = failed to write marshaled workflow [resource_type:WORKFLOW project:"flytesnacks" domain:"development" name:"blast.blastx_example.blast_wf" version:"v0.3.81" ] to storage <s3://flyte-cluster-bucket/metadata/admin/flytesnacks/development/blast.blastx_example.blast_wf/v0.3.81> with err Failed to write data [3170b] to path [metadata/admin/flytesnacks/development/blast.blastx_example.blast_wf/v0.3.81].: PutObject, putting object: WebIdentityErr: failed to retrieve credentials
caused by: ValidationError: Request ARN is invalid
But I can't seem to find which ARN this error is referring to
k

katrina

05/18/2022, 8:34 PM
hey it looks like flyteadmin is failing to offload the remote workflow closure. can you share what's in the storage block of your flyteadmin config? does the flyteadmin service account have s3 put permissions?
r

Roberto Ruiz

05/18/2022, 9:27 PM
config is
Copy code
storage:
  type: s3
  connection:
    auth-type: iam
    region: us-east-2
  container: flyte-cluster-bucket
The service account on the helm chart is
Copy code
serviceAccount:
    # -- If the service account is created by you, make this false, else a new service account will be created and the iam-role-flyte will be added
    # you can change the name of this role
    create: true
    annotations:
      <http://eks.amazonaws.com/role-arn|eks.amazonaws.com/role-arn>: arn:aws:iam::{{ .Values.userSettings.accountNumber }}:role/iam-role-flyte
The iam-role-flyte has AmazonS3FullAccess
k

katrina

05/18/2022, 9:55 PM
stupid question, but just to double check does it have s3 access for the
Copy code
<s3://flyte-cluster-bucket>
does the bucket itself have any specific policy?
r

Roberto Ruiz

05/18/2022, 10:02 PM
there's no policy on the bucket
k

katrina

05/19/2022, 6:51 PM
cc @Yuvraj @Haytham Abuelfutuh
r

Roberto Ruiz

05/23/2022, 2:35 PM
is there a way to know what ARN is causing the error? maybe I can debug it with that
I tried adding this very permissive policy on the bucket, but got the same result:
Copy code
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "Statement1",
      "Effect": "Allow",
      "Principal": {
        "AWS": "*"
      },
      "Action": "s3:*",
      "Resource": "arn:aws:s3:::flyte-cluster-bucket"
    }
  ]
}
k

katrina

05/23/2022, 4:07 PM
does
iam-role-flyte
have s3 put access?
the arn thing is confusing, not the erorr i would expect to see re: permissions 😕
r

Roberto Ruiz

05/23/2022, 6:25 PM
the problem was unrelated to AWS config, the problem was I had the Account ID without quotes in the Helm yaml
thanks for your help
k

katrina

05/23/2022, 6:26 PM
oh, annoying! thanks for the update and glad you figured it out
r

Roberto Ruiz

05/23/2022, 6:26 PM
I would change the "AWS (EKS) Manual Setup" document so that the Sample Value for the account ID to be
"173113148371"
instead of
173113148371
k

katrina

05/23/2022, 7:00 PM
hey @Roberto Ruiz would you want to open a PR for that 😄 or file an issue to track?
r

Roberto Ruiz

05/23/2022, 7:49 PM
k

katrina

05/23/2022, 8:26 PM
awesome, thank you!
63 Views