https://flyte.org
Join Slack
Hi, I have a flyte cluster on AWS with the manual setup. I'm having trouble registering workflows, f...
# flyte-deployment
b
Hi, I have a flyte cluster on AWS with the manual setup. I'm having trouble registering workflows, for example trying to register the examples with 
flytectl register examples -d development -p flytesnacks
I'm getting:
Copy code
Error: example 0xc0006ae770 failed to register rpc error: code = Internal desc = failed to write marshaled workflow [resource_type:WORKFLOW project:"flytesnacks" domain:"development" name:"blast.blastx_example.blast_wf" version:"v0.3.81" ] to storage <s3://flyte-cluster-bucket/metadata/admin/flytesnacks/development/blast.blastx_example.blast_wf/v0.3.81> with err Failed to write data [3170b] to path [metadata/admin/flytesnacks/development/blast.blastx_example.blast_wf/v0.3.81].: PutObject, putting object: WebIdentityErr: failed to retrieve credentials
caused by: ValidationError: Request ARN is invalid
But I can't seem to find which ARN this error is referring to
a
hey it looks like flyteadmin is failing to offload the remote workflow closure. can you share what's in the storage block of your flyteadmin config? does the flyteadmin service account have s3 put permissions?
b
config is
Copy code
storage:
  type: s3
  connection:
    auth-type: iam
    region: us-east-2
  container: flyte-cluster-bucket
The service account on the helm chart is
Copy code
serviceAccount:
    # -- If the service account is created by you, make this false, else a new service account will be created and the iam-role-flyte will be added
    # you can change the name of this role
    create: true
    annotations:
      <http://eks.amazonaws.com/role-arn|eks.amazonaws.com/role-arn>: arn:aws:iam::{{ .Values.userSettings.accountNumber }}:role/iam-role-flyte
The iam-role-flyte has AmazonS3FullAccess
a
stupid question, but just to double check does it have s3 access for the
Copy code
<s3://flyte-cluster-bucket>
does the bucket itself have any specific policy?
b
there's no policy on the bucket
a
cc @great-school-54368 @high-park-82026
b
is there a way to know what ARN is causing the error? maybe I can debug it with that
I tried adding this very permissive policy on the bucket, but got the same result:
Copy code
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "Statement1",
      "Effect": "Allow",
      "Principal": {
        "AWS": "*"
      },
      "Action": "s3:*",
      "Resource": "arn:aws:s3:::flyte-cluster-bucket"
    }
  ]
}
a
does 
iam-role-flyte
have s3 put access?
the arn thing is confusing, not the erorr i would expect to see re: permissions 😕
b
the problem was unrelated to AWS config, the problem was I had the Account ID without quotes in the Helm yaml
thanks for your help
a
oh, annoying! thanks for the update and glad you figured it out
b
I would change the "AWS (EKS) Manual Setup" document so that the Sample Value for the account ID to be 
"173113148371"
instead of 
173113148371
a
hey @big-glass-22633 would you want to open a PR for that 😄 or file an issue to track?
b
a
awesome, thank you!
407 Views
Open in Slack
PreviousNext
Powered by

Flyte

Flyte enables production-grade orchestration for machine learning workflows and data processing created to accelerate local workflows to production.

Powered by