https://flyte.org
Join Slack
Half solved the issue ^ and now I'm getting a s3 putObject auth error ```lytekit.exceptions.user.Fly...
# flyte-deployment
n
Half solved the issue ^ and now I'm getting a s3 putObject auth error
Copy code
lytekit.exceptions.user.FlyteAssertion: Failed to put data from /tmp/flyte-gnaguqof/sandbox/local_flytekit/engine_dir to <s3://flyte-bucket/metadata/propeller/flytetester-development-a9xg68jktjbhd7nz7sc7/n0/data/0> (recursive=True).

Original exception: Called process exited with error code: 1.  Stderr dump:

b'upload failed: ../../tmp/flyte-gnaguqof/sandbox/local_flytekit/engine_dir/error.pb to <s3://flyte-bucket/metadata/propeller/flytetester-development-a9xg68jktjbhd7nz7sc7/n0/data/0/error.pb> An error occurred (AccessDenied) when calling the PutObject operation: Access Denied\n'
the flyte-user-role has AmazonS3FullAccess, I verified that the failing pod has the env var 
AWS_ROLE_ARN
set to the flyte-user-role. And when I look inside the bucket, I can see that there's a 
<s3://flyfte-bucket/metadata/propeller/flytetester-development-a9xg68jktjbhd7nz7sc7/n0/data/inputs.pb>
So something is working correctly to put in objects but then fails? Not using minio either and docker image has 
awscli==1.25.94
actually, now that I'm looking at it: aws says the 
flyte-user-role
has never been accessed.
message has been deleted
Can it be this? This says default instead of flyte-user-role?
f
you have to use the service account
did you IAM for serviceaccounts?
and then associate that to the execution
you can ofcourse have a default for a project/domain
n
I'm launching these from the UI. How can I set the default for the project?
f
you have to open the fold “Advanced Options”
n
isnt that for specific execution?
f
and to set the default - either when you register set the service account - or when you run with pyflyte run
Copy code
pyflyte run --service-account
n
which arg is it in 
flytectl register
?
f
so this will help you set the default per project/domain ^
and in register
pyflyte register --service-account=“…”
n
I still get the error with
f
ohh not iam role
service account
sorry about the confusion here
iam role only works if you use kIAM (which I think you dont)
n
f
similarish
the EKS IAM role for ServiceAccounts is a thing that AWS has
you have to follow specific steps that they have
check the doc link on AWS that I shared
n
is this something that you always have to do for EKS deployments?
https://docs.flyte.org/en/latest/deployment/aws/manual.html#oidc-provider-for-the-eks-cluster there's some stuff here that has to do with the iamroles. I did do this.
so for following the doc you shared, the IAM role I use is flyte-user-role and if I've been using the default kube service account, then default for that too?
oh i got it working
❤️ 1
f
Sorry busy with kids now - but yeahhhh
j
It worked for us too! For posterity and more concrete example, I annotated the default service account for each namespace like this:
Copy code
$ kubectl annotate serviceaccount -n flytesnacks-development default <http://eks.amazonaws.com/role-arn=arn:aws:iam::xxxx:role/flyte-user-role|eks.amazonaws.com/role-arn=arn:aws:iam::xxxx:role/flyte-user-role>
Then a worker pod is spawn with AWS env vars like this:
Copy code
$ kubectl describe pod f81f7df8fd03f4f84b45-n0-0 -n flytesnacks-development | grep AWS
      AWS_STS_REGIONAL_ENDPOINTS:         regional
      AWS_DEFAULT_REGION:                 xxxxxx
      AWS_REGION:                         xxxxxx
      AWS_ROLE_ARN:                       arn:aws:iam::xxxxx:role/flyte-user-role
      AWS_WEB_IDENTITY_TOKEN_FILE:        /var/run/secrets/eks.amazonaws.com/serviceaccount/token
❤️ 1
thx 1
f
cc @average-finland-92144 to ensure we can capture some of the learnings here
💯 1
s
has this ever been incorporated into the guide? how to solve this during proper install?
i also see my worker pod not having flyte-worker-role despite following the flyte-the-hard-way guide
385 Views
Open in Slack
PreviousNext
Powered by

Flyte

Flyte enables production-grade orchestration for machine learning workflows and data processing created to accelerate local workflows to production.

Powered by