https://flyte.org
Join Slack
Hey, i was just trying to upgrade my flytekit version in of my workflows and as a result i get a S3 ...
# flyte-support
f
Hey, i was just trying to upgrade my flytekit version in of my workflows and as a result i get a S3 Access denied as a result. With the old dependencies everything works as expected. These are my old dependencies:
Copy code
s3fs==0.4.2
flytekit==1.3.2
awscli==1.27.118
mlstuff
And these are my updates ones:
Copy code
fsspec==2023.6.0
s3fs==2023.6.0
flytekit==1.8.1
awscli==1.27.161
mlstuff
At the moment i am attaching a aws iam role to the default service account, which is used by propeller. Someone knows whats up?
t
what’s the error?
1.3 is pretty old, a fair bit has changed.
but irsa should work fine. are you sure the role has access? user containers sometimes require different permissions than propeller.
f
Thats the Error:
Copy code
FlyteAssertion: Failed to get data from 
<s3://kineo-flyte-central-cluster-bucket/demo/development/YHTQ6ZBZFBCHRZMUFRW>
UQ3MGIE======/fast37dd06657c173c4af3b785092c82a7eb.tar.gz to /root/ 
(recursive=False).

Original exception: Access Denied
.
Quite straight forward but with the same role on flytekit 1.3 i have access
t
can you inspect the task pod? just to double check it’s running with the right service account? are there any config files the container is relying on?
the main change between 1.3 and 1.8 wrt data handling is usage of fsspec/s3fs.
but both the aws cli and s3fs ultimately fall back to boto for credentials handling.
f
I might be able to check tomorrow again but i iam pretty sure i saw the default service account with the correct role annotation.
t
k
f
For some reason i had the feeling the role could not be assumed at all. Because i saw also the role itself not updating for ‘last activity’ in AWs iam
t
i see
paste more errors too if you have them. also i think you can drop the awscli requirement.
f
Alright, i will. Thank you Yee for pointing me in the right direction!
Alright it was indeed just a matter of AWS IAM. It looks like flytekit 1.3 only needs the 
s3:GetObject
on the control plane bucket. flytekit 1.8 needs 
s3:ListBucket
in addition. I think i saw recursive=False for pyflyte-fast-execute somewhere but anyway happy its that simple 🙂
t
ohhhh
crap yeah i saw this yesterday too, didn’t make the connection.
f
No problem 😉
t
we were wondering why something wasn’t working - fsspec was trying to list dir on an http endpoint that didn’t support it.
this was just for a put operation. it was needing to check to see if it was not a directory. feel like head would be more appropriate though, but yeah
thanks again
👍 1
104 Views
Open in Slack
PreviousNext
Powered by

Flyte

Flyte enables production-grade orchestration for machine learning workflows and data processing created to accelerate local workflows to production.

Powered by