https://flyte.org logo
Powered by
#ask-the-community
Title
# ask-the-community
j

Jan Fiedler

07/31/2023, 3:53 PM
Hey, i was just trying to upgrade my flytekit version in of my workflows and as a result i get a S3 Access denied as a result. With the old dependencies everything works as expected. These are my old dependencies:
Copy code
s3fs==0.4.2
flytekit==1.3.2
awscli==1.27.118
mlstuff
And these are my updates ones:
Copy code
fsspec==2023.6.0
s3fs==2023.6.0
flytekit==1.8.1
awscli==1.27.161
mlstuff
At the moment i am attaching a aws iam role to the default service account, which is used by propeller. Someone knows whats up?
y

Yee

07/31/2023, 4:28 PM
what’s the error?
1.3 is pretty old, a fair bit has changed.
but irsa should work fine. are you sure the role has access? user containers sometimes require different permissions than propeller.
j

Jan Fiedler

07/31/2023, 5:25 PM
Thats the Error:
Copy code
FlyteAssertion: Failed to get data from 
<s3://kineo-flyte-central-cluster-bucket/demo/development/YHTQ6ZBZFBCHRZMUFRW>
UQ3MGIE======/fast37dd06657c173c4af3b785092c82a7eb.tar.gz to /root/ 
(recursive=False).

Original exception: Access Denied
.
Quite straight forward but with the same role on flytekit 1.3 i have access
y

Yee

07/31/2023, 5:51 PM
can you inspect the task pod? just to double check it’s running with the right service account? are there any config files the container is relying on?
the main change between 1.3 and 1.8 wrt data handling is usage of fsspec/s3fs.
but both the aws cli and s3fs ultimately fall back to boto for credentials handling.
j

Jan Fiedler

07/31/2023, 5:57 PM
I might be able to check tomorrow again but i iam pretty sure i saw the default service account with the correct role annotation.
y

Yee

07/31/2023, 5:59 PM
k
j

Jan Fiedler

07/31/2023, 5:59 PM
For some reason i had the feeling the role could not be assumed at all. Because i saw also the role itself not updating for ‘last activity’ in AWs iam
y

Yee

07/31/2023, 5:59 PM
i see
paste more errors too if you have them. also i think you can drop the awscli requirement.
j

Jan Fiedler

07/31/2023, 6:00 PM
Alright, i will. Thank you Yee for pointing me in the right direction!
Alright it was indeed just a matter of AWS IAM. It looks like flytekit 1.3 only needs the 
s3:GetObject
on the control plane bucket. flytekit 1.8 needs 
s3:ListBucket
in addition. I think i saw recursive=False for pyflyte-fast-execute somewhere but anyway happy its that simple 🙂
y

Yee

08/03/2023, 3:49 PM
ohhhh
crap yeah i saw this yesterday too, didn’t make the connection.
j

Jan Fiedler

08/03/2023, 3:52 PM
No problem 😉
y

Yee

08/03/2023, 3:52 PM
we were wondering why something wasn’t working - fsspec was trying to list dir on an http endpoint that didn’t support it.
this was just for a put operation. it was needing to check to see if it was not a directory. feel like head would be more appropriate though, but yeah
thanks again
15 Views
Powered by