Hey, i was just trying to upgrade my flytekit vers...
# ask-the-communityj
Jan Fiedler
07/31/2023, 3:53 PMHey, i was just trying to upgrade my flytekit version in of my workflows and as a result i get a S3 Access denied as a result. With the old dependencies everything works as expected.
These are my old dependencies:
Copy code
s3fs==0.4.2
flytekit==1.3.2
awscli==1.27.118
mlstuff Copy code
fsspec==2023.6.0
s3fs==2023.6.0
flytekit==1.8.1
awscli==1.27.161
mlstuffy
Yee
07/31/2023, 4:28 PM
what’s the error?
Yee
07/31/2023, 4:29 PM
1.3 is pretty old, a fair bit has changed.
Yee
07/31/2023, 4:29 PM
but irsa should work fine. are you sure the role has access? user containers sometimes require different permissions than propeller.
j
Jan Fiedler
07/31/2023, 5:25 PMThats the Error:
Copy code
FlyteAssertion: Failed to get data from
<s3://kineo-flyte-central-cluster-bucket/demo/development/YHTQ6ZBZFBCHRZMUFRW>
UQ3MGIE======/fast37dd06657c173c4af3b785092c82a7eb.tar.gz to /root/
(recursive=False).
Original exception: Access Denied
.Jan Fiedler
07/31/2023, 5:25 PMQuite straight forward but with the same role on flytekit 1.3 i have access
y
Yee
07/31/2023, 5:51 PM
can you inspect the task pod? just to double check it’s running with the right service account? are there any config files the container is relying on?
Yee
07/31/2023, 5:52 PM
the main change between 1.3 and 1.8 wrt data handling is usage of fsspec/s3fs.
Yee
07/31/2023, 5:52 PM
but both the aws cli and s3fs ultimately fall back to boto for credentials handling.
j
Jan Fiedler
07/31/2023, 5:57 PMI might be able to check tomorrow again but i iam pretty sure i saw the default service account with the correct role annotation.
y
Yee
07/31/2023, 5:59 PM
k
j
Jan Fiedler
07/31/2023, 5:59 PMFor some reason i had the feeling the role could not be assumed at all. Because i saw also the role itself not updating for ‘last activity’ in AWs iam
y
Yee
07/31/2023, 5:59 PM
i see
Yee
07/31/2023, 5:59 PM
paste more errors too if you have them. also i think you can drop the awscli requirement.
j
Jan Fiedler
07/31/2023, 6:00 PMAlright, i will. Thank you Yee for pointing me in the right direction!
Jan Fiedler
08/03/2023, 12:08 PMAlright it was indeed just a matter of AWS IAM. It looks like flytekit 1.3 only needs the
s3:GetObjects3:ListBuckety
Yee
08/03/2023, 3:49 PM
ohhhh
Yee
08/03/2023, 3:49 PM
crap yeah i saw this yesterday too, didn’t make the connection.
j
Jan Fiedler
08/03/2023, 3:52 PMNo problem 😉
y
Yee
08/03/2023, 3:52 PM
we were wondering why something wasn’t working - fsspec was trying to list dir on an http endpoint that didn’t support it.
Yee
08/03/2023, 3:53 PM
this was just for a put operation. it was needing to check to see if it was not a directory. feel like head would be more appropriate though, but yeah
Yee
08/03/2023, 3:53 PM
thanks again
60 Views