https://flyte.org logo
#ask-the-community
Title
# ask-the-community
b

Brian Tang

07/04/2023, 8:21 AM
we recently upgraded our flyte binary in EKS to
1.7.0
. we started the process of updating our docker images to use the latest
flytekit
as well with:
Copy code
RUN pip install boto3 awscli flytekit==1.7.0
but we’re getting conflicting dependencies related to `fsspec`:
Copy code
RROR: Cannot install flytekit and flytekit==1.7.0 because these package versions have conflicting dependencies.

The conflict is caused by:
    flytekit 1.7.0 depends on fsspec>=2023.3.0
    s3fs 2023.1.0 depends on fsspec==2023.1.0
this works if we downgraded back to
flytekit==1.5.0
s

Samhita Alla

07/05/2023, 4:56 AM
cc @Eduardo Apolinario (eapolinario)
k

Kevin Su

07/05/2023, 10:33 AM
flytekit 1.7 uses s3fs, so you don’t need to install boto3 and awscli
e

Eduardo Apolinario (eapolinario)

07/05/2023, 5:29 PM
@Brian Tang, do you have a specific use of boto3 and/or awscli in your flytekit images? Since flytekit 1.5 we moved the data persistence layer to rely solely on fsspec (which essentially means that we don't use boto3 nor awscli anymore as part of regular flytekit operations).
b

Brian Tang

07/06/2023, 1:06 AM
we need boto3 in the container image, as we’re interacting w dynamodb and s3 on our application layer. as for awscli, I did not know it’s no longer required after 1.5 - it’s there because it was previously a requirement for flyte container images
@Eduardo Apolinario (eapolinario) - i’ve tried removing
awscli
, i.e.
pip install boto3 flytekit==1.7.0
but still getting the deps conflict. we need
boto3
as it’s part of our application layer to interact w our aws infra
e

Eduardo Apolinario (eapolinario)

07/11/2023, 6:59 PM
we'll get back to you, @Brian Tang. Discussing this internally at the moment.
@Brian Tang, this conflict is caused by incompatible versions of botocore. Can you pin either
botocore==1.29.161
or
s3fs==2023.4.0
?
b

Brian Tang

07/12/2023, 2:30 AM
ok will test and get back to you, thanks Eduardo
7 Views