we recently upgraded our flyte binary in EKS to

1.7.0

. we started the process of updating our docker images to use the latest

flytekit

as well with:

RUN pip install boto3 awscli flytekit==1.7.0

but we’re getting conflicting dependencies related to `fsspec`:

RROR: Cannot install flytekit and flytekit==1.7.0 because these package versions have conflicting dependencies.

The conflict is caused by:
    flytekit 1.7.0 depends on fsspec>=2023.3.0
    s3fs 2023.1.0 depends on fsspec==2023.1.0

this works if we downgraded back to

flytekit==1.5.0

cc @Eduardo Apolinario (eapolinario)

flytekit 1.7 uses s3fs, so you don’t need to install boto3 and awscli

@Brian Tang, do you have a specific use of boto3 and/or awscli in your flytekit images? Since flytekit 1.5 we moved the data persistence layer to rely solely on fsspec (which essentially means that we don't use boto3 nor awscli anymore as part of regular flytekit operations).

we need boto3 in the container image, as we’re interacting w dynamodb and s3 on our application layer. as for awscli, I did not know it’s no longer required after 1.5 - it’s there because it was previously a requirement for flyte container images

we'll get back to you, @Brian Tang. Discussing this internally at the moment.

ok will test and get back to you, thanks Eduardo

Was this ever resolved? We have a need for Boto3 in our workflows and I can't seem to find a combination of versions of boto3 and flytekit that work. Downgrading Flytekit to 1.5.0 allows me to install boto3 but when I try to run pyflyte to register the workflow I get an errors, then when I upgrade Flytekit to a more recent version it works. Any assistance/suggestions would be greatly appreciated. Thank you