Hello Flyte community, related to Flytefile usage ...
# ask-the-community
n
Hello Flyte community, related to Flytefile usage - would it be feasible to handle S3 interactions via
boto3
instead of
aws s3
subprocesses? The reason being that
awscli
does not play nice with other dependencies I have, so I wanted to know if migrating that code to a boto client is an option to be able to drop
awscli
. Thanks in advance!
a
Not an answer to your question, but just a note that if you install awscli v2 or install awscli v1 via
pipx
it will not interfere with your other env dependencies. I believe there is an overhaul to the data persistence layer being worked on which could make a more fundamental change like what you're suggesting, but I don't know the details.
k
Yes this overhaul is in works and makes the base layer fsspec and uses arrow as the medium
But today we already have support for fsspec using the flytekit-data-fsspec plugin
102 Views