Flyte enables production-grade orchestration for machine learning workflows and data processing created to accelerate local workflows to production.

Flyte

Just noticed some interesting behavior in GitHub Actions when executing a launch plan during CI which had objects that used `PickleTransformer` passed as args to the LaunchPlan :thread:

Is there a particular reason `S3Persistence` is implemented using subprocess calls to awscli vs using boto3 to directly interact with S3? It appears that the version of awscli in our github actions runner image expects python 3.8, but python 3.9 is the only available distribution in the environment. Seems like this could be sidestepped entirely by avoiding the subprocess calls?

Yes this is legacy reason. As earlier we used subprocess, but we are actively recommending using fsspec - <https://docs.flyte.org/projects/flytekit/en/latest/plugins/fsspec.html|https://docs.flyte.org/projects/flytekit/en/latest/plugins/fsspec.html>

You can install it using the data persistence plugins and we are thinking of making it default - and not breaking folks will be hard - cc <@U0265RTUJ5B> / <@UNR3C6Y4T> 

<@U01J90KBSU9> can you point to docs of how to install flytekitplugins-data-fsspec[aws]

Got it, that makes sense. It’d be nice if it was the default, but understand not wanting to break workflows. Maybe something for the flytekit 2.x millstone?

Hmm we are actually thinking we can sidestep that - Flyte 1.3?

We are actually thinking of writing a faster layer

`flytekitplugins-data-fsspec[aws]` requires `flytekitplugins-data-fsspec` and `s3fs&gt;=2021.7.0` libraries.

Thanks very much <@U01J90KBSU9>, will give that a try