https://flyte.org logo
#flytekit
Title
# flytekit
r

Rahul Mehta

10/31/2022, 4:40 AM
Just noticed some interesting behavior in GitHub Actions when executing a launch plan during CI which had objects that used
PickleTransformer
passed as args to the LaunchPlan 🧵
Is there a particular reason
S3Persistence
is implemented using subprocess calls to awscli vs using boto3 to directly interact with S3? It appears that the version of awscli in our github actions runner image expects python 3.8, but python 3.9 is the only available distribution in the environment. Seems like this could be sidestepped entirely by avoiding the subprocess calls?
s

Samhita Alla

10/31/2022, 6:12 AM
cc: @Ketan (kumare3)
k

Ketan (kumare3)

10/31/2022, 6:21 AM
Yes this is legacy reason. As earlier we used subprocess, but we are actively recommending using fsspec - https://docs.flyte.org/projects/flytekit/en/latest/plugins/fsspec.html
You can install it using the data persistence plugins and we are thinking of making it default - and not breaking folks will be hard - cc @Eduardo Apolinario (eapolinario) / @Yee
@Samhita Alla can you point to docs of how to install flytekitplugins-data-fsspec[aws]
r

Rahul Mehta

10/31/2022, 6:23 AM
Got it, that makes sense. It’d be nice if it was the default, but understand not wanting to break workflows. Maybe something for the flytekit 2.x millstone?
k

Ketan (kumare3)

10/31/2022, 6:24 AM
Hmm we are actually thinking we can sidestep that - Flyte 1.3?
Would you be open to helping
let’s bring this up In a townhall
r

Rahul Mehta

10/31/2022, 6:24 AM
Sure, happy to chat more about this
k

Ketan (kumare3)

10/31/2022, 6:24 AM
Need help In benchmarking too
We are actually thinking of writing a faster layer
s

Samhita Alla

10/31/2022, 6:38 AM
flytekitplugins-data-fsspec[aws]
requires
flytekitplugins-data-fsspec
and
s3fs>=2021.7.0
libraries.
r

Rahul Mehta

10/31/2022, 2:03 PM
Thanks very much @Samhita Alla, will give that a try
2 Views