Scratching my head here a bit... Does anyone use flytekit + flytekitplugins-data-fsspec + fast register? It seems that fsspec does not handle the the default destination
.
(dot, like current directory) well and does not download the tar.gz at all.
Reproduced this with this minimal example:
Copy code
import os
from flytekit.tools.fast_registration import download_distribution as _download_distribution
os.environ["AWS_ACCESS_KEY_ID"] = "key_id"
os.environ["AWS_SECRET_ACCESS_KEY"] = "secret_access_key"
destination = "."
_download_distribution(
additional_distribution="<s3://bucket/fast-register.tar.gz>",
destination=".")
It works if I replace
destination
with is
./
in this example. The issue is, that
.
is set by propeller (this is at least my understanding). Is there any way I can configure that?
👀 1
b
boundless-pizza-95864
05/19/2022, 6:38 PM
I did try it a while a go but I ran into issues so I switched back. If I remember correctly it was in combination with fast register (regular download/upload worked fine) so it could be the same issue.
f
freezing-airport-6809
05/20/2022, 4:00 AM
@sticky-printer-535 I do not think propeller sets the path
freezing-airport-6809
05/20/2022, 4:00 AM
the path is set by flytekit itself and should be safe to update