https://flyte.org logo
s

Stephan Gref

05/19/2022, 5:15 PM
Scratching my head here a bit... Does anyone use flytekit + flytekitplugins-data-fsspec + fast register? It seems that fsspec does not handle the the default destination
.
(dot, like current directory) well and does not download the tar.gz at all. Reproduced this with this minimal example:
Copy code
import os

from flytekit.tools.fast_registration import download_distribution as _download_distribution

os.environ["AWS_ACCESS_KEY_ID"] = "key_id"
os.environ["AWS_SECRET_ACCESS_KEY"] = "secret_access_key"

destination = "."

_download_distribution(
    additional_distribution="<s3://bucket/fast-register.tar.gz>",
    destination=".")
It works if I replace
destination
with is
./
in this example. The issue is, that
.
is set by propeller (this is at least my understanding). Is there any way I can configure that?
👀 1
s

Sören Brunk

05/19/2022, 6:38 PM
I did try it a while a go but I ran into issues so I switched back. If I remember correctly it was in combination with fast register (regular download/upload worked fine) so it could be the same issue.
k

Ketan (kumare3)

05/20/2022, 4:00 AM
@Stephan Gref I do not think propeller sets the path
the path is set by flytekit itself and should be safe to update
5 Views