https://flyte.org logo
s

Stephan Gref

03/31/2022, 3:59 PM
Hey Flyte folks! 🙂 I am currently trying to get Flyte to work with Azure. I successfully configured "stow" config for admin and propeller and also managed to do an upload (fast-register) using an
abfs://
prefixed url. However, propeller produces a url like this for the pyflyte command:
pyflyte-execute --inputs <afs://flyte-data/metadata/propeller/test-azure-development-boygblti4g/n3/data/0/n0/inputs.pb>
which lead to the following error in flytekit:
Original exception: No plugin found for matching protocol of path <afs://flyte-data/metadata/propeller/test-azure-development-boygblti4g/n2/data/0/n0/0>
I believe the value
afs
is taken from https://github.com/graymeta/stow - however I could not identify the place in propeller yet. Did I miss how to configure this properly?
k

Ketan (kumare3)

03/31/2022, 4:16 PM
hi @Stephan Gref this is amazing - thank you so much. We are extremely excited about this work. Full community support from us
So couple questions, 1. In flytekit are you using the flytekitplugins-data-fsspec?
2. Did you install the azure blob store driver for fsspec?
s

Stephan Gref

03/31/2022, 4:18 PM
Sure, I use these versions:
Copy code
adlfs = "2022.2.0"
flytekit = "^0.30.0"
flytekitplugins-data-fsspec = "^0.30.0"
k

Ketan (kumare3)

03/31/2022, 4:19 PM
ok awesome, and it seems fsspec prefers abfs
s

Stephan Gref

03/31/2022, 4:19 PM
Uploading an
abfs
prefixed blob works - but
afs
seems not supported.
k

Ketan (kumare3)

03/31/2022, 4:19 PM
ya makes sense
s

Stephan Gref

03/31/2022, 4:19 PM
I did not find out where
afs
is coming from exactly
I know propeller replaces it, but not where it takes that constant from.
k

Ketan (kumare3)

03/31/2022, 4:20 PM
ya I do not think the flytebacked constructs the
afs
prefix. We use flytestdlib - storage
you are right, for the metadata, propeller is constructing the name using stow - the prefix path is controlled using this config
s

Stephan Gref

03/31/2022, 4:21 PM
ah yes - there it is: azure.Kind: func(bucket string) DataReference {
k

Ketan (kumare3)

03/31/2022, 4:22 PM
ya
k

Ketan (kumare3)

03/31/2022, 4:22 PM
but we can easily change this
we actually forked stow
also we have a storage layer on top of it
s

Stephan Gref

03/31/2022, 4:22 PM
that would be amazing 🙂
k

Ketan (kumare3)

03/31/2022, 4:22 PM
actually this is just in stdlib - ohh wow this is trivial.
you can open a PR to update it to abfs
else if you want I can
s

Stephan Gref

03/31/2022, 4:23 PM
I will do so.
So you say I can workaround this using
MetadataPrefix
in the meantime? By setting it to
abfs
or something like that?
k

Ketan (kumare3)

03/31/2022, 4:25 PM
no
no one uses afs
this is our work
s
7 Views