Stephan Gref

    Stephan Gref

    5 months ago
    Hey Flyte folks! 🙂 I am currently trying to get Flyte to work with Azure. I successfully configured "stow" config for admin and propeller and also managed to do an upload (fast-register) using an
    abfs://
    prefixed url. However, propeller produces a url like this for the pyflyte command:
    pyflyte-execute --inputs <afs://flyte-data/metadata/propeller/test-azure-development-boygblti4g/n3/data/0/n0/inputs.pb>
    which lead to the following error in flytekit:
    Original exception: No plugin found for matching protocol of path <afs://flyte-data/metadata/propeller/test-azure-development-boygblti4g/n2/data/0/n0/0>
    I believe the value
    afs
    is taken from https://github.com/graymeta/stow - however I could not identify the place in propeller yet. Did I miss how to configure this properly?
    Ketan (kumare3)

    Ketan (kumare3)

    5 months ago
    hi @Stephan Gref this is amazing - thank you so much. We are extremely excited about this work. Full community support from us
    So couple questions,1. In flytekit are you using the flytekitplugins-data-fsspec?
    2. Did you install the azure blob store driver for fsspec?
    Stephan Gref

    Stephan Gref

    5 months ago
    Sure, I use these versions:
    adlfs = "2022.2.0"
    flytekit = "^0.30.0"
    flytekitplugins-data-fsspec = "^0.30.0"
    Ketan (kumare3)

    Ketan (kumare3)

    5 months ago
    ok awesome, and it seems fsspec prefers abfs
    Stephan Gref

    Stephan Gref

    5 months ago
    Uploading an
    abfs
    prefixed blob works - but
    afs
    seems not supported.
    Ketan (kumare3)

    Ketan (kumare3)

    5 months ago
    ya makes sense
    Stephan Gref

    Stephan Gref

    5 months ago
    I did not find out where
    afs
    is coming from exactly
    I know propeller replaces it, but not where it takes that constant from.
    Ketan (kumare3)

    Ketan (kumare3)

    5 months ago
    ya I do not think the flytebacked constructs the
    afs
    prefix. We use flytestdlib - storage
    you are right, for the metadata, propeller is constructing the name using stow - the prefix path is controlled using this config
    Stephan Gref

    Stephan Gref

    5 months ago
    ah yes - there it is: azure.Kind: func(bucket string) DataReference {
    Ketan (kumare3)

    Ketan (kumare3)

    5 months ago
    ya
    Ketan (kumare3)

    Ketan (kumare3)

    5 months ago
    but we can easily change this
    we actually forked stow
    also we have a storage layer on top of it
    Stephan Gref

    Stephan Gref

    5 months ago
    that would be amazing 🙂
    Ketan (kumare3)

    Ketan (kumare3)

    5 months ago
    actually this is just in stdlib - ohh wow this is trivial.
    you can open a PR to update it to abfs
    else if you want I can
    Stephan Gref

    Stephan Gref

    5 months ago
    I will do so.
    So you say I can workaround this using
    MetadataPrefix
    in the meantime? By setting it to
    abfs
    or something like that?
    Ketan (kumare3)

    Ketan (kumare3)

    5 months ago
    no
    no one uses afs
    this is our work
    Stephan Gref

    Stephan Gref

    5 months ago