Hi folks I am attempting to set up azure blob through stow. My values for storage settings looks like so:

storage:
  type: custom
  enableMultiContainer: true
  limits:
    maxDownloadMBs: 500000
    type: custom
  bucketName: "{{ .Values.userSettings.azure.containerName }}"
  custom:
    container: "{{ .Values.userSettings.azure.containerName }}"
    enable-multicontainer: true
    connection: {}
    type: stow
    stow:
      kind: azure
      config:
        account: "{{ .Values.userSettings.azure.storageAccountName }}"
        key: "{{ .Values.userSettings.azure.storageAccountKey }}"

Trying to run the init workflow examples throws:

Traceback (most recent call last):                                                                                                                                                                                                                            │
│   File "/opt/venv/bin/pyflyte-execute", line 8, in <module>                                                                                                                                                                                                   │
│     sys.exit(execute_task_cmd())                                                                                                                                                                                                                              │
│   File "/opt/venv/lib/python3.8/site-packages/click/core.py", line 1130, in __call__                                                                                                                                                                          │
│     return self.main(*args, **kwargs)                                                                                                                                                                                                                         │
│   File "/opt/venv/lib/python3.8/site-packages/click/core.py", line 1055, in main                                                                                                                                                                              ││     rv = self.invoke(ctx)                                                                                                                                                                                                                                     ││   File "/opt/venv/lib/python3.8/site-packages/click/core.py", line 1404, in invoke                                                                                                                                                                            ││     return ctx.invoke(self.callback, **ctx.params)                                                                                                                                                                                                            ││   File "/opt/venv/lib/python3.8/site-packages/click/core.py", line 760, in invoke                                                                                                                                                                             ││     return __callback(*args, **kwargs)                                                                                                                                                                                                                        ││   File "/opt/venv/lib/python3.8/site-packages/flytekit/bin/entrypoint.py", line 470, in execute_task_cmd                                                                                                                                                      ││     _execute_task(                                                                                                                                                                                                                                            ││   File "/opt/venv/lib/python3.8/site-packages/flytekit/exceptions/scopes.py", line 160, in system_entry_point                                                                                                                                                 ││     return wrapped(*args, **kwargs)                                                                                                                                                                                                                           ││   File "/opt/venv/lib/python3.8/site-packages/flytekit/bin/entrypoint.py", line 348, in _execute_task                                                                                                                                                         ││     _handle_annotated_task(ctx, _task_def, inputs, output_prefix)                                                                                                                                                                                             ││   File "/opt/venv/lib/python3.8/site-packages/flytekit/bin/entrypoint.py", line 291, in _handle_annotated_task                                                                                                                                                ││     _dispatch_execute(ctx, task_def, inputs, output_prefix)                                                                                                                                                                                                   ││   File "/opt/venv/lib/python3.8/site-packages/flytekit/bin/entrypoint.py", line 160, in _dispatch_execute                                                                                                                                                     ││     ctx.file_access.put_data(ctx.execution_state.engine_dir, output_prefix, is_multipart=True)                                                                                                                                                                ││   File "/opt/venv/lib/python3.8/site-packages/flytekit/core/data_persistence.py", line 476, in put_data                                                                                                                                                       ││     raise FlyteAssertion(                                                                                                                                                                                                                                     ││ flytekit.exceptions.user.FlyteAssertion: Failed to put data from /tmp/flyte-z58pqpy5/sandbox/local_flytekit/engine_dir to <abfs://flyte-workflows/metadata/propeller/>...

Original exception: No plugin found for matching protocol of path <abfs://flyte-workflows/metadata/propeller/>...

Are there specific steps to enable/support the abfs protocol, beyond configuring stow as above?

Hi @Mathias Andersen! your configuration looks fine to me, we're using basically the same (expect you have an extra

type: custom

in the

limits

blocks that should not be required, I believe. We're using a fork of adlfs in our flyte deployments since some

FlyteDirectory

handling was broken with the upstream version (our PR has finally been merged, but no new release was created yet, so we still couldn't contribute it back to upstream flytekit), however that should not be the issue you're seeing right here.

Good observations. Thanks! - I'm running 1.4 and will give 1.1 a go. Should I consider which chart version I use for flyte-core deployment as well?

I'm honestly not sure what version of the rest of the flyte components you'll have to pick in order to be compatible with flytekit 1.1, we're running an internal fork of most components to allow for in-house flyte development before we push changes back upstream, so we're a bit out of sync with the upstream. we're also quite a few months back on updates due to lack of time on my side 😅

Roger that. I'll give it a jab and report back. Maybe we should add some docs for experimental AKS/azure blob setup? - Just to indicate the state of maturity and (hopefully) a known path to something that isn't smoking 😄

that's been on my todo list for a while now too (adding some basic docs), but I haven't gotten around to doing so unfortunately 🙈

Changing to flytekit 1.1 swapped the protocol to HTTPS, but results in response 400.

pyflyte run --remote workflows/example.py wf --name 'hello'

File "/home/user/.local/share/virtualenvs/first-attempt-to-flyte-Ae80d762/lib/python3.8/site-packages/flytekit/core/data_persistence.py", line 447, in put_data
    raise FlyteAssertion(
flytekit.exceptions.user.FlyteAssertion: Failed to put data from /tmp/tmprmqf8xzd/script_mode.tar.gz to <https://my.blob.core.windows.net/flyte-workflows/flytesnacks/development/FOVDCPZCJ447LTKJTX5HCLBKPA======/scriptmode.tar.gz?se=2023-04-04T13%3A50%3A53Z&sig=V7%2B9c7Y5e4Oc0sf2XbxVXOtWssJIdCmxst4wtWLzzyI%3D&sp=aw&spr=https&sr=b&sv=2019-12-12> (recursive=False).

Original exception: Value error!  Received: 400. Request to send data failed.

So @Nick Müller (MorpheusXAUT) @Mathias Andersen Flyte backend and flytekit are no tightly dependent on a version

the only bug we encountered with fsspec or adlfs more precisely was fixed by this: https://github.com/fsspec/adlfs/pull/398 unfortunately there's no new tag yet, so it's not fully available. but that should not cause the issue Mathias is seeing

@Nick Müller (MorpheusXAUT) has been promising me to get azure support upstreamed, but I have not seen anything and this has been months

we've been waiting for the adlfs fix to finally be merged, had another PR open for a few months before that - without that,

FlyteDirectory

didn't work properly, so contributing all changes upstream would've not been very useful

Also @Nick Müller (MorpheusXAUT) you should get onto flytekit 1.5 beta. Can you help test it. Supports lots of cool things Streaming files and directories Multi list maps Better caching on map tasks Pyflyte run multi file etc

I'd love to, there's been people requesting an update to a newer version anyways, unfortunately my time is quite limited atm. but that's a different topic/not related to Mathias' issue

I tired a bad key, which causes pods to fail at startup. I just tried degrading to flyte-core 1.1.0 as well, same issue, but I also note that when running pyflyte run --remote, I get the respose 400 on HTTPS error, however when I package and register and execute the workflow on flyte, it throws the plugin (for abfs) error.

Ohh ya you have to install 2 additional dependencies

I see. In the package image?

adlfs, not abfs, right? 🤔 at least that's what we've been using so far to handle the azure blob storage protocol

Roger

Ohh we recommend using serviceaccount to get the role On AWS we use eks serviceaccount for Iam role, On gcp workload identity On azure @Nick Müller (MorpheusXAUT)

Azure has pod identity (still in preview, as usual on Azure, but already being superseded 😄) and workload identity (also preview, but the new way to do things), however we have not used them in combination with Flyte yet, so I don't have any experience in that regard unfortunately

It should not be cumbersome - you can use default pod templates for this right

oh good point, fair enough, completely forgot about that. so yeah, if fsspec supports that (I'd have to check, don't know off the top of my head), you could also try mounting a secret with the service principal credentials to your pod and it might be able to pick it up and authenticate using that

fsspc+adlfs does indeed implement MSAL through the azure-identity package. I am now able to run the init example workflow on flyte without errors. - I see no "output" files on blob. I expected the output to be stored like the input, but can't spot it in azure storage explorer. Thank you for your help this far. - I think the remaining work on my end will be to implement workload identities in a nice manner and document the setup. Hopefully we will be able to share a getting started preview on AKS 🙂

Hey, yep I am getting similar error Received 400: Request to send data ... failed when running

pyflyte run --remote

I am on flyte 1.4.3 and flytekit 1.4.2. I am not sure if this is because the signed url collides with blob url dns.

I investigated if flytekit just needs some configuration locally, for targeting azure blob storage via fsspec and abfs//, but I have not found the "missing link"

Thank you for raising this, we should not need data plugin anymore

@Jeongwon SongYou can see a working storage setup towards azure blob in OP. It is fairly basic on the azure side of things. You may also need to config propeller to point raw data (different from meta data) to this blob storage: https://github.com/flyteorg/flyte/blob/master/charts/flyte-core/values-eks.yaml#L228 should be abfs://{{ .Values.userSettings.bucketName }}/ in the AKS/Azure blob scenario. So, the abfs// protocol name points flytekit towards the sfspc/adlf data layer plugin, which attempt to auth via azure identity/defaultcredentials. As a test I just added the storage account name and key to a custom contaniers env vars, but there is a nicer solution.

hey @Mathias Andersen do you have a sec to help me understand this?

It looks to me like you set it up correctly, but fsspec/adfls unsuccessfully looks for account_name and creds in the container. How you choose to pass these credentials is up to you: https://github.com/fsspec/adlfs#setting-credentials