#1842 Better Azure blob storage support Pull request opened by Tom-Newton TL;DR Support Azure blob storage for storing metadata and raw data including structured datasets, without interrupting other standard Azure authentication. Also ensure

storage_options

are set consistently for all uses of fsspec. Type Its debatable. The original github issue was a bug for fixing Azure authentication but it could be argued that Azure blob storage support is a new feature. ☑︎ Bug Fix ☑︎ Feature ☐ Plugin Are all requirements met? ☑︎ Code completed ☑︎ Smoke tested - Not really sure what is expected but I have used these change to deploy real workflows. ☑︎ Unit tests added - I added a bit of new coverage, e.g. initialising the

AzrueBlobFilesystem

. ☑︎ Code documentation added - There are code comments, but I think ideally there would be a page in the documentation for Azure. ☑︎ Any pending items have an associated Issue - There aren't any Complete description Added a new component to

DataConfig

for Azure similar to the ones for S3 and GCS. This allows configuring the flytekit's storage account without disrupting other Azure auth that may be used by flyte task implementations. This is done by setting the environment variables

FLYTE_AZURE_STORAGE_ACCOUNT_NAME

,

FLYTE_AZURE_STORAGE_ACCOUNT_KEY

,

FLYTE_AZURE_TENANT_ID

,

FLYTE_AZURE_CLIENT_ID

,

FLYTE_AZURE_CLIENT_SECRET

. You can set these using a pod template so that authentication to your flyte storage account works across your whole flyte deployment, inrespective of what authentication each task uses. Use

DataConfig

to construct fsspec storage options in

data_persistence.py

. We reuse the same function for fetchign

storage_options

elsewhere inlcuding for encode and decode of structured datasets. This ensures local, S3, GCS and Azure work wherever fsspec is used. You should specify

configmap.core.propeller.rawoutput-prefix

to be in the format

abfs://<container-name>/<path-within-container>

. Tracking Issue Related discussion was on flyteorg/flyte#3962 but technically that issues was about workload identity which was mostly fixed by #1813 Follow-up issue NA flyteorg/flytekit ✅ All checks have passed 2/2 successful checks

#1842 Better Azure blob storage support Pull request ready for review by Tom-Newton flyteorg/flytekit