Hi all, I was wondering if anyone experienced any problems with using Azure Blob Storage as flyte storage? It works great for "normal" tasks, but when I define a dynamic workflow, then there are some additional python-tasks started before executing the actual workflow tasks, and those containers do not get azure credentials and are failing. Some more details: for "normal" raw-container tasks, there will be a downloader (InitContainer) and a co-pilot (SideCar Container) - both using the same image

flyte-copilot

. Both the downloader and a co-pilot receive automatically command-line arguments

--storage.stow.config account=<AzureStorageAccount>

and

--storage.stow.config key=<AzureAccessKey>

which then allow the contianers to download and upload the data to Azure. However marking a workflow as @dynamic leads to some additional python-tasks (i.e. I do not define them explicitely in my workflow); those use the

flytekit

image and the corresponding executable are of course very different and do not get any credentials, but still try to download some data from Azure. As a result the task fails with sth like

[1/1] currentAttempt done. Last Error: USER::
[almhfbtcdk4tzqfwxpqt-n0-0] terminated with exit code (1). Reason [Error]. Message: 
ct to account for Must provide either a connection_string or account_name with credentials!!

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/bin/pyflyte-fast-execute", line 8, in <module>
    sys.exit(fast_execute_task_cmd())
             ^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/click/core.py", line 1157, in __call__
    return self.main(*args, **kwargs)

which is of course to be expected if no credentials are passed to the container. Has anybody seen this problem? Is it possible to provide the necessary credentials to the flytekit container somehow? Alternatively, is it possible to avoid these extra python-tasks to be started at all? (but retaining the dynamic nature of the workflow? Thanks a lot for any advice!

So dynamic needs a python task to run that generates the workflow to run

Thanks for your quick response! So flytekit is using fsspec, correct? And fsspec should be able to get the credentials, as long as they are provided by the service account, correct? Do you mean the k8s ServiceAccount here?

yes ServiceAccount

Hi @freezing-airport-6809, I included the environmental variables AZURE_STORAGE_ACCOUNT_NAME and AZURE_STORAGE_ACCOUNT_KEY to the pod and access seems to work now! However, there seems to be a new error:

{"asctime": "2024-07-08 12:00:44,343", "name": "flytekit", "levelname": "ERROR", "message": "Exception when executing task workflows.test-training.test_training_wf, reason 'PythonFunctionWorkflow' object has no attribute 'dispatch_execute'"}
{"asctime": "2024-07-08 12:00:44,343", "name": "flytekit", "levelname": "ERROR", "message": "!! Begin Unknown System Error Captured by Flyte !!"}
{"asctime": "2024-07-08 12:00:44,343", "name": "flytekit", "levelname": "ERROR", "message": "Traceback (most recent call last):\n  File \"/usr/local/lib/python3.11/site-packages/flytekit/bin/entrypoint.py\", line 99, in _dispatch_execute\n    outputs = _scoped_exceptions.system_entry_point(task_def.dispatch_execute)(ctx, idl_input_literals)\n                                                    ^^^^^^^^^^^^^^^^^^^^^^^^^\nAttributeError: 'PythonFunctionWorkflow' object has no attribute 'dispatch_execute'\n"}
{"asctime": "2024-07-08 12:00:44,343", "name": "flytekit", "levelname": "ERROR", "message": "!! End Error Captured by Flyte !!"}

Seems more like an internal error.. Perhaps not related to Azure at all (dispatch_execute seems to be very basic functionality of flytekit https://docs.flyte.org/en/latest/api/flytekit/generated/flytekit.PythonInstanceTask.html) Any ideas? Thanks!

Woah what? That is indeed weird? Can you help with the code snippet for us to reproduce

@freezing-airport-6809 This might be tricky... I could try to build a minimal example (we have a pretty complex setup with custom images in a private container registry etc. etc.), but what about testing this with azure? I guess I'd have to set up some dummy (public) storage account or sth like that. Or am I making things too complicated here? :-D

You cannot use workflow and dynamic both