acoustic-carpenter-78188
07/11/2024, 5:34 PM@workflow
def test_workflow(directory: FlyteDirectory):
...
When using Azure, launching this workflow with a directory containing more than one file causes
Invalid value for '--directory': Failed to convert param: <Option directory>, value: /path/on/local/ to type: <class 'flytekit.types.directory.types.FlyteDirectory'>. Reason USER:AssertionError: error=Failed to put data from │
│ /path/on/local/ to <flyte://data/10ba872c9015435996c14abeeb15f846> (recursive=True). │
│ │
│ Original exception: unable to connect to account for Must provide either a connection_string or account_name with credentials!!, cause=unable to connect to account for Must provide either a connection_string or account_name with credentials!!
Expected behavior
I would expect this to run without error.
Root cause
The files are actually uploaded fine using FlyteFS
. The problem is when it calls FlyteFS.extract_common()
which comes after successful upload. This contains the snippet
fs = fsspec.filesystem(get_protocol(native_urls[0]))
sep = fs.sep
The problem is that adlfs.AzureBlobFileSystem()
fails because the account_name
is unknown. I'm thinking it would be best to avoid needing to initialise an fsspec
filesystem just to determine the separator used on the filesystem in question.
Additional context to reproduce
No response
Screenshots
No response
Are you sure this issue hasn't been raised already?
☑︎ Yes
Have you read the Code of Conduct?
☑︎ Yes
flyteorg/flyteacoustic-carpenter-78188
07/11/2024, 5:34 PM