Terence Kent
09/19/2023, 2:06 PMstow
that adds Azure AD support (flyteorg/stow/pull/9, related convo here).
Part of the effort in shoring up that PR is testing the stow
branch in a Flyte deployment. The changes are tested within the stow
project, but being able to try it all together would be a nice early check.
Is that something somebody here is interesting is trying out? Or, should I just follow the development guide.David Espejo (he/him)
09/19/2023, 2:33 PMTerence Kent
09/19/2023, 4:32 PMflyteorg/stow
branch I've setup. Even deploying that locally (not in Azure) and having it interact with azure storage accounts would be useful.
If you (or anyone else) has an interest in doing that loop, I'd appreciate it! There is now a terraform module in my flyteorg/stow
branch for setting up an Azure storage account for testing.
If not, that's fine too! I just probably won't get around to sorting out a local build until late this week or next. Just a lot of other things to do 🙂.stow
implementation to use my local branch.
The good news is that stow
works just fine and the dependency upgrades appear to cause no issues in the flyte
project. On startup, using either AD auth or a shared key, a container with the name specified under storage.container
is correctly created. When running a workflow, some flytesnacks/...
blobs are created, so that's clearly working.
The bad news is that all my test workflows fail due to an exception in flytekit
. I haven't chased this down yet, but the ffspec
paths being created are rejected with the an error of:
Must provide either a connection_string or account_name with credentials!!
The fsspec created looks like:
<abfs://myflytetest/flytesnacks/development/VQMFYT6TQOXW63XGU3I2GO33IE======/scri>
pt_mode.tar.gz
Maybe somebody here already knows what this is. Otherwise, this will need to wait until I can walk through the python library to see what's going on.Samhita Alla
Terence Kent
09/20/2023, 4:48 PMAZURE_STORAGE_CONNECTION_STRING
and then see if there is an easier way to manage the required configs.