https://flyte.org logo
#ask-the-community
Title
# ask-the-community
v

Victor Delépine

08/10/2023, 4:09 PM
Hello! Back with more Azure adventures 🙂 I've managed to trigger a basic workflow from
flytesnacks
now, but in the UI I see the following error for my execution:
Copy code
FlyteAssertion: Failed to get data from 
<abfs://flyte/flytesnacks/development/A46TBP35TISQ2EVJVNOGLJTARM======/script_mod>
e.tar.gz to /root/ (recursive=False).

Original exception: unable to connect to account for Must provide either a 
connection_string or account_name with credentials!!
Here's my storage config in the helm chart:
Copy code
storage:
  type: custom
  bucketName: "flyte"
  custom:
    container: "flyte"
    enable-multicontainer: true
    connection: {}
    type: stow
    stow:
      kind: azure
      config:
        account: "<storage_account_name>"
        key: "<storage_account_key>"
  limits:
    maxDownloadMBs: 10
Does anyone know what would cause this? My first intuition would be that this URI is wrong:
<abfs://flyte/flytesnacks/development/A46TBP35TISQ2EVJVNOGLJTARM======/script_mode.tar.gz>
because in Azure the path should look like this:
abfs://<CONTAINER>@<STORAGE_ACCOUNT>.<http://dfs.core.windows.net/flytesnacks/development/A46TBP35TISQ2EVJVNOGLJTARM======/script_mode.tar.gz|dfs.core.windows.net/flytesnacks/development/A46TBP35TISQ2EVJVNOGLJTARM======/script_mode.tar.gz>
🤔 reading this: https://github.com/fsspec/adlfs/blob/main/adlfs/spec.py#L115 It seems like that is something that
fsspec
is supposed to handle. So somehow it cannot find my credentials
k

Kevin Su

08/10/2023, 5:17 PM
does you pod have IAM role to access data in abfs?
k

Ketan (kumare3)

08/10/2023, 8:30 PM
Yes this is delegated to fsspec. But we have heard fsspec azure file system has some bugs from some community folks but it was fixed in a later version
v

Victor Delépine

08/10/2023, 10:55 PM
Could be an issue with my pod’s role yeah 🤔 . But why would I be able to create the object and not retrieve it? Looking at the fsspec code, it seems like it cannot find my credentials. I’ll double check and report back!
g

Gopal Vashishtha

08/11/2023, 2:53 AM
Victor I was able to resolve this issue by setting an environment variable in my pod template:
Copy code
env:
          - name: AZURE_STORAGE_CONNECTION_STRING
            valueFrom:
              secretKeyRef:
                name: flyteruntimesecret
that
AZURE_STORAGE_CONNECTION_STRING
value came from the source code you linked above
k

Ketan (kumare3)

08/11/2023, 4:36 AM
is this because Azure does not have something like IAM roles for Service accounts or Workload identity?
v

Victor Delépine

08/11/2023, 6:56 AM
Oh nice! For which pod did you do that? I tried it earlier but you can only pass extra env variables to flyteadmin in the helm chart, and I think my error is in propeller
As to why, Azure does have Workload identity now but it’s not enabled on my cluster, I could give it another try with it enabled. I was counting on the account name and key to be propagated to pods somehow (from storage.yaml)
Thanks for your help !-
Oh I see what you mean @Gopal Vashishtha You used a pod template that defines that variable directly
I couldn't manage to apply a pod template via the helm values, as I need to give permissions to
flyteadmin
via a role binding beforehand. That probably means I need to set up workload identity in my cluster, so I'll give that a go
How does the cluster resource sync code authenticate with Azure? Does it support workload identity? I'm getting errors saying it's unable to create my
PodTemplate
despite me giving permissions to the
flyteadmin
service account
Copy code
{
  "json": {},
  "level": "warning",
  "msg": "Failed to create kubernetes object from config template [pod_template.yaml] for namespace [flytesnacks-development] with err: podtemplates is forbidden: User \"system:serviceaccount:flyte:flyteadmin\" cannot create resource \"podtemplates\" in API group \"\" in the namespace \"flytesnacks-development\"",
  "ts": "2023-08-11T15:41:02Z"
}
Copy code
{
  "json": {},
  "level": "warning",
  "msg": "Failed to create cluster resources for namespace [flytesnacks-development] with err: Failed to create kubernetes object from config template [pod_template.yaml] for namespace [flytesnacks-development] with err: podtemplates is forbidden: User \"system:serviceaccount:flyte:flyteadmin\" cannot create resource \"podtemplates\" in API group \"\" in the namespace \"flytesnacks-development\"",
  "ts": "2023-08-11T15:41:02Z"
}
Oh I think I get it. This uses
flytek8s
which has everything it needs to create resources in the cluster. The issues I'm having are related to a chicken and egg problem I have with terraform. The helm chart creates the flyteadmin service account, but instead I should create it via terraform and do the role assignments before the helm chart is applied. Sorry if I'm rambling 😅
k

Ketan (kumare3)

08/12/2023, 4:31 AM
Omg azure @Gopal Vashishtha and other can someone please help us write a working azure override
v

Victor Delépine

08/12/2023, 6:03 AM
Will definitely write something once I have a working setup
t

tk

09/20/2023, 10:57 PM
To dust off this thread a bit, I also bumped into this issue when testing in connection with flyteorg/stow/pull/9. I was pointed here by Samhita (thanks for that, btw). I can add a little more context to the situation. While adding the
AZURE_STORAGE_CONNECTION_STRING
env var to all task pods will definitely work, it's not the only path. Adding
AZURE_STORAGE_ACCOUNT_NAME
to all pods will also work, as long as the credentials can be loaded from the usual spots which you can find here.
d

dzianisskakalski81

02/19/2024, 6:23 PM
hi guys, I have faced the same issue described above. I have kubernetes cluster in Azure cloud, now I have created storage account and PostgresSQL, everything have started successfully, but during executing workflow I am getting such errors in logs, ---------------------- FlyteAssertion: Failed to get data from abfs://my-flyte-container/flytesnacks/development/I62Z4WQMINA7777777777RU==== ==/script_mode.tar.gz to /root/ (recursive=False). Original exception: unable to connect to account for Must provide either a connection_string or account_name with credentials!! ----------------------- I have edited the deployment (flyte-backend-flyte-binary) and manually added the ENV variable to the config for temporary testing, but no success spec: containers: - args: - start - --config - /etc/flyte/config.d/*.yaml env: - name: POD_NAME valueFrom: fieldRef: apiVersion: v1 fieldPath: metadata.name - name: POD_NAMESPACE valueFrom: fieldRef: apiVersion: v1 fieldPath: metadata.namespace - name: AZURE_STORAGE_CONNECTION_STRING value: abfs://my-flyte-container@flyte.dfs.core.windows.net/flytesnacks/development/I62Z4WQMINAWG7777777RU======/script_mode.tar.gz Could you advice, where am I wrong?
79 Views