https://flyte.org logo
Powered by
#ask-the-community
Title
# ask-the-community
v

Victor Delépine

08/10/2023, 4:09 PM
Hello! Back with more Azure adventures 🙂 I've managed to trigger a basic workflow from 
flytesnacks
now, but in the UI I see the following error for my execution:
Copy code
FlyteAssertion: Failed to get data from 
<abfs://flyte/flytesnacks/development/A46TBP35TISQ2EVJVNOGLJTARM======/script_mod>
e.tar.gz to /root/ (recursive=False).

Original exception: unable to connect to account for Must provide either a 
connection_string or account_name with credentials!!
Here's my storage config in the helm chart:
Copy code
storage:
  type: custom
  bucketName: "flyte"
  custom:
    container: "flyte"
    enable-multicontainer: true
    connection: {}
    type: stow
    stow:
      kind: azure
      config:
        account: "<storage_account_name>"
        key: "<storage_account_key>"
  limits:
    maxDownloadMBs: 10
Does anyone know what would cause this? My first intuition would be that this URI is wrong: 
<abfs://flyte/flytesnacks/development/A46TBP35TISQ2EVJVNOGLJTARM======/script_mode.tar.gz>
because in Azure the path should look like this: 
abfs://<CONTAINER>@<STORAGE_ACCOUNT>.<http://dfs.core.windows.net/flytesnacks/development/A46TBP35TISQ2EVJVNOGLJTARM======/script_mode.tar.gz|dfs.core.windows.net/flytesnacks/development/A46TBP35TISQ2EVJVNOGLJTARM======/script_mode.tar.gz>
🤔 reading this: https://github.com/fsspec/adlfs/blob/main/adlfs/spec.py#L115 It seems like that is something that 
fsspec
is supposed to handle. So somehow it cannot find my credentials
k

Kevin Su

08/10/2023, 5:17 PM
does you pod have IAM role to access data in abfs?
k

Ketan (kumare3)

08/10/2023, 8:30 PM
Yes this is delegated to fsspec. But we have heard fsspec azure file system has some bugs from some community folks but it was fixed in a later version
v

Victor Delépine

08/10/2023, 10:55 PM
Could be an issue with my pod’s role yeah 🤔 . But why would I be able to create the object and not retrieve it? Looking at the fsspec code, it seems like it cannot find my credentials. I’ll double check and report back!
g

Gopal Vashishtha

08/11/2023, 2:53 AM
Victor I was able to resolve this issue by setting an environment variable in my pod template:
Copy code
env:
          - name: AZURE_STORAGE_CONNECTION_STRING
            valueFrom:
              secretKeyRef:
                name: flyteruntimesecret
that 
AZURE_STORAGE_CONNECTION_STRING
value came from the source code you linked above
k

Ketan (kumare3)

08/11/2023, 4:36 AM
is this because Azure does not have something like IAM roles for Service accounts or Workload identity?
v

Victor Delépine

08/11/2023, 6:56 AM
Oh nice! For which pod did you do that? I tried it earlier but you can only pass extra env variables to flyteadmin in the helm chart, and I think my error is in propeller
As to why, Azure does have Workload identity now but it’s not enabled on my cluster, I could give it another try with it enabled. I was counting on the account name and key to be propagated to pods somehow (from storage.yaml)
Thanks for your help !-
Oh I see what you mean @Gopal Vashishtha You used a pod template that defines that variable directly
I couldn't manage to apply a pod template via the helm values, as I need to give permissions to 
flyteadmin
via a role binding beforehand. That probably means I need to set up workload identity in my cluster, so I'll give that a go
How does the cluster resource sync code authenticate with Azure? Does it support workload identity? I'm getting errors saying it's unable to create my 
PodTemplate
despite me giving permissions to the 
flyteadmin
service account
Copy code
{
  "json": {},
  "level": "warning",
  "msg": "Failed to create kubernetes object from config template [pod_template.yaml] for namespace [flytesnacks-development] with err: podtemplates is forbidden: User \"system:serviceaccount:flyte:flyteadmin\" cannot create resource \"podtemplates\" in API group \"\" in the namespace \"flytesnacks-development\"",
  "ts": "2023-08-11T15:41:02Z"
}
Copy code
{
  "json": {},
  "level": "warning",
  "msg": "Failed to create cluster resources for namespace [flytesnacks-development] with err: Failed to create kubernetes object from config template [pod_template.yaml] for namespace [flytesnacks-development] with err: podtemplates is forbidden: User \"system:serviceaccount:flyte:flyteadmin\" cannot create resource \"podtemplates\" in API group \"\" in the namespace \"flytesnacks-development\"",
  "ts": "2023-08-11T15:41:02Z"
}
Oh I think I get it. This uses 
flytek8s
which has everything it needs to create resources in the cluster. The issues I'm having are related to a chicken and egg problem I have with terraform. The helm chart creates the flyteadmin service account, but instead I should create it via terraform and do the role assignments before the helm chart is applied. Sorry if I'm rambling 😅
k

Ketan (kumare3)

08/12/2023, 4:31 AM
Omg azure @Gopal Vashishtha and other can someone please help us write a working azure override
v

Victor Delépine

08/12/2023, 6:03 AM
Will definitely write something once I have a working setup
t

tk

09/20/2023, 10:57 PM
To dust off this thread a bit, I also bumped into this issue when testing in connection with flyteorg/stow/pull/9. I was pointed here by Samhita (thanks for that, btw). I can add a little more context to the situation. While adding the 
AZURE_STORAGE_CONNECTION_STRING
env var to all task pods will definitely work, it's not the only path. Adding 
AZURE_STORAGE_ACCOUNT_NAME
to all pods will also work, as long as the credentials can be loaded from the usual spots which you can find here.
21 Views
Powered by