https://flyte.org logo
#ask-the-community
Title
# ask-the-community
d

Denis Skakalski

02/21/2024, 1:57 PM
hi guys, I have deployed the Flyte (version in Helm chart 1.10.7) on Azure AKS cloud, now I have created storage account (name is 'flyte', data lake is activated) and PostgresSQL (ver.16 ), everything have started successfully, but during executing workflow I am getting such errors in logs: ---------------------- FlyteAssertion: Failed to get data from abfs://my-flyte-container/flytesnacks/development/I62Z4WQMINA7777777777RU==== ==/script_mode.tar.gz to /root/ (recursive=False). Original exception: unable to connect to account for Must provide either a connection_string or account_name with credentials!! ----------------------- This is my value.yaml ---------------------------------------------------------- flyte-core-components: propeller: # disabled Disables flytepropeller disabled: false rawoutput-prefix: my-flyte-container@flyte.dfs.core.windows.net configuration: database: username: flyte password: uuuuuu host: flyte8.postgres.database.azure.com port: 5432 dbname: postgres options: sslmode=disable storage: #Learn more about how Flyte handles data: https://docs.flyte.org/en/latest/concepts/data_management.html metadataContainer: "my-flyte-container@flyte.dfs.core.windows.net" userDataContainer: "my-flyte-container@flyte.dfs.core.windows.net" provider: azure providerConfig: azure: # configDomainSuffix Domain name suffix configDomainSuffix: "" # configUploadConcurrency Upload Concurrency (default 4) configUploadConcurrency: 4 # account Storage Account name account: "flyte" # key Storage Account key if used key: "50f4xgsGkx669GMruuuuuuuuuBkocNGw60YlI+AStnH0gyA==" ------------------------------------------------------------------------- and that is the final template: ----------------------------------------------- data: 000-core.yaml: | admin: endpoint: localhost:8089 insecure: true catalog-cache: endpoint: localhost:8081 insecure: true type: datacatalog cluster_resources: standaloneDeployment: false templatePath: /etc/flyte/cluster-resource-templates logger: show-source: true level: 1 propeller: create-flyteworkflow-crd: true webhook: certDir: /var/run/flyte/certs localCert: true secretName: flyte-backend-flyte-binary-webhook-secret serviceName: flyte-backend-flyte-binary-webhook servicePort: 443 flyte: admin: disableClusterResourceManager: false disableScheduler: false disabled: false seedProjects: - flytesnacks dataCatalog: disabled: false propeller: disableWebhook: false disabled: false rawoutput-prefix: my-flyte-container@flyte.dfs.core.windows.net 001-plugins.yaml: | tasks: task-plugins: default-for-task-types: container: container container_array: k8s-array sidecar: sidecar enabled-plugins: - container - sidecar - k8s-array - agent-service plugins: logs: kubernetes-enabled: false cloudwatch-enabled: false stackdriver-enabled: false k8s: co-pilot: image: "cr.flyte.org/flyteorg/flytecopilot-release:v1.10.7" k8s-array: logs: config: kubernetes-enabled: false cloudwatch-enabled: false stackdriver-enabled: false 002-database.yaml: | database: postgres: username: flyte host: flyte8.postgres.database.azure.com port: 5432 dbname: postgres options: "sslmode=disable" 003-storage.yaml: | propeller: rawoutput-prefix: abfs://my-flyte-container@flyte.dfs.core.windows.net/data storage: type: stow stow: kind: azure config: account: flyte key: 50f4xtttttttttttttttttttttttttttttttttttttttttttttttttlI+AStnH0gyA== configUploadConcurrency: 4 container: my-flyte-container -------------------------------------------- Maybe you can advice, where am I wrong?
c

Chris Grass

02/21/2024, 3:01 PM
Hi @Denis Skakalski - have you tried using an Azure Managed Identity as opposed to a SA shared key?
It's odd that the error message doesn't show the account it's trying to connect to ("Original exception: unable to connect to account for"), so I think there is a second problem. but using managed identity might help account management issues and improve security
can you try removing
@flyte.dfs.core.windows.net
from your
metadataContainer
and
userDataContainer
values?
also, have you created
my-flyte-container
in your
flyte
SA?
d

Denis Skakalski

02/21/2024, 3:35 PM
my-flyte-container are creating authomatically inside of storage account previously created. I have faced the same issue described here https://discuss.flyte.org/t/13904302/hello-back-with-more-azure-adventures-slightly-smiling-face-
@Gopal Vashishtha, maybe you can look into.
c

Chris Grass

02/21/2024, 3:39 PM
ah - are you setting the container name in your task pod template? e.g.:
Copy code
inline:
    plugins:
      k8s:
        metadata:
          namespace: flyte-development
        inject-finalizer: true
        default-env-vars:
          - FLYTE_AZURE_STORAGE_ACCOUNT_NAME: SA_NAME
d

Denis Skakalski

02/21/2024, 3:47 PM
Will it be correct? Just name of storage account. clusterResourceTemplates: inline: #This section automates the creation of the project-domain namespaces 001_namespace.yaml: | apiVersion: v1 kind: Namespace metadata: name: '{{ namespace }}' # This block performs the automated annotation of KSAs across all project-domain namespaces 002_serviceaccount.yaml: | apiVersion: v1 kind: ServiceAccount metadata: name: default namespace: '{{ namespace }}' plugins: k8s: metadata: name: '{{ namespace }}' default-env-vars: - FLYTE_AZURE_STORAGE_ACCOUNT_NAME: "flyte"
c

Chris Grass

02/21/2024, 3:50 PM
if you're using a storage account shared key you will need to provide that as an env_var as well
FLYTE_AZURE_STORAGE_ACCOUNT_KEY
d

Denis Skakalski

02/21/2024, 3:52 PM
this ones, correct?
key, not connection string
c

Chris Grass

02/21/2024, 3:53 PM
correct
d

Denis Skakalski

02/21/2024, 3:53 PM
let me redeploy
now, the previous error passed, but in storage account has not been created folder "metadata" (previously it was) only 'flytesnacks' created. The error in VSC console.
maybe metadataContainer: "my-flyte-container" userDataContainer: "my-flyte-container" should been different?
c

Chris Grass

02/21/2024, 4:27 PM
they can be the same
you might try removing
@flyte...
from your
rawoutput-prefix
- or even removing
rawoutput-prefix
entirely.
try using the format
rawoutput-prefix: <abfs://my-flyte-container/data>
for that field, or removing it entirely. that should be built automatically by the
_helpers.tpl
file
i also see that your
values.yaml
file is missing the
bucketName
field. i'm not certain that is still required, but you might try adding it below:
storage: userDataContainer: my-flyte-container metadataContainer: my-flyte-container bucketName: flyte it looks like that field is no longer parsed in flyte-binary
d

Denis Skakalski

02/21/2024, 5:29 PM
it was necessary to recreate postgressql as well
image.png
now everything is good, thanks for support
d

David Espejo (he/him)

02/21/2024, 6:38 PM
thank you @Chris Grass! Do you think we could team up to write an Azure section as part of Flyte the Hard Way?
c

Chris Grass

02/21/2024, 6:40 PM
@David Espejo (he/him) - i think that's a good idea. there are a few quirks, especially if someone wants to use workload identity, that distinguishes azure setup from aws
d

David Espejo (he/him)

02/21/2024, 6:42 PM
yeah, feel free to add even a single page with all those pitfalls that you have encountered (especially with stow and IAM) and we can take it from there, reproduce it and expand it. wdyt?
c

Chris Grass

02/21/2024, 6:56 PM
sounds reasonable - i'll chat with @Terence Kent to figure out a timeline and dm you to talk about the plan
e

Ethan Brown

02/21/2024, 7:15 PM
Super awesome to see the Azure progress -- it's something we're pretty interested in as well.
15 Views