Hi there, We're trying to put together a clustered environment on a baremetal microk8s cluster. We've provisioned our postgresql server and our minio server and we're trying to work with the flyte-binary helm chart. We've configured the necessary

configuration.database

and

configuration.storage.providerConfig

configs with the following:

configuration.database.host = "postgres-svc"
configuration.database.username = "postgres"
configuration.database.password = "<passwd>"
configuration.storage.metadataContainer = "bucket1"
configuration.storage.userDataContainer = "bucket1"
configuration.storage.providerConfig.s3.endpoint = "minio-svc"
configuration.storage.providerConfig.s3.authType = "acceskey"
configuration.storage.providerConfig.s3.accessKey = "flyteuser"
configuration.storage.providerConfig.s3.secretKey = "<flyteuserpasswd>"
configuration.storage.providerConfig.s3.disableSSL = true

We're able to register workflows and tasks and we're able to execute them, although the pods running our hello world tasks are not able to connect to our minio instance (at least that's what we get from the following error)

│ │ _execute_task                                                                │                                                                                                                              ││ │                                                                              │                                                                                                                              │
│ │ ❱ 362 │   │   _handle_annotated_task(ctx, _task_def, inputs, output_prefix)  │                                                                                                                              │
│ │                                                                              │                                                                                                                              │
│ │ /opt/venv/lib/python3.8/site-packages/flytekit/bin/entrypoint.py:304 in      │                                                                                                                              │
│ │ _handle_annotated_task                                                       │                                                                                                                              ││ │                                                                              │                                                                                                                              │
│ │ ❱ 304 │   _dispatch_execute(ctx, task_def, inputs, output_prefix)            │                                                                                                                              │
│ │                                                                              │                                                                                                                              │
│ │ /opt/venv/lib/python3.8/site-packages/flytekit/bin/entrypoint.py:160 in      │                                                                                                                              │
│ │ _dispatch_execute                                                            │                                                                                                                              │
│ │                                                                              │                                                                                                                              │
│ │ ❱ 160 │   ctx.file_access.put_data(ctx.execution_state.engine_dir, output_pr │                                                                                                                              │
│ │                                                                              │                                                                                                                              │
│ │ /opt/venv/lib/python3.8/site-packages/flytekit/core/data_persistence.py:320  │                                                                                                                              │
│ │ in put_data                                                                  │                                                                                                                              │
│ │                                                                              │                                                                                                                              │
│ │ ❱ 320 │   │   │   raise FlyteAssertion(                                      │                                                                                                                              │
│ ╰──────────────────────────────────────────────────────────────────────────────╯                                                                                                                              │
│ FlyteAssertion: Failed to put data from                                                                                                                                                                       │
│ /tmp/flyte-erp4nnjc/sandbox/local_flytekit/engine_dir to                                                                                                                                                      │
│ <s3://flyte/metadata/propeller/flytesnacks-development-fa6db7ab7ed90473588b/n0/da>                                                                                                                              │
│ ta/0 (recursive=True).                                                                                                                                                                                        │
│                                                                                                                                                                                                               │
│ Original exception: Unable to locate credentials

Any idea what's going wrong?

We would need to set default environment variables for the tasks pointing to the internal minio cluster. You can see how we do this for sandbox for instance. If you need more control over the task pod, take a look at https://docs.flyte.org/en/latest/deployment/configuration/general.html#id1. Also check out #on-prem-deployment

Thanks for your response. Actually the inline configuration allows for setting this although the environment variables forwarded to the task pod is all lower case thus it fails to authenticate to minio. Probably it’s gonna be the podtemplate approach we go for.

Sorry i don’t understand. Can you paste a redacted snippet?

When the

configuration.inline.plugins.k8s.default-env-vars

array with

FLYTE_AWS_ENDPOINT

,

FLYTE_AWS_ACCESS_KEY_ID

,

FLYTE_AWS_SECRET_ACCESS_KEY

was configured these environment variables were propagated to the task's pod with their lowercase variants which resulted in a failed authentication to minio since it could not pick up the values correctly.

these environment variables were propagated to the task's pod with their lowercase variants

wow. that is really surprising for propagating secrets, i believe we can use cluster resource templates for this. that would automatically create the secret when a new project is provisioned.