Hi everyone. I deployed flyte-binary and flyte-deps to use flyte with minio on an aks cluster. The connection of these two does not seem to work. I get a timed out connection once I run a workflow. both pods are running in the same namespace. I tried different endpoint definitions for flyte binary, here is the latest storage config from value.yaml:

storage:
    metadataContainer: my-s3-bucket
    userDataContainer: my-s3-bucket
    provider: s3
    providerConfig:
      s3:
        disableSSL: true
        v2Signing: true
        endpoint: <http://minio.flyte.svc.cluster.local:9000>
        authType: accesskey
        accessKey: minio
        secretKey: miniostorage

the error is:

HTTPConnectionPool(host='minio.flyte.svc.cluster.local', port=9000): Max retries exceeded with url: /my-s3-bucket/flytesnacks/development/7OSB6SPSGTXJALEHAXOWIZSNSI%3D%3D%3D%3D%3D%3D/script_mode.tar.gz?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=minio%2F20230824%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20230824T135828Z&X-Amz-Expires=3600&X-Amz-SignedHeaders=content-md5%3Bhost&X-Amz-Signature=d6a717a64a40e76cd1a77c8e434dbe7d5f5d5e3048ebef3ebd2ef1554a72b1c4 (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f66e019af20>: Failed to establish a new connection: [Errno -3] Temporary failure in name resolution'))

I tried it as well with the actual ip, because this hints to a dns problem. It then results in

Connection to <ip> timed out. (connect timeout=None)

any hints ? thanks in advance 🙂

This matches what sandbox is doing right?

I think so. but running on a real cluster. In the end I'd prefer to have blob storage connected, but I couldn't find a working solution. -- We don't have acces keys activated for our storage accounts.

And I hear AKS has no Iam role for serviceaccount or the like?

yeah there are managed identities

@Philip Thomsen I haven't run this on AKS, but on local K8s, it needs additional info on the

values

file:

inline:
    plugins:
      k8s:
        inject-finalizer: true
        default-env-vars:
          - FLYTE_AWS_ENDPOINT: "<http://minio.flyte.svc.cluster.local:9000>"
          - FLYTE_AWS_ACCESS_KEY_ID: "your-access-key-id"
          - FLYTE_AWS_SECRET_ACCESS_KEY: "your-secret-access-key"

nice, thanks @David Espejo (he/him) 🙏. this should be in the values file for the

flyte-binary

chart, correct ?

Right

The error unfortunately stays the same.

are you running the workflow using

pyflyte run

from your laptop? does the

minio.flyte.svc.cluster.local

name is resolvable from your machine?

yes that's true. I thought data would be transferred from the flyte pod to minio and not from my laptop to minio, but this is not the case ?

pyflyte

uploads the code to blob storage using a signed URL. And both the flyte Pod and

pyflyte

need to access blob storage using the same URL which is probably fine for the Pod since it uses service discovery but not for the CLI if your OS cannot resolve This guide covers the process of deploying

flyte-binary

to a "local" or let's say "unmanaged" Kubernetes environment. It could be useful

flyte-the-hard-way

😄, I like it.