Hey everyone I'm busy working trying to deploy Flyte to k8s hosted on Hetzner but struggling as existing deployment patterns seem to be geared toward AWS / GCP. Flyte looks awesome but at this point I can't even deploy it well enough to test it in our environment. I've tried using the

flyte-binary

helm chart. Also having trouble working around this: https://github.com/flyteorg/flyte/issues/3769 Could use a hand if anyone has one to spare.

Did you try

flyte-core

, I found flyte-core give you more config options and more examples than flyte-binary.

@Super Bo I haven't. To be honest I didn't get very far with

flyte-binary

because I was trying to find a way around putting plain text secrets into my values. Our configs (incl. helm applications) are picked up from a git repo and I am not to keen on the idea of committing secrets. I did try to work around this with Kustomize but there's just so much that would need to change or be redone

Hi @Greg Linklater, if you want to avoid plain db plaintext password for

flyte-binary

, you can refer to my configuration values:

configuration:
  database:
    passwordPath: /etc/db-password-secret/password

deployment:
  extraVolumeMounts:
    - name: db-password
      mountPath: /etc/db-password-secret
      readOnly: true
  extraVolumes:
    - name: db-password
      secret:
        secretName: database-secret

The way we address this is by using external secret operator https://external-secrets.io/v0.8.3/ . The secret value is stored in external secret storage (we use google secret manager) and in our git repo we created the external secret resource which refer to the secret hence no secret value is stored in the repo.

And that works because https://github.com/flyteorg/flyte/blob/master/charts/flyte-binary/templates/db-password-secret.yaml#L1 - i.e. absent the password in the configuration it doesn't create the default Secret - allowing the external-secrets secret to work. Another solution (the one we've gone with) is to use tools which allow you to encrypt the value in the git repo - and it's only decrypted when being applied to the cluster. edit: here's the equiv line for flyte-core: https://github.com/flyteorg/flyte/blob/master/charts/flyte-core/templates/common/secret.yaml

Thanks @Super Bo @Pradithya Aria Pura and @Mike Morgan! @Greg Linklater please let us know if this is helpful for you

Hey everyone, thanks for the responses. I’m in the EU so please excuse the delay. I am aware of the facility for the db password to be pulled from a secret mounted as a file, however what about other secrets like for S3 and OIDC? Additionally some of those configurable values (e.g. S3 connection parameters) I would prefer to be injected alongside the secret to make it easier to change those configurations later if necessary. We are also pulling secrets from Vault but they are injected at deploy time. The external secrets operator is not something I’ve seen before but we use something similar that populates a secret template.

@Greg Linklater they're not (at least, not well). Given you already have a tool for injecting secrets at run time you can omit the password from value.yaml, the Secret that the chart attempts to create will not be created, and as long as you match naming with the expectation your secrets injection will work just fine. You mention wanting to provide S3 / OIDC / etc secrets; I don't know flyte well enough to know which specific pieces of its architecture need which specific credentials - but generally the deployment.yaml in the helm chart have tools for passing in secrets. For example: flyte propeller allows you to provide

volumeMounts

and

volumes

via values.yaml: • volumeMounts • volumes This allows your normal secret injection to run, and you can reference the secret name in the volume config to put the file/secret wherever you want it to be for your client code to work correctly. Tools like AWS CLI will also look at the environment for its credentials, and the helm chart allows defining the environment via values.yaml as well. For example; the admin service has this: .Values.flyteadmin.env which you can populate via a secret using env - valueFrom -> secretKeyRef>.

@Mike Ossareh thanks I'll look into it. I guess it is just figuring out what the expected names of the fields etc. I was under the impression that most of those were

kind: ConfigMap

though and not

kind: Secret

... probably just need to take a close look.

While this might be doable, it needs better docs, and also better support in flyte-binary. Basically, we need to mount another flyte config file from a pre-created k8s secret that will be merged into the final config that flyte-binary will parse. That will require some knowledge of the config struct.

@jeev Much appreciated, will check your PR as soon as I am able.

briefly, I think

inlineSecretRef

would allow you to mount a secret that specifies overrides for the base config created by the chart. that could capture anything that you deem a secret.

Something like that would be ideal, I would just some kind of example of how it merges the values supplied; i.e. what specific fields names I should specify.

docs are forthcoming for these. but briefly, flyte merges multiple config files. you can specify overrides in separate files this way. however, you will need to have a good grasp on the flyte's underlying config structure - the helm chart templates will be pretty useful here. if you have a question about something specific, I can help.

This is perfect; much appreciated!

you can add

clientId

in here as well.