https://flyte.org logo
#flyte-deployment
Title
# flyte-deployment
g

Greg Linklater

06/22/2023, 2:10 PM
Hey everyone I'm busy working trying to deploy Flyte to k8s hosted on Hetzner but struggling as existing deployment patterns seem to be geared toward AWS / GCP. Flyte looks awesome but at this point I can't even deploy it well enough to test it in our environment. I've tried using the
flyte-binary
helm chart. Also having trouble working around this: https://github.com/flyteorg/flyte/issues/3769 Could use a hand if anyone has one to spare.
s

Super Bo

06/22/2023, 2:38 PM
Did you try
flyte-core
, I found flyte-core give you more config options and more examples than flyte-binary.
g

Greg Linklater

06/22/2023, 2:42 PM
@Super Bo I haven't. To be honest I didn't get very far with
flyte-binary
because I was trying to find a way around putting plain text secrets into my values. Our configs (incl. helm applications) are picked up from a git repo and I am not to keen on the idea of committing secrets. I did try to work around this with Kustomize but there's just so much that would need to change or be redone
s

Super Bo

06/22/2023, 3:36 PM
Hi @Greg Linklater, if you want to avoid plain db plaintext password for
flyte-binary
, you can refer to my configuration values:
Copy code
configuration:
  database:
    passwordPath: /etc/db-password-secret/password

deployment:
  extraVolumeMounts:
    - name: db-password
      mountPath: /etc/db-password-secret
      readOnly: true
  extraVolumes:
    - name: db-password
      secret:
        secretName: database-secret
p

Pradithya Aria Pura

06/22/2023, 4:16 PM
The way we address this is by using external secret operator https://external-secrets.io/v0.8.3/ . The secret value is stored in external secret storage (we use google secret manager) and in our git repo we created the external secret resource which refer to the secret hence no secret value is stored in the repo.
m

Mike Ossareh

06/22/2023, 4:31 PM
And that works because https://github.com/flyteorg/flyte/blob/master/charts/flyte-binary/templates/db-password-secret.yaml#L1 - i.e. absent the password in the configuration it doesn't create the default Secret - allowing the external-secrets secret to work. Another solution (the one we've gone with) is to use tools which allow you to encrypt the value in the git repo - and it's only decrypted when being applied to the cluster. edit: here's the equiv line for flyte-core: https://github.com/flyteorg/flyte/blob/master/charts/flyte-core/templates/common/secret.yaml
d

David Espejo (he/him)

06/22/2023, 8:48 PM
Thanks @Super Bo @Pradithya Aria Pura and @Mike Morgan! @Greg Linklater please let us know if this is helpful for you
g

Greg Linklater

06/23/2023, 5:44 AM
Hey everyone, thanks for the responses. I’m in the EU so please excuse the delay. I am aware of the facility for the db password to be pulled from a secret mounted as a file, however what about other secrets like for S3 and OIDC? Additionally some of those configurable values (e.g. S3 connection parameters) I would prefer to be injected alongside the secret to make it easier to change those configurations later if necessary. We are also pulling secrets from Vault but they are injected at deploy time. The external secrets operator is not something I’ve seen before but we use something similar that populates a secret template.
@Mike Ossareh I think I see what you were referring to with those files. What secrets can I put in there? Those fields being pulled from
values.yaml
don't seem to be documented.
m

Mike Ossareh

06/23/2023, 2:14 PM
@Greg Linklater they're not (at least, not well). Given you already have a tool for injecting secrets at run time you can omit the password from value.yaml, the Secret that the chart attempts to create will not be created, and as long as you match naming with the expectation your secrets injection will work just fine. You mention wanting to provide S3 / OIDC / etc secrets; I don't know flyte well enough to know which specific pieces of its architecture need which specific credentials - but generally the deployment.yaml in the helm chart have tools for passing in secrets. For example: flyte propeller allows you to provide
volumeMounts
and
volumes
via values.yaml: • volumeMountsvolumes This allows your normal secret injection to run, and you can reference the secret name in the volume config to put the file/secret wherever you want it to be for your client code to work correctly. Tools like AWS CLI will also look at the environment for its credentials, and the helm chart allows defining the environment via values.yaml as well. For example; the admin service has this: .Values.flyteadmin.env which you can populate via a secret using env - valueFrom -> secretKeyRef>.
g

Greg Linklater

06/23/2023, 2:18 PM
@Mike Ossareh thanks I'll look into it. I guess it is just figuring out what the expected names of the fields etc. I was under the impression that most of those were
kind: ConfigMap
though and not
kind: Secret
... probably just need to take a close look.
j

jeev

06/26/2023, 1:35 PM
While this might be doable, it needs better docs, and also better support in flyte-binary. Basically, we need to mount another flyte config file from a pre-created k8s secret that will be merged into the final config that flyte-binary will parse. That will require some knowledge of the config struct.
Happy to work with you on this @Greg Linklater
g

Greg Linklater

06/27/2023, 6:45 AM
@jeev Much appreciated, will check your PR as soon as I am able.
j

jeev

06/27/2023, 6:47 AM
briefly, I think
inlineSecretRef
would allow you to mount a secret that specifies overrides for the base config created by the chart. that could capture anything that you deem a secret.
g

Greg Linklater

06/27/2023, 6:49 AM
Something like that would be ideal, I would just some kind of example of how it merges the values supplied; i.e. what specific fields names I should specify.
j

jeev

06/27/2023, 6:54 AM
docs are forthcoming for these. but briefly, flyte merges multiple config files. you can specify overrides in separate files this way. however, you will need to have a good grasp on the flyte's underlying config structure - the helm chart templates will be pretty useful here. if you have a question about something specific, I can help.
here is an example:
Copy code
$ cat <<EOF | kubectl apply -f -
 apiVersion: v1
 kind: Secret
 metadata:
   name: flyte-binary-client-secrets-external-secret
   namespace: flyte
 type: Opaque
 stringData:
   client_secret: <CLIENT_SECRET>
   oidc_client_secret: <OIDC_CLIENT_SECRET>
 ---
 apiVersion: v1
 kind: Secret
 metadata:
   name: flyte-binary-inline-config-secret
   namespace: flyte
 type: Opaque
 stringData:
   202-database-secrets.yaml: |
     database:
       postgres:
         password: <DB_PASSWORD>
   204-auth-secrets.yaml: |
     auth:
       appAuth:
         selfAuthServer:
           staticClients:
             flytepropeller:
               client_secret: <CLIENT_SECRET_HASH>
 EOF
g

Greg Linklater

06/27/2023, 6:57 AM
This is perfect; much appreciated!
j

jeev

06/27/2023, 6:57 AM
you can add
clientId
in here as well.
74 Views