Matheus Moreno
flyte.secrets/sX
, but no secret is actually being found by the task. How can I debug this?
A little bit more of context: we are injecting environment variables and service account JSONs in the tasks. Because of that, we are actually retrieving secrets directly from /etc/flyte/secrets
, because the SecretsManager()
ends up applying upper()
and lower()
to the keys, which messes up the configuration of the env vars (and files). But apparently no secret is being added to this path...flyte@afdxj58wfblmgn52glcb-n1-0:/$ cd /etc/flyte/secrets
bash: cd: /etc/flyte/secrets: No such file or directory
flyte@afdxj58wfblmgn52glcb-n1-0:/$ cd /etc/secrets
bash: cd: /etc/secrets: No such file or directory
katrina
Yee
Matheus Moreno
Yee
Matheus Moreno
Yee
Matheus Moreno
Yee
logger:
level: 5
show-source: true
that is the bit to add for logging{"json":{"src":"secrets.go:54"},"level":"info","msg":"Failed to inject a secret using injector [Global]. Error: secrets not found - Env [FLYTE_SECRET_TEST-GROUP_TEST-ENV], file [/etc/secrets/test-group/test-env]","ts":"2022-07-18T20:03:47Z"}
{"json":{"src":"secrets.go:54"},"level":"info","msg":"Failed to inject a secret using injector [Global]. Error: secrets not found - Env [FLYTE_SECRET_TEST-GROUP_TEST-FILE], file [/etc/secrets/test-group/test-file]","ts":"2022-07-18T20:03:47Z"}
{
"o0": "Hello world, these are my secrets: TESTING_ENV / TESTING_FILE"
}
Matheus Moreno
Yee
Matheus Moreno
configmap:
logger:
level: 6
show-source: true
Yee
storage.yaml: |
logger:
level: 5
show-source: true
storage:
type: minio
...
Matheus Moreno
/etc/flyte/config $ cat logger.yaml
level: 6
show-source: true
/etc/flyte/config $
logger:
field right?Yee
Matheus Moreno
logger:
and did a helm upgradeYee
Matheus Moreno
Yee
Matheus Moreno
Yee
Matheus Moreno
Yee
Matheus Moreno
Yee
Matheus Moreno
Yee
Matheus Moreno
Yee
Matheus Moreno
Yee
Matheus Moreno
Yee
Matheus Moreno
Yee
- name: orsxg4bnm4zg54lql3
secret:
defaultMode: 420
items:
- key: test-file
path: test-file
secretName: test-group
Matheus Moreno
Yee
Matheus Moreno
metadata:
annotations:
<http://cluster-autoscaler.kubernetes.io/safe-to-evict|cluster-autoscaler.kubernetes.io/safe-to-evict>: "false"
flyte.secrets/s0: ...
flyte.secrets/s1: ...
...
flyte.secrets
Yee
FLYTE_SECRETS_DEFAULT_DIR
env var specified?Matheus Moreno
Yee
Matheus Moreno
Yee
Matheus Moreno
FLYTE_SECRETS_DEFAULT_DIR
isn't set, so no secret is mounted?Yee
Matheus Moreno
Yee
""
Matheus Moreno
Yee
$ k get mutatingwebhookconfigurations
NAME WEBHOOKS AGE
flyte-pod-webhook 1 2d23h
Matheus Moreno
Yee
Matheus Moreno
NAME WEBHOOKS AGE
datadog-webhook 2 95d
flyte-pod-webhook 1 4d1h
Yee
Matheus Moreno
Yee
Matheus Moreno
Yee
Matheus Moreno
Yee
service:
name: flyte-pod-webhook
namespace: flyte
path: /mutate--v1-pod
port: 443
failurePolicy: Ignore
matchPolicy: Equivalent
name: <http://flyte-pod-webhook.flyte.org|flyte-pod-webhook.flyte.org>
namespaceSelector: {}
objectSelector:
matchLabels:
inject-flyte-secrets: "true"
reinvocationPolicy: Never
rules:
- apiGroups:
- '*'
apiVersions:
- v1
operations:
- CREATE
resources:
- pods
scope: '*'
sideEffects: NoneOnDryRun
timeoutSeconds: 10
Matheus Moreno
Yee
Matheus Moreno
Yee
Matheus Moreno
Yee
kubectl get mutatingwebhookconfigurations flyte-pod-webhook -o yaml
Matheus Moreno
service:
name: flyte-pod-webhook
namespace: ml-dev
path: /mutate--v1-pod
port: 443
failurePolicy: Ignore
matchPolicy: Equivalent
name: <http://flyte-pod-webhook.flyte.org|flyte-pod-webhook.flyte.org>
namespaceSelector: {}
objectSelector:
matchLabels:
inject-flyte-secrets: "true"
reinvocationPolicy: Never
rules:
- apiGroups:
- '*'
apiVersions:
- v1
operations:
- CREATE
resources:
- pods
scope: '*'
sideEffects: NoneOnDryRun
timeoutSeconds: 10
Yee
ml-dev
?ml-flyte
?Matheus Moreno
service:
name: flyte-pod-webhook
namespace: ml-flyte
path: /mutate--v1-pod
port: 443
failurePolicy: Ignore
matchPolicy: Equivalent
name: <http://flyte-pod-webhook.flyte.org|flyte-pod-webhook.flyte.org>
namespaceSelector: {}
objectSelector:
matchLabels:
inject-flyte-secrets: "true"
reinvocationPolicy: Never
rules:
- apiGroups:
- '*'
apiVersions:
- v1
operations:
- CREATE
resources:
- pods
scope: '*'
sideEffects: NoneOnDryRun
timeoutSeconds: 10
Yee
Matheus Moreno
Yee
Matheus Moreno
No plugin found for Handler-type [python-task], defaulting to [container],
Yee
Matheus Moreno
Yee
inject-flyte-secrets: "true"
in your task pod labels right?apiVersion: v1
kind: Pod
Matheus Moreno
Yee
Matheus Moreno
jeev
Matheus Moreno
secret_requests
. Is there another way?ml-flyte-projects
.jeev
Yee
jeev
Yee
Matheus Moreno
configmap:
k8s:
k8s:
default-pod-template-name: <PodTemplate created in the same namespace as FlytePropeller>
jeev
Matheus Moreno
jeev
PodTemplate
object still has to be valid, so many of the fields will be required.Matheus Moreno
jeev
Yee
Dan Rammer (hamersaw)
Matheus Moreno
V1SecretVolumeSource
and a V1VolumeMount
with the required secrets. Since it's a sidecar task, I believe it's something similar to what Jeev is doing right now. For now, it works perfectly! đYee
flyte.secrets/s0: m4zg54lqhiqce4dfon1c1z2sn41xaiqknnsxsoraej1gk32ufvsw34rcbjww54loorpxezlrovuxezlnmvxhioraivhfmx1wifjau
right?Matheus Moreno
Yee
Matheus Moreno
Yee
Matheus Moreno
Yee