sparse-window-1536
07/15/2022, 6:43 PMflyte.secrets/sX
, but no secret is actually being found by the task. How can I debug this?
A little bit more of context: we are injecting environment variables and service account JSONs in the tasks. Because of that, we are actually retrieving secrets directly from /etc/flyte/secrets
, because the SecretsManager()
ends up applying upper()
and lower()
to the keys, which messes up the configuration of the env vars (and files). But apparently no secret is being added to this path...sparse-window-1536
07/15/2022, 6:45 PMsparse-window-1536
07/15/2022, 7:19 PMflyte@afdxj58wfblmgn52glcb-n1-0:/$ cd /etc/flyte/secrets
bash: cd: /etc/flyte/secrets: No such file or directory
flyte@afdxj58wfblmgn52glcb-n1-0:/$ cd /etc/secrets
bash: cd: /etc/secrets: No such file or directory
sparse-window-1536
07/15/2022, 8:14 PMacceptable-policeman-57188
thankful-minister-83577
sparse-window-1536
07/15/2022, 8:55 PMthankful-minister-83577
sparse-window-1536
07/15/2022, 9:01 PMsparse-window-1536
07/15/2022, 9:02 PMsparse-window-1536
07/15/2022, 9:02 PMsparse-window-1536
07/15/2022, 9:02 PMsparse-window-1536
07/15/2022, 9:03 PMthankful-minister-83577
sparse-window-1536
07/15/2022, 10:36 PMsparse-window-1536
07/15/2022, 10:36 PMthankful-minister-83577
thankful-minister-83577
thankful-minister-83577
thankful-minister-83577
logger:
level: 5
show-source: true
that is the bit to add for loggingthankful-minister-83577
thankful-minister-83577
thankful-minister-83577
thankful-minister-83577
{"json":{"src":"secrets.go:54"},"level":"info","msg":"Failed to inject a secret using injector [Global]. Error: secrets not found - Env [FLYTE_SECRET_TEST-GROUP_TEST-ENV], file [/etc/secrets/test-group/test-env]","ts":"2022-07-18T20:03:47Z"}
{"json":{"src":"secrets.go:54"},"level":"info","msg":"Failed to inject a secret using injector [Global]. Error: secrets not found - Env [FLYTE_SECRET_TEST-GROUP_TEST-FILE], file [/etc/secrets/test-group/test-file]","ts":"2022-07-18T20:03:47Z"}
thankful-minister-83577
thankful-minister-83577
{
"o0": "Hello world, these are my secrets: TESTING_ENV / TESTING_FILE"
}
thankful-minister-83577
thankful-minister-83577
sparse-window-1536
07/18/2022, 8:17 PMsparse-window-1536
07/18/2022, 8:26 PMsparse-window-1536
07/18/2022, 8:26 PMthankful-minister-83577
thankful-minister-83577
thankful-minister-83577
sparse-window-1536
07/18/2022, 8:27 PMconfigmap:
logger:
level: 6
show-source: true
sparse-window-1536
07/18/2022, 8:28 PMthankful-minister-83577
thankful-minister-83577
storage.yaml: |
logger:
level: 5
show-source: true
storage:
type: minio
...
thankful-minister-83577
thankful-minister-83577
thankful-minister-83577
sparse-window-1536
07/18/2022, 8:31 PMsparse-window-1536
07/18/2022, 8:31 PM/etc/flyte/config $ cat logger.yaml
level: 6
show-source: true
/etc/flyte/config $
sparse-window-1536
07/18/2022, 8:31 PMsparse-window-1536
07/18/2022, 8:32 PMlogger:
field right?thankful-minister-83577
thankful-minister-83577
thankful-minister-83577
thankful-minister-83577
thankful-minister-83577
sparse-window-1536
07/18/2022, 8:34 PMlogger:
and did a helm upgradethankful-minister-83577
sparse-window-1536
07/18/2022, 8:34 PMthankful-minister-83577
thankful-minister-83577
sparse-window-1536
07/18/2022, 8:35 PMthankful-minister-83577
sparse-window-1536
07/18/2022, 8:38 PMthankful-minister-83577
sparse-window-1536
07/18/2022, 8:38 PMsparse-window-1536
07/18/2022, 8:38 PMthankful-minister-83577
sparse-window-1536
07/18/2022, 8:39 PMthankful-minister-83577
sparse-window-1536
07/18/2022, 8:39 PMthankful-minister-83577
thankful-minister-83577
sparse-window-1536
07/18/2022, 8:40 PMthankful-minister-83577
sparse-window-1536
07/18/2022, 8:41 PMthankful-minister-83577
thankful-minister-83577
sparse-window-1536
07/18/2022, 8:41 PMsparse-window-1536
07/18/2022, 8:41 PMthankful-minister-83577
thankful-minister-83577
- name: orsxg4bnm4zg54lql3
secret:
defaultMode: 420
items:
- key: test-file
path: test-file
secretName: test-group
sparse-window-1536
07/18/2022, 8:42 PMsparse-window-1536
07/18/2022, 8:42 PMthankful-minister-83577
thankful-minister-83577
thankful-minister-83577
sparse-window-1536
07/18/2022, 8:44 PMsparse-window-1536
07/18/2022, 8:44 PMmetadata:
annotations:
<http://cluster-autoscaler.kubernetes.io/safe-to-evict|cluster-autoscaler.kubernetes.io/safe-to-evict>: "false"
flyte.secrets/s0: ...
flyte.secrets/s1: ...
...
sparse-window-1536
07/18/2022, 8:44 PMflyte.secrets
sparse-window-1536
07/18/2022, 8:44 PMthankful-minister-83577
FLYTE_SECRETS_DEFAULT_DIR
env var specified?sparse-window-1536
07/18/2022, 8:47 PMthankful-minister-83577
thankful-minister-83577
thankful-minister-83577
sparse-window-1536
07/18/2022, 8:48 PMthankful-minister-83577
thankful-minister-83577
thankful-minister-83577
sparse-window-1536
07/18/2022, 8:55 PMFLYTE_SECRETS_DEFAULT_DIR
isn't set, so no secret is mounted?sparse-window-1536
07/18/2022, 8:55 PMthankful-minister-83577
thankful-minister-83577
sparse-window-1536
07/18/2022, 8:56 PMsparse-window-1536
07/18/2022, 8:57 PMsparse-window-1536
07/18/2022, 8:57 PMthankful-minister-83577
thankful-minister-83577
thankful-minister-83577
thankful-minister-83577
thankful-minister-83577
""
thankful-minister-83577
thankful-minister-83577
thankful-minister-83577
thankful-minister-83577
sparse-window-1536
07/18/2022, 9:07 PMsparse-window-1536
07/18/2022, 9:08 PMthankful-minister-83577
$ k get mutatingwebhookconfigurations
NAME WEBHOOKS AGE
flyte-pod-webhook 1 2d23h
thankful-minister-83577
sparse-window-1536
07/18/2022, 9:16 PMsparse-window-1536
07/18/2022, 9:17 PMthankful-minister-83577
thankful-minister-83577
sparse-window-1536
07/18/2022, 9:17 PMNAME WEBHOOKS AGE
datadog-webhook 2 95d
flyte-pod-webhook 1 4d1h
sparse-window-1536
07/18/2022, 9:17 PMthankful-minister-83577
thankful-minister-83577
sparse-window-1536
07/18/2022, 9:18 PMthankful-minister-83577
sparse-window-1536
07/18/2022, 9:19 PMthankful-minister-83577
thankful-minister-83577
sparse-window-1536
07/18/2022, 9:20 PMsparse-window-1536
07/18/2022, 9:20 PMthankful-minister-83577
thankful-minister-83577
thankful-minister-83577
thankful-minister-83577
service:
name: flyte-pod-webhook
namespace: flyte
path: /mutate--v1-pod
port: 443
failurePolicy: Ignore
matchPolicy: Equivalent
name: <http://flyte-pod-webhook.flyte.org|flyte-pod-webhook.flyte.org>
namespaceSelector: {}
objectSelector:
matchLabels:
inject-flyte-secrets: "true"
reinvocationPolicy: Never
rules:
- apiGroups:
- '*'
apiVersions:
- v1
operations:
- CREATE
resources:
- pods
scope: '*'
sideEffects: NoneOnDryRun
timeoutSeconds: 10
sparse-window-1536
07/18/2022, 9:21 PMthankful-minister-83577
sparse-window-1536
07/18/2022, 9:21 PMthankful-minister-83577
sparse-window-1536
07/18/2022, 9:22 PMthankful-minister-83577
kubectl get mutatingwebhookconfigurations flyte-pod-webhook -o yaml
sparse-window-1536
07/18/2022, 9:23 PMservice:
name: flyte-pod-webhook
namespace: ml-dev
path: /mutate--v1-pod
port: 443
failurePolicy: Ignore
matchPolicy: Equivalent
name: <http://flyte-pod-webhook.flyte.org|flyte-pod-webhook.flyte.org>
namespaceSelector: {}
objectSelector:
matchLabels:
inject-flyte-secrets: "true"
reinvocationPolicy: Never
rules:
- apiGroups:
- '*'
apiVersions:
- v1
operations:
- CREATE
resources:
- pods
scope: '*'
sideEffects: NoneOnDryRun
timeoutSeconds: 10
sparse-window-1536
07/18/2022, 9:23 PMthankful-minister-83577
ml-dev
?thankful-minister-83577
ml-flyte
?sparse-window-1536
07/18/2022, 9:24 PMsparse-window-1536
07/18/2022, 9:24 PMsparse-window-1536
07/18/2022, 9:25 PMservice:
name: flyte-pod-webhook
namespace: ml-flyte
path: /mutate--v1-pod
port: 443
failurePolicy: Ignore
matchPolicy: Equivalent
name: <http://flyte-pod-webhook.flyte.org|flyte-pod-webhook.flyte.org>
namespaceSelector: {}
objectSelector:
matchLabels:
inject-flyte-secrets: "true"
reinvocationPolicy: Never
rules:
- apiGroups:
- '*'
apiVersions:
- v1
operations:
- CREATE
resources:
- pods
scope: '*'
sideEffects: NoneOnDryRun
timeoutSeconds: 10
sparse-window-1536
07/18/2022, 9:25 PMthankful-minister-83577
sparse-window-1536
07/18/2022, 9:26 PMthankful-minister-83577
thankful-minister-83577
thankful-minister-83577
sparse-window-1536
07/18/2022, 9:30 PMNo plugin found for Handler-type [python-task], defaulting to [container],
sparse-window-1536
07/18/2022, 9:30 PMsparse-window-1536
07/18/2022, 9:31 PMthankful-minister-83577
thankful-minister-83577
thankful-minister-83577
sparse-window-1536
07/18/2022, 9:36 PMthankful-minister-83577
inject-flyte-secrets: "true"
in your task pod labels right?thankful-minister-83577
thankful-minister-83577
apiVersion: v1
kind: Pod
sparse-window-1536
07/18/2022, 10:14 PMsparse-window-1536
07/18/2022, 10:19 PMthankful-minister-83577
thankful-minister-83577
thankful-minister-83577
thankful-minister-83577
thankful-minister-83577
thankful-minister-83577
thankful-minister-83577
sparse-window-1536
07/18/2022, 10:26 PMfreezing-boots-56761
sparse-window-1536
07/18/2022, 10:37 PMsparse-window-1536
07/18/2022, 10:38 PMsecret_requests
. Is there another way?sparse-window-1536
07/18/2022, 10:38 PMml-flyte-projects
.freezing-boots-56761
freezing-boots-56761
thankful-minister-83577
freezing-boots-56761
thankful-minister-83577
sparse-window-1536
07/19/2022, 12:56 AMconfigmap:
k8s:
k8s:
default-pod-template-name: <PodTemplate created in the same namespace as FlytePropeller>
sparse-window-1536
07/19/2022, 12:58 AMfreezing-boots-56761
freezing-boots-56761
sparse-window-1536
07/19/2022, 1:00 AMsparse-window-1536
07/19/2022, 3:39 PMfreezing-boots-56761
PodTemplate
object still has to be valid, so many of the fields will be required.sparse-window-1536
07/19/2022, 5:28 PMfreezing-boots-56761
thankful-minister-83577
thankful-minister-83577
hallowed-mouse-14616
07/19/2022, 6:28 PMsparse-window-1536
07/19/2022, 8:25 PMV1SecretVolumeSource
and a V1VolumeMount
with the required secrets. Since it's a sidecar task, I believe it's something similar to what Jeev is doing right now. For now, it works perfectly! 🙏thankful-minister-83577
thankful-minister-83577
thankful-minister-83577
flyte.secrets/s0: m4zg54lqhiqce4dfon1c1z2sn41xaiqknnsxsoraej1gk32ufvsw34rcbjww54loorpxezlrovuxezlnmvxhioraivhfmx1wifjau
right?sparse-window-1536
07/19/2022, 8:51 PMsparse-window-1536
07/19/2022, 8:52 PMthankful-minister-83577
thankful-minister-83577
thankful-minister-83577
thankful-minister-83577
sparse-window-1536
07/19/2022, 9:10 PMthankful-minister-83577
sparse-window-1536
07/19/2022, 9:12 PMsparse-window-1536
07/19/2022, 9:12 PMthankful-minister-83577