Matheus Moreno
07/15/2022, 6:43 PMflyte.secrets/sX
, but no secret is actually being found by the task. How can I debug this?
A little bit more of context: we are injecting environment variables and service account JSONs in the tasks. Because of that, we are actually retrieving secrets directly from /etc/flyte/secrets
, because the SecretsManager()
ends up applying upper()
and lower()
to the keys, which messes up the configuration of the env vars (and files). But apparently no secret is being added to this path...Matheus Moreno
07/15/2022, 6:45 PMMatheus Moreno
07/15/2022, 7:19 PMflyte@afdxj58wfblmgn52glcb-n1-0:/$ cd /etc/flyte/secrets
bash: cd: /etc/flyte/secrets: No such file or directory
flyte@afdxj58wfblmgn52glcb-n1-0:/$ cd /etc/secrets
bash: cd: /etc/secrets: No such file or directory
Matheus Moreno
07/15/2022, 8:14 PMkatrina
Yee
Matheus Moreno
07/15/2022, 8:55 PMYee
Matheus Moreno
07/15/2022, 9:01 PMMatheus Moreno
07/15/2022, 9:02 PMMatheus Moreno
07/15/2022, 9:02 PMMatheus Moreno
07/15/2022, 9:02 PMMatheus Moreno
07/15/2022, 9:03 PMYee
Matheus Moreno
07/15/2022, 10:36 PMMatheus Moreno
07/15/2022, 10:36 PMYee
Yee
Yee
Yee
logger:
level: 5
show-source: true
that is the bit to add for loggingYee
Yee
Yee
Yee
{"json":{"src":"secrets.go:54"},"level":"info","msg":"Failed to inject a secret using injector [Global]. Error: secrets not found - Env [FLYTE_SECRET_TEST-GROUP_TEST-ENV], file [/etc/secrets/test-group/test-env]","ts":"2022-07-18T20:03:47Z"}
{"json":{"src":"secrets.go:54"},"level":"info","msg":"Failed to inject a secret using injector [Global]. Error: secrets not found - Env [FLYTE_SECRET_TEST-GROUP_TEST-FILE], file [/etc/secrets/test-group/test-file]","ts":"2022-07-18T20:03:47Z"}
Yee
Yee
{
"o0": "Hello world, these are my secrets: TESTING_ENV / TESTING_FILE"
}
Yee
Yee
Matheus Moreno
07/18/2022, 8:17 PMMatheus Moreno
07/18/2022, 8:26 PMMatheus Moreno
07/18/2022, 8:26 PMYee
Yee
Yee
Matheus Moreno
07/18/2022, 8:27 PMconfigmap:
logger:
level: 6
show-source: true
Matheus Moreno
07/18/2022, 8:28 PMYee
Yee
storage.yaml: |
logger:
level: 5
show-source: true
storage:
type: minio
...
Yee
Yee
Yee
Matheus Moreno
07/18/2022, 8:31 PMMatheus Moreno
07/18/2022, 8:31 PM/etc/flyte/config $ cat logger.yaml
level: 6
show-source: true
/etc/flyte/config $
Matheus Moreno
07/18/2022, 8:31 PMMatheus Moreno
07/18/2022, 8:32 PMlogger:
field right?Yee
Yee
Yee
Yee
Yee
Matheus Moreno
07/18/2022, 8:34 PMlogger:
and did a helm upgradeYee
Matheus Moreno
07/18/2022, 8:34 PMYee
Yee
Matheus Moreno
07/18/2022, 8:35 PMYee
Matheus Moreno
07/18/2022, 8:38 PMYee
Matheus Moreno
07/18/2022, 8:38 PMMatheus Moreno
07/18/2022, 8:38 PMYee
Matheus Moreno
07/18/2022, 8:39 PMYee
Matheus Moreno
07/18/2022, 8:39 PMYee
Yee
Matheus Moreno
07/18/2022, 8:40 PMYee
Matheus Moreno
07/18/2022, 8:41 PMYee
Yee
Matheus Moreno
07/18/2022, 8:41 PMMatheus Moreno
07/18/2022, 8:41 PMYee
Yee
- name: orsxg4bnm4zg54lql3
secret:
defaultMode: 420
items:
- key: test-file
path: test-file
secretName: test-group
Matheus Moreno
07/18/2022, 8:42 PMMatheus Moreno
07/18/2022, 8:42 PMYee
Yee
Yee
Matheus Moreno
07/18/2022, 8:44 PMMatheus Moreno
07/18/2022, 8:44 PMmetadata:
annotations:
<http://cluster-autoscaler.kubernetes.io/safe-to-evict|cluster-autoscaler.kubernetes.io/safe-to-evict>: "false"
flyte.secrets/s0: ...
flyte.secrets/s1: ...
...
Matheus Moreno
07/18/2022, 8:44 PMflyte.secrets
Matheus Moreno
07/18/2022, 8:44 PMYee
FLYTE_SECRETS_DEFAULT_DIR
env var specified?Matheus Moreno
07/18/2022, 8:47 PMYee
Yee
Matheus Moreno
07/18/2022, 8:48 PMYee
Yee
Yee
Matheus Moreno
07/18/2022, 8:55 PMFLYTE_SECRETS_DEFAULT_DIR
isn't set, so no secret is mounted?Matheus Moreno
07/18/2022, 8:55 PMYee
Yee
Matheus Moreno
07/18/2022, 8:56 PMMatheus Moreno
07/18/2022, 8:57 PMMatheus Moreno
07/18/2022, 8:57 PMYee
Yee
Yee
Yee
Yee
""
Yee
Yee
Yee
Yee
Matheus Moreno
07/18/2022, 9:07 PMMatheus Moreno
07/18/2022, 9:08 PMYee
$ k get mutatingwebhookconfigurations
NAME WEBHOOKS AGE
flyte-pod-webhook 1 2d23h
Yee
Matheus Moreno
07/18/2022, 9:16 PMMatheus Moreno
07/18/2022, 9:17 PMYee
Yee
Matheus Moreno
07/18/2022, 9:17 PMNAME WEBHOOKS AGE
datadog-webhook 2 95d
flyte-pod-webhook 1 4d1h
Matheus Moreno
07/18/2022, 9:17 PMYee
Yee
Matheus Moreno
07/18/2022, 9:18 PMYee
Matheus Moreno
07/18/2022, 9:19 PMYee
Yee
Matheus Moreno
07/18/2022, 9:20 PMMatheus Moreno
07/18/2022, 9:20 PMYee
Yee
Yee
Yee
service:
name: flyte-pod-webhook
namespace: flyte
path: /mutate--v1-pod
port: 443
failurePolicy: Ignore
matchPolicy: Equivalent
name: <http://flyte-pod-webhook.flyte.org|flyte-pod-webhook.flyte.org>
namespaceSelector: {}
objectSelector:
matchLabels:
inject-flyte-secrets: "true"
reinvocationPolicy: Never
rules:
- apiGroups:
- '*'
apiVersions:
- v1
operations:
- CREATE
resources:
- pods
scope: '*'
sideEffects: NoneOnDryRun
timeoutSeconds: 10
Matheus Moreno
07/18/2022, 9:21 PMYee
Matheus Moreno
07/18/2022, 9:21 PMYee
Matheus Moreno
07/18/2022, 9:22 PMYee
kubectl get mutatingwebhookconfigurations flyte-pod-webhook -o yaml
Matheus Moreno
07/18/2022, 9:23 PMservice:
name: flyte-pod-webhook
namespace: ml-dev
path: /mutate--v1-pod
port: 443
failurePolicy: Ignore
matchPolicy: Equivalent
name: <http://flyte-pod-webhook.flyte.org|flyte-pod-webhook.flyte.org>
namespaceSelector: {}
objectSelector:
matchLabels:
inject-flyte-secrets: "true"
reinvocationPolicy: Never
rules:
- apiGroups:
- '*'
apiVersions:
- v1
operations:
- CREATE
resources:
- pods
scope: '*'
sideEffects: NoneOnDryRun
timeoutSeconds: 10
Matheus Moreno
07/18/2022, 9:23 PMYee
ml-dev
?Yee
ml-flyte
?Matheus Moreno
07/18/2022, 9:24 PMMatheus Moreno
07/18/2022, 9:24 PMMatheus Moreno
07/18/2022, 9:25 PMservice:
name: flyte-pod-webhook
namespace: ml-flyte
path: /mutate--v1-pod
port: 443
failurePolicy: Ignore
matchPolicy: Equivalent
name: <http://flyte-pod-webhook.flyte.org|flyte-pod-webhook.flyte.org>
namespaceSelector: {}
objectSelector:
matchLabels:
inject-flyte-secrets: "true"
reinvocationPolicy: Never
rules:
- apiGroups:
- '*'
apiVersions:
- v1
operations:
- CREATE
resources:
- pods
scope: '*'
sideEffects: NoneOnDryRun
timeoutSeconds: 10
Matheus Moreno
07/18/2022, 9:25 PMYee
Matheus Moreno
07/18/2022, 9:26 PMYee
Yee
Yee
Matheus Moreno
07/18/2022, 9:30 PMNo plugin found for Handler-type [python-task], defaulting to [container],
Matheus Moreno
07/18/2022, 9:30 PMMatheus Moreno
07/18/2022, 9:31 PMYee
Yee
Yee
Matheus Moreno
07/18/2022, 9:36 PMYee
inject-flyte-secrets: "true"
in your task pod labels right?Yee
Yee
apiVersion: v1
kind: Pod
Matheus Moreno
07/18/2022, 10:14 PMMatheus Moreno
07/18/2022, 10:19 PMYee
Yee
Yee
Yee
Yee
Yee
Yee
Matheus Moreno
07/18/2022, 10:26 PMjeev
Matheus Moreno
07/18/2022, 10:37 PMMatheus Moreno
07/18/2022, 10:38 PMsecret_requests
. Is there another way?Matheus Moreno
07/18/2022, 10:38 PMml-flyte-projects
.jeev
jeev
Yee
jeev
Yee
Matheus Moreno
07/19/2022, 12:56 AMconfigmap:
k8s:
k8s:
default-pod-template-name: <PodTemplate created in the same namespace as FlytePropeller>
Matheus Moreno
07/19/2022, 12:58 AMjeev
jeev
Matheus Moreno
07/19/2022, 1:00 AMMatheus Moreno
07/19/2022, 3:39 PMjeev
PodTemplate
object still has to be valid, so many of the fields will be required.Matheus Moreno
07/19/2022, 5:28 PMjeev
Yee
Yee
Dan Rammer (hamersaw)
07/19/2022, 6:28 PMMatheus Moreno
07/19/2022, 8:25 PMV1SecretVolumeSource
and a V1VolumeMount
with the required secrets. Since it's a sidecar task, I believe it's something similar to what Jeev is doing right now. For now, it works perfectly! 🙏Yee
Yee
Yee
flyte.secrets/s0: m4zg54lqhiqce4dfon1c1z2sn41xaiqknnsxsoraej1gk32ufvsw34rcbjww54loorpxezlrovuxezlnmvxhioraivhfmx1wifjau
right?Matheus Moreno
07/19/2022, 8:51 PMMatheus Moreno
07/19/2022, 8:52 PMYee
Yee
Yee
Yee
Matheus Moreno
07/19/2022, 9:10 PMYee
Matheus Moreno
07/19/2022, 9:12 PMMatheus Moreno
07/19/2022, 9:12 PMYee