Matheus Moreno
07/15/2022, 6:43 PMflyte.secrets/sX
, but no secret is actually being found by the task. How can I debug this?
A little bit more of context: we are injecting environment variables and service account JSONs in the tasks. Because of that, we are actually retrieving secrets directly from /etc/flyte/secrets
, because the SecretsManager()
ends up applying upper()
and lower()
to the keys, which messes up the configuration of the env vars (and files). But apparently no secret is being added to this path...flyte@afdxj58wfblmgn52glcb-n1-0:/$ cd /etc/flyte/secrets
bash: cd: /etc/flyte/secrets: No such file or directory
flyte@afdxj58wfblmgn52glcb-n1-0:/$ cd /etc/secrets
bash: cd: /etc/secrets: No such file or directory
katrina
Yee
Matheus Moreno
07/15/2022, 8:55 PMYee
Matheus Moreno
07/15/2022, 9:01 PMYee
Matheus Moreno
07/15/2022, 10:36 PMYee
logger:
level: 5
show-source: true
that is the bit to add for logging{"json":{"src":"secrets.go:54"},"level":"info","msg":"Failed to inject a secret using injector [Global]. Error: secrets not found - Env [FLYTE_SECRET_TEST-GROUP_TEST-ENV], file [/etc/secrets/test-group/test-env]","ts":"2022-07-18T20:03:47Z"}
{"json":{"src":"secrets.go:54"},"level":"info","msg":"Failed to inject a secret using injector [Global]. Error: secrets not found - Env [FLYTE_SECRET_TEST-GROUP_TEST-FILE], file [/etc/secrets/test-group/test-file]","ts":"2022-07-18T20:03:47Z"}
{
"o0": "Hello world, these are my secrets: TESTING_ENV / TESTING_FILE"
}
Matheus Moreno
07/18/2022, 8:17 PMYee
Matheus Moreno
07/18/2022, 8:27 PMconfigmap:
logger:
level: 6
show-source: true
Yee
storage.yaml: |
logger:
level: 5
show-source: true
storage:
type: minio
...
Matheus Moreno
07/18/2022, 8:31 PM/etc/flyte/config $ cat logger.yaml
level: 6
show-source: true
/etc/flyte/config $
logger:
field right?Yee
Matheus Moreno
07/18/2022, 8:34 PMlogger:
and did a helm upgradeYee
Matheus Moreno
07/18/2022, 8:34 PMYee
Matheus Moreno
07/18/2022, 8:35 PMYee
Matheus Moreno
07/18/2022, 8:38 PMYee
Matheus Moreno
07/18/2022, 8:38 PMYee
Matheus Moreno
07/18/2022, 8:39 PMYee
Matheus Moreno
07/18/2022, 8:39 PMYee
Matheus Moreno
07/18/2022, 8:40 PMYee
Matheus Moreno
07/18/2022, 8:41 PMYee
Matheus Moreno
07/18/2022, 8:41 PMYee
- name: orsxg4bnm4zg54lql3
secret:
defaultMode: 420
items:
- key: test-file
path: test-file
secretName: test-group
Matheus Moreno
07/18/2022, 8:42 PMYee
Matheus Moreno
07/18/2022, 8:44 PMmetadata:
annotations:
<http://cluster-autoscaler.kubernetes.io/safe-to-evict|cluster-autoscaler.kubernetes.io/safe-to-evict>: "false"
flyte.secrets/s0: ...
flyte.secrets/s1: ...
...
flyte.secrets
Yee
FLYTE_SECRETS_DEFAULT_DIR
env var specified?Matheus Moreno
07/18/2022, 8:47 PMYee
Matheus Moreno
07/18/2022, 8:48 PMYee
Matheus Moreno
07/18/2022, 8:55 PMFLYTE_SECRETS_DEFAULT_DIR
isn't set, so no secret is mounted?Yee
Matheus Moreno
07/18/2022, 8:56 PMYee
""
Matheus Moreno
07/18/2022, 9:07 PMYee
$ k get mutatingwebhookconfigurations
NAME WEBHOOKS AGE
flyte-pod-webhook 1 2d23h
Matheus Moreno
07/18/2022, 9:16 PMYee
Matheus Moreno
07/18/2022, 9:17 PMNAME WEBHOOKS AGE
datadog-webhook 2 95d
flyte-pod-webhook 1 4d1h
Yee
Matheus Moreno
07/18/2022, 9:18 PMYee
Matheus Moreno
07/18/2022, 9:19 PMYee
Matheus Moreno
07/18/2022, 9:20 PMYee
service:
name: flyte-pod-webhook
namespace: flyte
path: /mutate--v1-pod
port: 443
failurePolicy: Ignore
matchPolicy: Equivalent
name: <http://flyte-pod-webhook.flyte.org|flyte-pod-webhook.flyte.org>
namespaceSelector: {}
objectSelector:
matchLabels:
inject-flyte-secrets: "true"
reinvocationPolicy: Never
rules:
- apiGroups:
- '*'
apiVersions:
- v1
operations:
- CREATE
resources:
- pods
scope: '*'
sideEffects: NoneOnDryRun
timeoutSeconds: 10
Matheus Moreno
07/18/2022, 9:21 PMYee
Matheus Moreno
07/18/2022, 9:21 PMYee
Matheus Moreno
07/18/2022, 9:22 PMYee
kubectl get mutatingwebhookconfigurations flyte-pod-webhook -o yaml
Matheus Moreno
07/18/2022, 9:23 PMservice:
name: flyte-pod-webhook
namespace: ml-dev
path: /mutate--v1-pod
port: 443
failurePolicy: Ignore
matchPolicy: Equivalent
name: <http://flyte-pod-webhook.flyte.org|flyte-pod-webhook.flyte.org>
namespaceSelector: {}
objectSelector:
matchLabels:
inject-flyte-secrets: "true"
reinvocationPolicy: Never
rules:
- apiGroups:
- '*'
apiVersions:
- v1
operations:
- CREATE
resources:
- pods
scope: '*'
sideEffects: NoneOnDryRun
timeoutSeconds: 10
Yee
ml-dev
?ml-flyte
?Matheus Moreno
07/18/2022, 9:24 PMservice:
name: flyte-pod-webhook
namespace: ml-flyte
path: /mutate--v1-pod
port: 443
failurePolicy: Ignore
matchPolicy: Equivalent
name: <http://flyte-pod-webhook.flyte.org|flyte-pod-webhook.flyte.org>
namespaceSelector: {}
objectSelector:
matchLabels:
inject-flyte-secrets: "true"
reinvocationPolicy: Never
rules:
- apiGroups:
- '*'
apiVersions:
- v1
operations:
- CREATE
resources:
- pods
scope: '*'
sideEffects: NoneOnDryRun
timeoutSeconds: 10
Yee
Matheus Moreno
07/18/2022, 9:26 PMYee
Matheus Moreno
07/18/2022, 9:30 PMNo plugin found for Handler-type [python-task], defaulting to [container],
Yee
Matheus Moreno
07/18/2022, 9:36 PMYee
inject-flyte-secrets: "true"
in your task pod labels right?apiVersion: v1
kind: Pod
Matheus Moreno
07/18/2022, 10:14 PMYee
Matheus Moreno
07/18/2022, 10:26 PMjeev
Matheus Moreno
07/18/2022, 10:37 PMsecret_requests
. Is there another way?ml-flyte-projects
.jeev
Yee
jeev
Yee
Matheus Moreno
07/19/2022, 12:56 AMconfigmap:
k8s:
k8s:
default-pod-template-name: <PodTemplate created in the same namespace as FlytePropeller>
jeev
Matheus Moreno
07/19/2022, 1:00 AMjeev
PodTemplate
object still has to be valid, so many of the fields will be required.Matheus Moreno
07/19/2022, 5:28 PMjeev
Yee
Dan Rammer (hamersaw)
07/19/2022, 6:28 PMMatheus Moreno
07/19/2022, 8:25 PMV1SecretVolumeSource
and a V1VolumeMount
with the required secrets. Since it's a sidecar task, I believe it's something similar to what Jeev is doing right now. For now, it works perfectly! 🙏Yee
flyte.secrets/s0: m4zg54lqhiqce4dfon1c1z2sn41xaiqknnsxsoraej1gk32ufvsw34rcbjww54loorpxezlrovuxezlnmvxhioraivhfmx1wifjau
right?Matheus Moreno
07/19/2022, 8:51 PMYee
Matheus Moreno
07/19/2022, 9:10 PMYee
Matheus Moreno
07/19/2022, 9:12 PMYee