Katrina P
07/06/2022, 6:02 PMflyte-pod-webook
won't come up due to: Failed to start webhook. Error: open /etc/webhook/certs/tls.crt: no such file or directory.
Anyone have some insight into this?Haytham Abuelfutuh
Katrina P
07/06/2022, 9:14 PMgenerate-secrets
pod:
`
time="2022-07-06T21:17:16Z" level=info msg=------------------------------------------------------------------------
time="2022-07-06T21:17:16Z" level=info msg="App [flytepropeller], Version [unknown], BuildSHA [unknown], BuildTS [2022-07-06 21:17:16.101877907 +0000 UTC m=+0.038773580]"
time="2022-07-06T21:17:16Z" level=info msg=------------------------------------------------------------------------
time="2022-07-06T21:17:16Z" level=info msg="Detected: 2 CPU's\n"
Haytham Abuelfutuh
kubectl logs -n flyte deploy/flytepropeller-webhook -c generate-secrets
is that what you are running?Katrina P
07/06/2022, 9:31 PMtime="2022-07-06T21:28:44Z" level=info msg=------------------------------------------------------------------------
time="2022-07-06T21:28:44Z" level=info msg="App [flytepropeller], Version [unknown], BuildSHA [unknown], BuildTS [2022-07-06 21:28:44.093828325 +0000 UTC m=+0.029201245]"
time="2022-07-06T21:28:44Z" level=info msg=------------------------------------------------------------------------
time="2022-07-06T21:28:44Z" level=info msg="Detected: 2 CPU's\n"
{"metrics-prefix":"flyte:","certDir":"/etc/webhook/certs","localCert":false,"listenPort":9443,"serviceName":"flyte-pod-webhook","servicePort":443,"secretName":"flyte-pod-webhook","secretManagerType":"K8s","awsSecretManager":{"sidecarImage":"<http://docker.io/amazon/aws-secrets-manager-secret-sidecar:v0.1.4|docker.io/amazon/aws-secrets-manager-secret-sidecar:v0.1.4>","resources":{"limits":{"cpu":"200m","memory":"500Mi"},"requests":{"cpu":"200m","memory":"500Mi"}}},"vaultSecretManager":{"role":"flyte","kvVersion":"2"}}
{"json":{},"level":"fatal","msg":"Failed to start webhook. Error: open /etc/webhook/certs/tls.crt: no such file or directory","ts":"2022-07-06T21:28:44Z"}
I have no idea why its referencing aws secrets manager there though; misconfigured something maybe?Shahwar Saleem
07/06/2022, 9:35 PMvalues.yaml
solved my problem.Haytham Abuelfutuh
Katrina P
07/06/2022, 9:38 PMkatrina
Katrina P
07/06/2022, 9:59 PM{"json":{"src":"viper.go:398"},"level":"debug","msg":"Config section [storage] updated. No update handler registered.","ts":"2022-07-06T21:54:08Z"}
{"json":{"src":"init_cert.go:50"},"level":"info","msg":"Issuing certs","ts":"2022-07-06T21:54:08Z"}
{"json":{"src":"init_cert.go:61"},"level":"info","msg":"Creating secret [flyte-pod-webhook] in Namespace [flyte]","ts":"2022-07-06T21:54:11Z"}
{"json":{"src":"init_cert.go:115"},"level":"info","msg":"A secret already exists with the same name. Validating.","ts":"2022-07-06T21:54:11Z"}
Haytham Abuelfutuh
Katrina P
07/06/2022, 10:00 PMtime="2022-07-06T21:57:08Z" level=info msg=------------------------------------------------------------------------
time="2022-07-06T21:57:08Z" level=info msg="App [flytepropeller], Version [unknown], BuildSHA [unknown], BuildTS [2022-07-06 21:57:08.170414367 +0000 UTC m=+0.033019880]"
time="2022-07-06T21:57:08Z" level=info msg=------------------------------------------------------------------------
time="2022-07-06T21:57:08Z" level=info msg="Detected: 2 CPU's\n"
{"json":{"src":"viper.go:398"},"level":"debug","msg":"Config section [plugins] updated. No update handler registered.","ts":"2022-07-06T21:57:08Z"}
{"json":{"src":"viper.go:398"},"level":"debug","msg":"Config section [plugins.aws] updated. No update handler registered.","ts":"2022-07-06T21:57:08Z"}
{"json":{"src":"viper.go:398"},"level":"debug","msg":"Config section [plugins.aws.batch] updated. No update handler registered.","ts":"2022-07-06T21:57:08Z"}
{"json":{"src":"viper.go:398"},"level":"debug","msg":"Config section [plugins.sagemaker] updated. No update handler registered.","ts":"2022-07-06T21:57:08Z"}
{"json":{"src":"viper.go:398"},"level":"debug","msg":"Config section [plugins.bigquery] updated. No update handler registered.","ts":"2022-07-06T21:57:08Z"}
{"json":{"src":"viper.go:398"},"level":"debug","msg":"Config section [plugins.snowflake] updated. No update handler registered.","ts":"2022-07-06T21:57:08Z"}
{"json":{"src":"viper.go:398"},"level":"debug","msg":"Config section [plugins.catalogcache] updated. No update handler registered.","ts":"2022-07-06T21:57:08Z"}
{"json":{"src":"viper.go:398"},"level":"debug","msg":"Config section [plugins.k8s] updated. No update handler registered.","ts":"2022-07-06T21:57:08Z"}
{"json":{"src":"viper.go:398"},"level":"debug","msg":"Config section [plugins.logs] updated. No update handler registered.","ts":"2022-07-06T21:57:08Z"}
{"json":{"src":"viper.go:398"},"level":"debug","msg":"Config section [plugins.k8s-array] updated. No update handler registered.","ts":"2022-07-06T21:57:08Z"}
{"json":{"src":"viper.go:398"},"level":"debug","msg":"Config section [plugins.qubole] updated. No update handler registered.","ts":"2022-07-06T21:57:08Z"}
{"json":{"src":"viper.go:398"},"level":"debug","msg":"Config section [plugins.spark] updated. No update handler registered.","ts":"2022-07-06T21:57:08Z"}
{"json":{"src":"viper.go:398"},"level":"debug","msg":"Config section [plugins.athena] updated. No update handler registered.","ts":"2022-07-06T21:57:08Z"}
{"json":{"src":"viper.go:398"},"level":"debug","msg":"Config section [secrets] updated. No update handler registered.","ts":"2022-07-06T21:57:08Z"}
{"json":{"src":"viper.go:400"},"level":"debug","msg":"Config section [admin] updated. Firing updated event.","ts":"2022-07-06T21:57:08Z"}
{"json":{"src":"viper.go:398"},"level":"debug","msg":"Config section [event] updated. No update handler registered.","ts":"2022-07-06T21:57:08Z"}
{"json":{"src":"viper.go:398"},"level":"debug","msg":"Config section [catalog-cache] updated. No update handler registered.","ts":"2022-07-06T21:57:08Z"}
{"json":{"src":"viper.go:398"},"level":"debug","msg":"Config section [storage] updated. No update handler registered.","ts":"2022-07-06T21:57:08Z"}
{"json":{"src":"viper.go:398"},"level":"debug","msg":"Config section [propeller] updated. No update handler registered.","ts":"2022-07-06T21:57:08Z"}
{"json":{"src":"viper.go:398"},"level":"debug","msg":"Config section [propeller.admin-launcher] updated. No update handler registered.","ts":"2022-07-06T21:57:08Z"}
{"json":{"src":"viper.go:398"},"level":"debug","msg":"Config section [propeller.resourcemanager] updated. No update handler registered.","ts":"2022-07-06T21:57:08Z"}
{"json":{"src":"viper.go:398"},"level":"debug","msg":"Config section [propeller.workflowstore] updated. No update handler registered.","ts":"2022-07-06T21:57:08Z"}
{"json":{"src":"viper.go:398"},"level":"debug","msg":"Config section [webhook] updated. No update handler registered.","ts":"2022-07-06T21:57:08Z"}
{"metrics-prefix":"flyte:","certDir":"/etc/webhook/certs","localCert":false,"listenPort":9443,"serviceName":"flyte-pod-webhook","servicePort":443,"secretName":"flyte-pod-webhook","secretManagerType":"K8s","awsSecretManager":{"sidecarImage":"<http://docker.io/amazon/aws-secrets-manager-secret-sidecar:v0.1.4|docker.io/amazon/aws-secrets-manager-secret-sidecar:v0.1.4>","resources":{"limits":{"cpu":"200m","memory":"500Mi"},"requests":{"cpu":"200m","memory":"500Mi"}}},"vaultSecretManager":{"role":"flyte","kvVersion":"2"}}
{"json":{"src":"server.go:96"},"level":"info","msg":"Starting profiling server on port [10254]","ts":"2022-07-06T21:57:08Z"}
{"json":{"src":"entrypoint.go:88"},"level":"info","msg":"Creating MutatingWebhookConfiguration [flyte/flyte-pod-webhook]","ts":"2022-07-06T21:57:08Z"}
{"json":{"src":"entrypoint.go:93"},"level":"info","msg":"Failed to create MutatingWebhookConfiguration. Will attempt to update. Error: <http://mutatingwebhookconfigurations.admissionregistration.k8s.io|mutatingwebhookconfigurations.admissionregistration.k8s.io> \"flyte-pod-webhook\" already exists","ts":"2022-07-06T21:57:08Z"}
{"json":{"src":"entrypoint.go:103"},"level":"info","msg":"Successfully updated existing mutating webhook config.","ts":"2022-07-06T21:57:08Z"}
{"json":{"src":"pod.go:143"},"level":"info","msg":"Registering path [/mutate--v1-pod]","ts":"2022-07-06T21:57:08Z"}
{"json":{"src":"entrypoint.go:55"},"level":"info","msg":"Starting controller-runtime manager","ts":"2022-07-06T21:57:08Z"}
{"json":{"src":"webhook.go:119"},"level":"fatal","msg":"Failed to start webhook. Error: open /etc/webhook/certs/tls.crt: no such file or directory","ts":"2022-07-06T21:57:08Z"}
Haytham Abuelfutuh
kubectl get secret -n flyte flyte-propeller-webhook -o yaml
tls.crt
kubectl get deploy -n flyte flyte-propeller-webhook -o yaml
Katrina P
07/06/2022, 10:02 PMHaytham Abuelfutuh
Katrina P
07/06/2022, 10:03 PM# Source: flyte-core/templates/propeller/webhook.yaml
# Create the actual deployment
apiVersion: apps/v1
kind: Deployment
metadata:
name: flyte-pod-webhook
namespace: flyte
labels:
app: flyte-pod-webhook
spec:
selector:
matchLabels:
app: flyte-pod-webhook
template:
metadata:
labels:
app: flyte-pod-webhook
<http://app.kubernetes.io/name|app.kubernetes.io/name>: flyte-pod-webhook
<http://app.kubernetes.io/version|app.kubernetes.io/version>: v1.0.0
annotations:
configChecksum: "94a14941954f8d44256768b8600c65108c984e8ea369ed3f61b79ffbbebfc6b"
spec:
securityContext:
fsGroup: 65534
runAsUser: 1001
fsGroupChangePolicy: "Always"
serviceAccountName: flyte-pod-webhook
initContainers:
- name: generate-secrets
image: "<http://cr.flyte.org/flyteorg/flytepropeller:v1.0.0|cr.flyte.org/flyteorg/flytepropeller:v1.0.0>"
imagePullPolicy: "IfNotPresent"
command:
- flytepropeller
args:
- webhook
- init-certs
- --config
- /etc/flyte/config/*.yaml
env:
- name: POD_NAME
valueFrom:
fieldRef:
fieldPath: metadata.name
- name: POD_NAMESPACE
valueFrom:
fieldRef:
fieldPath: metadata.namespace
volumeMounts:
- name: config-volume
mountPath: /etc/flyte/config
containers:
- name: webhook
image: "<http://cr.flyte.org/flyteorg/flytepropeller:v1.0.0|cr.flyte.org/flyteorg/flytepropeller:v1.0.0>"
imagePullPolicy: "IfNotPresent"
command:
- flytepropeller
args:
- webhook
- --config
- /etc/flyte/config/*.yaml
env:
- name: POD_NAME
valueFrom:
fieldRef:
fieldPath: metadata.name
- name: POD_NAMESPACE
valueFrom:
fieldRef:
fieldPath: metadata.namespace
volumeMounts:
- name: config-volume
mountPath: /etc/flyte/config
readOnly: true
- name: webhook-certs
mountPath: /etc/webhook/certs
readOnly: true
volumes:
- name: config-volume
configMap:
name: flyte-propeller-config
- name: webhook-certs
secret:
secretName: flyte-pod-webhook
nodeSelector:
role: flyte
Haytham Abuelfutuh
Shahwar Saleem
07/06/2022, 10:06 PMKatrina P
07/06/2022, 10:06 PMapiVersion: v1
kind: Pod
metadata:
annotations:
<http://artifact.spinnaker.io/location|artifact.spinnaker.io/location>: flyte
<http://artifact.spinnaker.io/name|artifact.spinnaker.io/name>: flyte-pod-webhook
<http://artifact.spinnaker.io/type|artifact.spinnaker.io/type>: kubernetes/deployment
<http://artifact.spinnaker.io/version|artifact.spinnaker.io/version>: ''
<http://cni.projectcalico.org/containerID|cni.projectcalico.org/containerID>: d7dd05a9a4e3e73642e66acc045a8e92c9a2ea631c09a1ece4593d49ea152ab8
configChecksum: 94a14941954f8d44256768b8600c65108c984e8ea369ed3f61b79ffbbebfc6b
<http://moniker.spinnaker.io/application|moniker.spinnaker.io/application>: flyte
<http://moniker.spinnaker.io/cluster|moniker.spinnaker.io/cluster>: deployment flyte-pod-webhook
creationTimestamp: '2022-07-06T22:51:06Z'
generateName: flyte-pod-webhook-dcbfbdf9-
labels:
app: flyte-pod-webhook
<http://app.kubernetes.io/managed-by|app.kubernetes.io/managed-by>: spinnaker
<http://app.kubernetes.io/name|app.kubernetes.io/name>: flyte-pod-webhook
<http://app.kubernetes.io/version|app.kubernetes.io/version>: v1.1.0
pod-template-hash: dcbfbdf9
managedFields:
- apiVersion: v1
fieldsType: FieldsV1
fieldsV1:
'f:metadata':
'f:annotations':
'f:<http://cni.projectcalico.org/containerID|cni.projectcalico.org/containerID>': {}
'f:<http://cni.projectcalico.org/podIP|cni.projectcalico.org/podIP>': {}
'f:<http://cni.projectcalico.org/podIPs|cni.projectcalico.org/podIPs>': {}
manager: calico
operation: Update
subresource: status
time: '2022-07-06T22:51:06Z'
- apiVersion: v1
fieldsType: FieldsV1
fieldsV1:
'f:metadata':
'f:annotations':
.: {}
'f:<http://artifact.spinnaker.io/location|artifact.spinnaker.io/location>': {}
'f:<http://artifact.spinnaker.io/name|artifact.spinnaker.io/name>': {}
'f:<http://artifact.spinnaker.io/type|artifact.spinnaker.io/type>': {}
'f:<http://artifact.spinnaker.io/version|artifact.spinnaker.io/version>': {}
'f:configChecksum': {}
'f:<http://moniker.spinnaker.io/application|moniker.spinnaker.io/application>': {}
'f:<http://moniker.spinnaker.io/cluster|moniker.spinnaker.io/cluster>': {}
'f:generateName': {}
'f:labels':
.: {}
'f:app': {}
'f:<http://app.kubernetes.io/managed-by|app.kubernetes.io/managed-by>': {}
'f:<http://app.kubernetes.io/name|app.kubernetes.io/name>': {}
'f:<http://app.kubernetes.io/version|app.kubernetes.io/version>': {}
'f:pod-template-hash': {}
'f:ownerReferences':
.: {}
'k:{"uid":"1098743a-7e43-449b-8d8e-5e02595325a7"}': {}
'f:spec':
'f:containers':
'k:{"name":"webhook"}':
.: {}
'f:args': {}
'f:command': {}
'f:env':
.: {}
'k:{"name":"POD_NAME"}':
.: {}
'f:name': {}
'f:valueFrom':
.: {}
'f:fieldRef': {}
'k:{"name":"POD_NAMESPACE"}':
.: {}
'f:name': {}
'f:valueFrom':
.: {}
'f:fieldRef': {}
'f:image': {}
'f:imagePullPolicy': {}
'f:name': {}
'f:resources': {}
'f:terminationMessagePath': {}
'f:terminationMessagePolicy': {}
'f:volumeMounts':
.: {}
'k:{"mountPath":"/etc/flyte/config"}':
.: {}
'f:mountPath': {}
'f:name': {}
'f:readOnly': {}
'k:{"mountPath":"/etc/webhook/certs"}':
.: {}
'f:mountPath': {}
'f:name': {}
'f:readOnly': {}
'f:dnsPolicy': {}
'f:enableServiceLinks': {}
'f:initContainers':
.: {}
'k:{"name":"generate-secrets"}':
.: {}
'f:args': {}
'f:command': {}
'f:env':
.: {}
'k:{"name":"POD_NAME"}':
.: {}
'f:name': {}
'f:valueFrom':
.: {}
'f:fieldRef': {}
'k:{"name":"POD_NAMESPACE"}':
.: {}
'f:name': {}
'f:valueFrom':
.: {}
'f:fieldRef': {}
'f:image': {}
'f:imagePullPolicy': {}
'f:name': {}
'f:resources': {}
'f:terminationMessagePath': {}
'f:terminationMessagePolicy': {}
'f:volumeMounts':
.: {}
'k:{"mountPath":"/etc/flyte/config"}':
.: {}
'f:mountPath': {}
'f:name': {}
'f:nodeSelector': {}
'f:restartPolicy': {}
'f:schedulerName': {}
'f:securityContext':
.: {}
'f:fsGroup': {}
'f:fsGroupChangePolicy': {}
'f:runAsUser': {}
'f:serviceAccount': {}
'f:serviceAccountName': {}
'f:terminationGracePeriodSeconds': {}
'f:volumes':
.: {}
'k:{"name":"config-volume"}':
.: {}
'f:configMap':
.: {}
'f:defaultMode': {}
'f:name': {}
'f:name': {}
'k:{"name":"webhook-certs"}':
.: {}
'f:name': {}
'f:secret':
.: {}
'f:defaultMode': {}
'f:secretName': {}
manager: kube-controller-manager
operation: Update
time: '2022-07-06T22:51:06Z'
- apiVersion: v1
fieldsType: FieldsV1
fieldsV1:
'f:status':
'f:conditions':
'k:{"type":"ContainersReady"}':
.: {}
'f:lastProbeTime': {}
'f:lastTransitionTime': {}
'f:message': {}
'f:reason': {}
'f:status': {}
'f:type': {}
'k:{"type":"Initialized"}':
.: {}
'f:lastProbeTime': {}
'f:lastTransitionTime': {}
'f:status': {}
'f:type': {}
'k:{"type":"Ready"}':
.: {}
'f:lastProbeTime': {}
'f:lastTransitionTime': {}
'f:message': {}
'f:reason': {}
'f:status': {}
'f:type': {}
'f:containerStatuses': {}
'f:hostIP': {}
'f:initContainerStatuses': {}
'f:phase': {}
'f:podIP': {}
'f:podIPs':
.: {}
'k:{"ip":"192.168.243.225"}':
.: {}
'f:ip': {}
'f:startTime': {}
manager: kubelet
operation: Update
subresource: status
time: '2022-07-07T15:57:08Z'
name: flyte-pod-webhook-dcbfbdf9-p5j6g
namespace: flyte
ownerReferences:
- apiVersion: apps/v1
blockOwnerDeletion: true
controller: true
kind: ReplicaSet
name: flyte-pod-webhook-dcbfbdf9
uid: 1098743a-7e43-449b-8d8e-5e02595325a7
resourceVersion: '210870974'
uid: abbad89c-13ef-410c-be39-bafb3a9db183
spec:
containers:
- args:
- webhook
- '--config'
- /etc/flyte/config/*.yaml
command:
- flytepropeller
env:
- name: POD_NAME
valueFrom:
fieldRef:
apiVersion: v1
fieldPath: metadata.name
- name: POD_NAMESPACE
valueFrom:
fieldRef:
apiVersion: v1
fieldPath: metadata.namespace
image: '<http://cr.flyte.org/flyteorg/flytepropeller:v1.1.0|cr.flyte.org/flyteorg/flytepropeller:v1.1.0>'
imagePullPolicy: IfNotPresent
name: webhook
resources: {}
terminationMessagePath: /dev/termination-log
terminationMessagePolicy: File
volumeMounts:
- mountPath: /etc/flyte/config
name: config-volume
readOnly: true
- mountPath: /etc/webhook/certs
name: webhook-certs
readOnly: true
- mountPath: /var/run/secrets/kubernetes.io/serviceaccount
name: kube-api-access-454tj
readOnly: true
dnsPolicy: ClusterFirst
enableServiceLinks: true
initContainers:
- args:
- webhook
- init-certs
- '--config'
- /etc/flyte/config/*.yaml
command:
- flytepropeller
env:
- name: POD_NAME
valueFrom:
fieldRef:
apiVersion: v1
fieldPath: metadata.name
- name: POD_NAMESPACE
valueFrom:
fieldRef:
apiVersion: v1
fieldPath: metadata.namespace
image: '<http://cr.flyte.org/flyteorg/flytepropeller:v1.1.0|cr.flyte.org/flyteorg/flytepropeller:v1.1.0>'
imagePullPolicy: IfNotPresent
name: generate-secrets
resources: {}
terminationMessagePath: /dev/termination-log
terminationMessagePolicy: File
volumeMounts:
- mountPath: /etc/flyte/config
name: config-volume
- mountPath: /var/run/secrets/kubernetes.io/serviceaccount
name: kube-api-access-454tj
readOnly: true
nodeName: ip-10-1-2-146.us-east-2.compute.internal
nodeSelector:
role: flyte
preemptionPolicy: PreemptLowerPriority
priority: 0
restartPolicy: Always
schedulerName: default-scheduler
securityContext:
fsGroup: 65534
fsGroupChangePolicy: Always
runAsUser: 1001
serviceAccount: flyte-pod-webhook
serviceAccountName: flyte-pod-webhook
terminationGracePeriodSeconds: 30
tolerations:
- effect: NoExecute
key: <http://node.kubernetes.io/not-ready|node.kubernetes.io/not-ready>
operator: Exists
tolerationSeconds: 300
- effect: NoExecute
key: <http://node.kubernetes.io/unreachable|node.kubernetes.io/unreachable>
operator: Exists
tolerationSeconds: 300
volumes:
- configMap:
defaultMode: 420
name: flyte-propeller-config-v001
name: config-volume
- name: webhook-certs
secret:
defaultMode: 420
secretName: flyte-pod-webhook-v000
- name: kube-api-access-454tj
projected:
defaultMode: 420
sources:
- serviceAccountToken:
expirationSeconds: 3607
path: token
- configMap:
items:
- key: ca.crt
path: ca.crt
name: kube-root-ca.crt
- downwardAPI:
items:
- fieldRef:
apiVersion: v1
fieldPath: metadata.namespace
path: namespace
Ketan (kumare3)
Katrina P
07/11/2022, 2:58 PMKetan (kumare3)