Flyte enables production-grade orchestration for machine learning workflows and data processing created to accelerate local workflows to production.

Flyte

image.png

Im having an issue where my task keep `Queued` without any explanation. The previous task was able to be scheduled and complete, as was the creation of the dynamic workflow.
Some more information on my pipeline in the 2nd screenshot.
`flytekit=1.10.1`

• ~How can I debug this, logs of which system pod should I be looking at?~
• How to renew `webhook` cert?

Relevant logs from `flytepropeller`

```{"json":{"exec_id":"f21b2ac0573ec444b9e4","ns":"default-staging","res_ver":"417075896","routine":"worker-18","wf":"default:staging:project_name..flyte.scheduled_workflow.manual_workflow.manual_scheduled_static_wf"},"level":"error","msg":"Error when trying to reconcile workflow. Error [failed at Node[dn0]. RuntimeExecutionError: failed during plugin execution, caused by: failed to execute handle for plugin [sidecar]: [InternalError] failed to create resource, caused by: Internal error occurred: failed calling webhook \"flyte-pod-webhook.flyte.org\": failed to call webhook: Post \"<https://flyte-pod-webhook.flyte.svc:443/mutate--v1-pod?timeout=10s>\": tls: failed to verify certificate: x509: certificate has expired or is not yet valid: current time 2024-04-22T17:13:08Z is after 2024-04-03T11:48:49Z]. Error Type[*errors.NodeErrorWithCause]","ts":"2024-04-22T17:13:08Z"}```

`flyte-pod-webhook`


```2024/04/22 16:36:48 http: TLS handshake error from 10.24.2.4:59964: EOF
2024/04/22 16:37:07 http: TLS handshake error from 10.24.3.21:56912: EOF
2024/04/22 16:39:07 http: TLS handshake error from 10.24.2.4:53208: remote error: tls: bad certificate
2024/04/22 16:41:08 http: TLS handshake error from 10.24.3.21:34860: EOF
2024/04/22 16:48:55 http: TLS handshake error from 10.24.3.21:41614: EOF
2024/04/22 16:49:13 http: TLS handshake error from 10.24.3.21:32886: EOF
2024/04/22 16:51:14 http: TLS handshake error from 10.24.2.4:58510: remote error: tls: bad certificate
2024/04/22 16:53:14 http: TLS handshake error from 10.24.2.4:60120: EOF
2024/04/22 16:56:01 http: TLS handshake error from 10.24.2.4:60434: EOF```

Since this task requires spinning up an explicit compute node this could also be related

I think this is it <https://github.com/flyteorg/flyte/issues/2871>

have you tried restarting the deployment? not sure if the role of the `generate-secrets` init container (<https://github.com/flyteorg/flyte/blob/e3e614a7c7505e473144fa2fea720953b558c641/charts/flyte-core/templates/propeller/webhook.yaml#L47|source>) is precisely to generate the cert

I'm assuming this is a`flyte-core` deployment