New RunLLM response generated: >>>*# Question* Hi team, In our PROD environment,

flyteadmin

pod often gets restarted due to timed out

Liveness

probe: # Answer The issue with the

flyteadmin

pod restarting due to a timed-out

Liveness

probe is likely related to the probe configuration or the readiness of the services it depends on. According to the flyte_helm_controlplane_generated.yaml file, the

Liveness

probe checks both HTTP and gRPC health endpoints. Here are a few steps to troubleshoot and potentially resolve the issue: 1. Increase Initial Delay: The initial delay for the

Liveness

probe might be too short. Increase the

initialDelaySeconds

to give the service more time to become healthy. 2. Check Service Dependencies: Ensure that all dependencies (like the database and other services) are up and running before

flyteadmin

starts. 3. Review Logs: Check the logs of the

flyteadmin

pod to identify any startup issues or delays. Use

kubectl logs <flyteadmin-pod-name>

to inspect the logs. 4. Probe Configuration: Verify that the health endpoints are correctly configured and accessible. The probe commands should be able to reach

http://127.0.0.1:8088/healthcheck

and the gRPC health check. Example of modifying the `initialDelaySeconds`:

livenessProbe:
  exec:
    command: 
      - sh
      - -c
      - |
        reply=$(curl -s -o /dev/null -w %{http_code} http://127.0.0.1:8088/healthcheck); 
        if [ "$reply" -lt 200 -o "$reply" -ge 400 ]; then exit 1; fi;
        grpc_health_probe -addr=:8089
  initialDelaySeconds: 30  # Increased from 20 to 30 seconds
  periodSeconds: 5

Would you like more detailed guidance on modifying the probe configuration or further steps for troubleshooting? Sources: - flyte_helm_controlplane_generated.yaml - flyte_aws_scheduler_helm_generated.yaml Slack Link https://flyte-org.slack.com/archives/CP2HDHKE1/p1719925444.625439 👍 0 button 👎 0 button 💡 Hint: Mention @runllm in the thread for followups.