Hi all I m very happy with the flyte community which seems t

Question

Hi all I m very happy with the flyte community which seems to be very active slightly smiling face I m trying to deploy flyte binary on EKS using the terraform template provided here by < David Espejo> and some fixes from < Uria Franko> The flyte binary pod seems to be starting and I don t see any critical error in the logs however the pod is restarted `CrashLoopBackoff` because the liveness amp readiness probes fail to communicate with it E g `Liveness probe failed Get dial tcp 10 3 125 74 8088 connect connection refused` Has anyone else witnessed the same behavior Or may I have misconfigured either the terraform template or the helm chart

Yee · Answer

get the logs ` p`

Quentin Chenevier · Answer

Here are the logs I get with `kubectl logs flyte binary b958c88b6 mh2nh n flyte c flyte`

Quentin Chenevier · Answer

The thing I m investigating right now is that I m having doubts about the values I ve put there for OIDC in the helm configuration <https github com flyteorg flyte blob master charts flyte binary eks production yaml L23 L29>

Yee · Answer

` p`

Yee · Answer

well `get pod` first want to see if there are restarts

Yee · Answer

and if there are restarts then ` p` to see the previous logs

Quentin Chenevier · Answer

haaaaaa I understand now why the previous logs are interesting to watch Smart

Quentin Chenevier · Answer

logs

Quentin Chenevier · Answer

I forgot to say that there are restarts indeed since the liveness amp readiness probes fail to connect the pod is restarted

David Espejo (he/him) · Answer

< Quentin Chenevier> is that the `baseUrl`that you re using bc that won t work in your environment

Quentin Chenevier · Answer

Nope I ve put the OIDC Connect provider url shown on the cluster page in EKS console

Yee · Answer

we need to be better about errors

Yee · Answer

Yee · Answer

that error is a terminating error

Yee · Answer

can you make it so that that error goes away

Yee · Answer

unf in the single binary we don t capture the error and restart then it would be obvious

Yee · Answer

the thread just dies

Yee · Answer

disappointed

Quentin Chenevier · Answer

Haaaaa so I ve misconfigured the thing OIDC is not found

Quentin Chenevier · Answer

It works

Quentin Chenevier · Answer

Since I m very new to kube I m not very used to digg into logs Thanks for helping finding the root cause and I guess it s time for me to go to sleep now

Yee · Answer

sure let us know

Yee · Answer

we definitely recommend doing auth last

Yee · Answer

as it s the trickiest bit to get right

Quentin Chenevier · Answer

What was strange was seeing the pod as `Running` I thought that if the probes couldn t reach it was due to a networking issue But indeed the pod wasn t really running

Quentin Chenevier · Answer

< Yee> Yeah thanks for the tip I m learning on the way sweat smile

Yee · Answer

yeah we need to add a panic to serve probably and recover in the go routine <https github com flyteorg flyte blob e57cac0990fe5ec321e590fc49147014827e6dfd cmd single start go L95>