https://flyte.org
Join Slack
Hi all, I'm very happy with the flyte community which seems to be very active. :slightly_smiling_fac...
# flyte-support
r
Hi all, I'm very happy with the flyte community which seems to be very active. 🙂 I'm trying to deploy flyte-binary on EKS using the terraform template provided here: https://github.com/unionai-oss/deploy-flyte/tree/main/environments/aws by @cold-lock-43986 (and some fixes from @hallowed-autumn-63270 ). The flyte-binary pod seems to be starting and I don't see any critical error in the logs, however the pod is restarted (
CrashLoopBackoff
) because the liveness & readiness probes fail to communicate with it. E.g: 
Liveness probe failed: Get "<http://10.3.125.74:8088/healthcheck>": dial tcp 10.3.125.74:8088: connect: connection refused
Has anyone else witnessed the same behavior ? Or may I have misconfigured either the terraform template or the helm chart ?
t
get the logs 
-p
?
r
Here are the logs I get with 
kubectl logs flyte-binary-b958c88b6-mh2nh -n flyte -c flyte
logs
The thing I'm investigating right now is that I'm having doubts about the values I've put there for OIDC in the helm configuration: https://github.com/flyteorg/flyte/blob/master/charts/flyte-binary/eks-production.yaml#L23-L29
t
-p
?
well 
get pod
first… want to see if there are restarts.
and if there are restarts, then 
-p
to see the previous logs
r
haaaaaa I understand now why the previous logs are interesting to watch. Smart.
logs
logs
(I forgot to say that there are restarts indeed: since the liveness & readiness probes fail to connect the pod is restarted)
a
@rough-sugar-4818 is that the `baseUrl`that you're using? bc that won't work in your environment
r
Nope. I've put the OIDC Connect provider url shown on the cluster page in EKS console.
t
we need to be better about errors
that error is a terminating error…
can you make it so that that error goes away
unf. in the single binary we don’t capture the error and restart. then it would be obvious
the thread just dies
😞
r
Haaaaa so I've misconfigured the thing: OIDC is not found
It works !
Since I'm very new to kube, I'm not very used to digg into logs. Thanks for helping finding the root cause. (and I guess it's time for me to go to sleep now).
t
sure let us know…
we definitely recommend doing auth last
as it’s the trickiest bit to get right
r
What was strange was seeing the pod as 
Running
, I thought that if the probes couldn't reach it was due to a networking issue. But indeed the pod wasn't really running.
@thankful-minister-83577 Yeah thanks for the tip. I'm learning on the way. 😅
t
yeah we need to add a panic to serve probably and recover in the go routine https://github.com/flyteorg/flyte/blob/e57cac0990fe5ec321e590fc49147014827e6dfd/cmd/single/start.go#L95
Open in Slack
PreviousNext
Powered by

Flyte

Flyte enables production-grade orchestration for machine learning workflows and data processing created to accelerate local workflows to production.

Powered by