https://flyte.org logo
#flyte-deployment
Title
# flyte-deployment
k

karthikraj

12/05/2022, 4:54 AM
Hi Team, Getting this issue in the FlyteConsole. It was working fine and I have just redeployed the helm. What could be the reason?
s

Samhita Alla

12/05/2022, 6:22 AM
Are all your services up and running — flytepropeller, admin, etc.?
k

karthikraj

12/05/2022, 6:33 AM
yeah, All the services are up and running. Dont see any unusual error/debug log in the services.
s

Samhita Alla

12/05/2022, 7:06 AM
@Yee, would you be able to help here?
y

Yee

12/05/2022, 4:59 PM
hey @karthikraj did you change the version of the helm chart? or just re-install it?
which helm chart are you using?
could you do a
kubectl -n flyte get pod
and look for a pod that’s crashing? mind copy/pasting any error logs you find there?
k

karthikraj

12/05/2022, 5:39 PM
@Yee Weirdly none of the pods are Crashing. All are in running state.
Copy code
terraform]$ kubectl -n flyte get pod
NAME                                 READY   STATUS    RESTARTS   AGE
datacatalog-b747d6c5b-hjqfp          1/1     Running   0          11h
datacatalog-b747d6c5b-vfhgx          1/1     Running   0          11h
flyte-pod-webhook-864db964f8-59nrw   1/1     Running   0          11h
flyteadmin-5bc6894758-25nqr          1/1     Running   0          11h
flyteadmin-5bc6894758-zlvbt          1/1     Running   0          11h
flyteconsole-8d595f6d6-9x4d9         1/1     Running   0          11h
flyteconsole-8d595f6d6-z9zwr         1/1     Running   0          11h
flytepropeller-8f9b64448-88mj5       1/1     Running   0          11h
flytepropeller-8f9b64448-q2kzp       1/1     Running   0          11h
flytescheduler-66c56fb6cf-v7mkw      1/1     Running   0          11h
syncresources-687f7df97b-2mscs       1/1     Running   0          11h
Copy code
terraform]$ helm list -n flyte
NAME            NAMESPACE       REVISION        UPDATED                                 STATUS          CHART                   APP VERSION
dev-max-flyte   flyte           1               2022-12-05 06:27:21.544709499 +0000 UTC deployed        flyte-core-v0.1.10
y

Yee

12/05/2022, 5:41 PM
can you open up the chrome debugger window and see which call or calls is 503'ing?
k

karthikraj

12/05/2022, 9:16 PM
@Yee Here is the result
y

Yee

12/05/2022, 9:19 PM
can you go to a new tab and just navigate to https://flyte.dev.dap.wbd.com/api/v1/projects and confirm that you’re also getting a 503
k

karthikraj

12/05/2022, 9:26 PM
@Yee yeah, I see
What could be the possible reason for this you want to me check?
y

Yee

12/05/2022, 10:39 PM
hi sorry - can you
kubectl -n flyte port-forward service/flyteadmin
? i think you might need ports as well. maybe
8088:8088
but don’t quite remember. And then try to hit
localhost:<port>/api/v1/projects
i think the issue is happening at the ingress layer, if the flyte service pods are running, and you’re getting a 503, it’s usually an ingress issue.
k

karthikraj

12/06/2022, 5:13 AM
Hi @Yee,I have tried both https port and grpc port. Both are throwing error. Clueless why this happened all of sudden. 😞
Copy code
[terraform]$ kubectl -n flyte port-forward service/flyteadmin 8089:8089
error: Service flyteadmin does not have a service port 8089
Copy code
[terraform]$ kubectl -n flyte port-forward service/flyteadmin 8088:8088
error: Service flyteadmin does not have a service port 8088
Copy code
[ terraform]$ kubectl get svc -n flyte
NAME                TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)                                    AGE
datacatalog         NodePort    172.20.247.89   <none>        8089:31462/TCP,88:30445/TCP,89:31941/TCP   42m
flyte-pod-webhook   ClusterIP   172.20.212.71   <none>        443/TCP                                    42m
flyteadmin          ClusterIP   172.20.136.44   <none>        80/TCP,81/TCP,87/TCP,10254/TCP             42m
flyteconsole        ClusterIP   172.20.189.86   <none>        80/TCP                                     42m
y

Yee

12/06/2022, 5:20 AM
try port 8080:80 and localhost:8080
14 Views