https://flyte.org
Join Slack
Hi Team, Getting this issue in the FlyteConsole. It was working fine and I have just redeployed the...
# flyte-deployment
d
Hi Team, Getting this issue in the FlyteConsole. It was working fine and I have just redeployed the helm. What could be the reason?
t
Are all your services up and running — flytepropeller, admin, etc.?
d
yeah, All the services are up and running. Dont see any unusual error/debug log in the services.
t
@thankful-minister-83577, would you be able to help here?
t
hey @delightful-lion-57360 did you change the version of the helm chart? or just re-install it?
which helm chart are you using?
could you do a 
kubectl -n flyte get pod
and look for a pod that’s crashing? mind copy/pasting any error logs you find there?
d
@thankful-minister-83577 Weirdly none of the pods are Crashing. All are in running state.
Copy code
terraform]$ kubectl -n flyte get pod
NAME                                 READY   STATUS    RESTARTS   AGE
datacatalog-b747d6c5b-hjqfp          1/1     Running   0          11h
datacatalog-b747d6c5b-vfhgx          1/1     Running   0          11h
flyte-pod-webhook-864db964f8-59nrw   1/1     Running   0          11h
flyteadmin-5bc6894758-25nqr          1/1     Running   0          11h
flyteadmin-5bc6894758-zlvbt          1/1     Running   0          11h
flyteconsole-8d595f6d6-9x4d9         1/1     Running   0          11h
flyteconsole-8d595f6d6-z9zwr         1/1     Running   0          11h
flytepropeller-8f9b64448-88mj5       1/1     Running   0          11h
flytepropeller-8f9b64448-q2kzp       1/1     Running   0          11h
flytescheduler-66c56fb6cf-v7mkw      1/1     Running   0          11h
syncresources-687f7df97b-2mscs       1/1     Running   0          11h
Copy code
terraform]$ helm list -n flyte
NAME            NAMESPACE       REVISION        UPDATED                                 STATUS          CHART                   APP VERSION
dev-max-flyte   flyte           1               2022-12-05 06:27:21.544709499 +0000 UTC deployed        flyte-core-v0.1.10
t
can you open up the chrome debugger window and see which call or calls is 503'ing?
d
@thankful-minister-83577 Here is the result
t
can you go to a new tab and just navigate to https://flyte.dev.dap.wbd.com/api/v1/projects and confirm that you’re also getting a 503
d
@thankful-minister-83577 yeah, I see
What could be the possible reason for this you want to me check?
t
hi sorry - can you 
kubectl -n flyte port-forward service/flyteadmin
? i think you might need ports as well. maybe 
8088:8088
but don’t quite remember. And then try to hit 
localhost:<port>/api/v1/projects
i think the issue is happening at the ingress layer, if the flyte service pods are running, and you’re getting a 503, it’s usually an ingress issue.
d
Hi @thankful-minister-83577,I have tried both https port and grpc port. Both are throwing error. Clueless why this happened all of sudden. 😞
Copy code
[terraform]$ kubectl -n flyte port-forward service/flyteadmin 8089:8089
error: Service flyteadmin does not have a service port 8089
Copy code
[terraform]$ kubectl -n flyte port-forward service/flyteadmin 8088:8088
error: Service flyteadmin does not have a service port 8088
Copy code
[ terraform]$ kubectl get svc -n flyte
NAME                TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)                                    AGE
datacatalog         NodePort    172.20.247.89   <none>        8089:31462/TCP,88:30445/TCP,89:31941/TCP   42m
flyte-pod-webhook   ClusterIP   172.20.212.71   <none>        443/TCP                                    42m
flyteadmin          ClusterIP   172.20.136.44   <none>        80/TCP,81/TCP,87/TCP,10254/TCP             42m
flyteconsole        ClusterIP   172.20.189.86   <none>        80/TCP                                     42m
t
try port 8080:80 and localhost:8080
184 Views
Open in Slack
PreviousNext
Powered by

Flyte

Flyte enables production-grade orchestration for machine learning workflows and data processing created to accelerate local workflows to production.

Powered by