Channels
#ask-the-community
Title
# ask-the-community
d
Derek Yu
02/08/2023, 9:26 PMHi there, does anyone have any tips on debugging why workflows are stuck in queue indefinitely?
y
Yubo Wang
02/08/2023, 9:47 PMnot an expert but have you tried
flytectl get execution -p flytesnacks -d development <execution_id> --detailsk
Ketan (kumare3)
02/08/2023, 9:50 PM
in queue indefinitely? can you talk more?
b
Björn
02/08/2023, 9:52 PMI would check kubernetes resources used vs what the flyte project is allowed to consume.
y
Yubo Wang
02/08/2023, 9:52 PM@Ketan (kumare3) FYI previously I had a task queued indefinitely was due to resource exceeded quota
Copy code
└── start-node - SUCCEEDED - 2022-10-15 03:11:44.931890319 +0000 UTC - 2022-10-15 03:11:44.931890319 +0000 UTC
└── n0 - RUNNING - 2022-10-15 03:11:44.950847963 +0000 UTC - 2022-10-15 03:11:45.080042711 +0000 UTC
└── Attempt :0
└── Task - WAITING_FOR_RESOURCES - 2022-10-15 03:11:45.075151304 +0000 UTC - 2022-10-15 03:11:45.075151304 +0000 UTC
└── Task Type - python-task
└── Reason - Exceeded resourcequota: [BackOffError] The operation was attempted but failed, caused by: pods "feaeaf1c97c574fd0aa8-n0-0" is forbidden: exceeded quota: project-quota, requested: limits.memory=500Mi, used: limits.memory=3000Mi, limited: limits.memory=3000Mi
└── Metadata
│ ├── Generated Name : feaeaf1c97c574fd0aa8-n0-0
│ ├── Plugin Identifier : container
│ ├── External Resources
│ ├── Resource Pool Info
└── Logs :k
Ketan (kumare3)
02/08/2023, 9:53 PM
i do agree that is the usual case
for some reason our default helm chart had a very restrictive quota
we have now removed it, but that is only recent
@Derek Yu you might want to do -
kubectl describe resourcequotad
Derek Yu
02/16/2023, 2:15 PMThank you all for your help 🙂 and sorry for the lack of details. We dug into it and it turned out to be that we needed a larger node with more memory than the ones we had available to run the task.