Join Slack

Hi there, does anyone have any tips on debugging why workflows are stuck in queue indefinitely?

# flyte-support

straight-businessperson-54649

02/08/2023, 9:26 PM

Hi there, does anyone have any tips on debugging why workflows are stuck in queue indefinitely?

limited-raincoat-94253

02/08/2023, 9:47 PM

not an expert but have you tried

flytectl get execution -p flytesnacks -d development <execution_id> --details

freezing-airport-6809

02/08/2023, 9:50 PM

in queue indefinitely? can you talk more?

quick-salesclerk-18019

02/08/2023, 9:52 PM

I would check kubernetes resources used vs what the flyte project is allowed to consume.

👍 1

limited-raincoat-94253

02/08/2023, 9:52 PM

@freezing-airport-6809 FYI previously I had a task queued indefinitely was due to resource exceeded quota

👍 1

limited-raincoat-94253

02/08/2023, 9:52 PM

Copy code

└── start-node - SUCCEEDED - 2022-10-15 03:11:44.931890319 +0000 UTC - 2022-10-15 03:11:44.931890319 +0000 UTC
└── n0 - RUNNING - 2022-10-15 03:11:44.950847963 +0000 UTC - 2022-10-15 03:11:45.080042711 +0000 UTC
    └── Attempt :0
        └── Task - WAITING_FOR_RESOURCES - 2022-10-15 03:11:45.075151304 +0000 UTC - 2022-10-15 03:11:45.075151304 +0000 UTC
        └── Task Type - python-task
        └── Reason - Exceeded resourcequota: [BackOffError] The operation was attempted but failed, caused by: pods "feaeaf1c97c574fd0aa8-n0-0" is forbidden: exceeded quota: project-quota, requested: limits.memory=500Mi, used: limits.memory=3000Mi, limited: limits.memory=3000Mi
        └── Metadata
        │   ├── Generated Name : feaeaf1c97c574fd0aa8-n0-0
        │   ├── Plugin Identifier : container
        │   ├── External Resources
        │   ├── Resource Pool Info
        └── Logs :

freezing-airport-6809

02/08/2023, 9:53 PM

i do agree that is the usual case

freezing-airport-6809

02/08/2023, 9:53 PM

for some reason our default helm chart had a very restrictive quota

freezing-airport-6809

02/08/2023, 9:53 PM

we have now removed it, but that is only recent

freezing-airport-6809

02/08/2023, 9:54 PM

@straight-businessperson-54649 you might want to do -

kubectl describe resourcequota

straight-businessperson-54649

02/16/2023, 2:15 PM

Thank you all for your help 🙂 and sorry for the lack of details. We dug into it and it turned out to be that we needed a larger node with more memory than the ones we had available to run the task.

170 Views

Open in Slack

Previous Next

Flyte

Flyte enables production-grade orchestration for machine learning workflows and data processing created to accelerate local workflows to production.