What's the best way to monitor resource quota usage?

It seems likely that the best way to monitor resource quota usage in Flyte is by leveraging Kubernetes resource quotas at the namespace level, which are configured per project and domain. These quotas (CPU, memory) are set via the Flyte cluster resource manager and can be viewed and managed using Kubernetes tools like

kubectl

, as well as through Flyte's UI, which displays resource usage metrics for workflows and tasks, including memory, CPU, and GPU usage. However, Flyte's built-in dashboards (e.g., Grafana) may require additional configuration to accurately visualize quota usage, and some metrics may depend on your Prometheus setup or may need to be recreated using cluster-level metrics if not available out of the box. According to recent issues, the Flyte UI and logs can also indicate when workflows are delayed or blocked due to quota limits, and tasks may enter a "WaitingForResources" state when quotas are exceeded, providing further visibility into quota-related issues (d5b246c8, b5ecd45d, dad61526, aa7d1f86, a632937c). Would you like more detail on how to set up dashboards, interpret Flyte UI metrics, or query Kubernetes directly for quota usage? Sources: - dad61526 - d5b246c8 - b5ecd45d - aa7d1f86 - a632937c 👍 0 button 👎 0 button 💡 Hint: Mention @runllm in the thread for followups.

@ancient-wolf-19325 I'm more interested in project level resource quotas. In particular, how to tell if we've got tasks that are spending too much time WAITING_FOR_RESOURCES

To monitor if tasks are spending excessive time in the WAITING_FOR_RESOURCES state due to project-level resource quotas, you should look for tasks in this phase in the Flyte UI or via logs. The Flyte UI is being improved to show this state explicitly, and tasks will enter WAITING_FOR_RESOURCES when quota is exceeded (PR #5195). Additionally, FlytePropeller logs will indicate when tasks are waiting for resources, and the UI's task status pane should display this information (Issue #3357). For more systematic monitoring, you may need to aggregate task execution data (e.g., via Prometheus or Flyte's event logs) to track how long tasks remain in WAITING_FOR_RESOURCES. There is ongoing work to improve observability, including breaking down execution timelines by phases such as resource waiting, which will help identify bottlenecks (Issue #3357, PR #5195). Would you like more detail on how to extract or visualize this data programmatically? Sources: - PR #5195 - Issue #3357 👍 0 button 👎 0 button 💡 Hint: Mention @runllm in the thread for followups.