#2933 [Core feature] Fast fail tasks if resource quota request exceeds k8s quota limits Issue created by hamersaw Motivation: Why do you think this is important? Currently, FlytePropeller detects resource quota issues using k8s error messages with code 403 and "exceeds quota" in the message. However, this approach is unable to differentiate between scenarios where: (1) the requested quotas + the currently used quotas are more than k8s resource quota limits (2) the requested quotas exceed k8s resource quota limits When detecting resource quota issues, FlytePropeller periodically attempts to schedule the task. In the former case, this will succeed once other tasks complete and resources become available. However, in the later case, the resources will never be available and FlytePropeller leaves the task in "waiting" state until the node is terminated based on the

node-active-deadline

(which defaults to 48h). Goal: What should the final outcome look like, ideally? Tasks which request resources larger than the available k8s resource quota limits should immediately fail rather than periodically attempt to schedule. Describe alternatives you've considered FlyteAdmin exposes a number of configuration options (ex.

task-resource-defaults

) which stop users from registering tasks with resource quotas larger than the configured values. However, these are typically register-time checks and do not apply if FlyteAdmin (1) is not configured to manage cluster resources or (2) k8s resource quotas (or admin configuration) is changed after task registration. Additionally, handling these issues at runtime means diverse environments (many projects / domains) can be handled independently. Propose: Link/Inline OR Additional context The FlytePropeller backoff handler code has a GetComputeResourceAndQuantityRequested function which parses the exceeded resource quota error messages to expose requests, used, and limits on resource quotas. If FlytePropeller detects an attempted task execution exceeds resource quotas (ex. here) it should then check if the task resource quotas exceed k8s resource quota limits, and if so fail immediately rather than attempt to execute when more resources are available. Are you sure this issue hasn't been raised already? ☑︎ Yes Have you read the Code of Conduct? ☑︎ Yes flyteorg/flyte