#6740 Fix: Immediately fail if plugin.BuildResource fails instead of retrying until system retry budget is exhausted Pull request opened by fg91 ## Why are the changes needed? One of our engineers launched a Ray task with an invalid custom pod spec and had to wait ~1.5 hours before he was notified of the mistake because the error, while fatal, was classified as a transient system error that can be retried with exponential backoff. Details: In this case, a Ray task with a custom pod spec was registered that specified

"4GB"

as a resource request instead of

"4G"

or

"4Gi"

. In the Ray plugin, here, the

k8sPod.GetPodSpec()

could not be unmarshaled into a

v1.PodSpec

because the

"GB"

violated

Err: [quantities must match the regular expression '^([+-]?[0-9.]+)([eEinumkKMGTP]*[-+]?[0-9]*)$']

. As a consequence, `plugin.BuildResource(...)` in the plugin manager failed with an error. However, we currently treat errors in

plugin.BuildResource()

with: if err != nil { return pluginsCore.UnknownTransition, err } The consequence is that this will be retried until the system retry budget is exhausted (including time wasted on backoff): RuntimeExecutionError: max number of system retry attempts [51/50] exhausted. Last known status message: failed at Node[dn0]. RuntimeExecutionError: failed during plugin execution, caused by: failed to execute handle for plugin [ray]: [BadTaskSpecification] Unable to unmarshal pod ... Err: [quantities must match the regular expression

BuildResource

constructs the Kubernetes manifest that the plugin manager will then apply to the cluster but

BuildResource

itself is not calling any external systems. Therefore, I would argue that this is not a transient error that we would expect to eventually work. I would argue that when the K8s manifest cannot be constructed by

BuildResource

we want to fail immediately so that the user has a chance to fix the task definition instead of being mislead for ~1.5h that the task will eventually run. ## What changes were proposed in this pull request? If

e.plugin.BuildResource(ctx, k8sTaskCtx)

fails in the plugin manager, return an error that is not retryable. ## How was this patch tested? Ran a flytepropeller locally with a debugger attached. Made sure that with this change, in the scenario above, the user is directly shown the error. https://bito.ai#summarystart| ### Summary by Bito • This pull request fixes a critical issue where errors in the resource construction process were treated as transient, leading to unnecessary retries and delays. • The changes ensure that if the

plugin.BuildResource

function fails, the system will immediately return an error instead of retrying. • This fix allows users to correct their task definitions without prolonged waiting. • Overall, this update addresses the resource construction process in the

plugin.BuildResource

function and improves user feedback on fatal errors. flyteorg/flyte