<#6531 [BUG] Flyte Propeller treats a BadRequest c...
# flyte-github
a
#6531 [BUG] Flyte Propeller treats a BadRequest code from K8s when launching a resource as a retryable failure Issue created by swarupsrini Flyte & Flytekit version 1.15 Describe the bug Currently, the plugin manager code has a line commented: flyte/flytepropeller/pkg/controller/nodes/task/k8s/plugin_manager.go Lines 257 to 259 in</flyteorg/flyte/commit/61b5b4aec1b03c9c9604de8e78c4166ee415ccdc|61b5b4a> | } else if k8serrors.IsBadRequest(err) \|| k8serrors.IsInvalid(err) { | | -------------------------------------------------------------------------------------------------------- | | logger.Errorf(ctx, "Badly formatted resource for plugin [%s], err %s", e.id, err) | | // return pluginsCore.DoTransition(pluginsCore.PhaseInfoFailure("BadTaskFormat", err.Error(), nil)), nil | which results in any BadRequest status codes from K8s to be treated as a retryable failure, since that is the base case: flyte/flytepropeller/pkg/controller/nodes/task/k8s/plugin_manager.go Lines 264 to 266 in</flyteorg/flyte/commit/61b5b4aec1b03c9c9604de8e78c4166ee415ccdc|61b5b4a> | reason := k8serrors.ReasonForError(err) | | ----------------------------------------------------------------------------------------------------------------- | | logger.Errorf(ctx, "Failed to launch job, system error. err: %v", err) | | return pluginsCore.UnknownTransition, errors.Wrapf(stdErrors.ErrorCode(reason), err, "failed to create resource") | This is particularly problematic for our use-case where we have a validating admission webhook to reject any invalid pods. By default webhook rejections are BadRequests, which do make sense for validations, but this should not be treated as retryable since there is no chance for success, and we don't want the workflow to end up in an aborted state. Our mitigation has been to make the validating webhook send an EntityTooLarge code, which seems to be the only case treated as non retryable: flyte/flytepropeller/pkg/controller/nodes/task/k8s/plugin_manager.go Lines 260 to 262 in</flyteorg/flyte/commit/61b5b4aec1b03c9c9604de8e78c4166ee415ccdc|61b5b4a> | } else if k8serrors.IsRequestEntityTooLargeError(err) { | | ------------------------------------------------------------------------------------------------------ | | logger.Errorf(ctx, "Badly formatted resource for plugin [%s], err %s", e.id, err) | | return pluginsCore.DoTransition(pluginsCore.PhaseInfoFailure("EntityTooLarge", err.Error(), nil)), nil | I think just uncommenting L259 would fix this bug, but since the line exists and has been explicitly commented out, I'm opening this ticket to get some insight about the motivation, and if it's safe to uncomment. Expected behavior Ideally, BadRequest codes should be treated as non retryable. Additional context to reproduce No response Screenshots No response Are you sure this issue hasn't been raised already? • Yes Have you read the Code of Conduct? • Yes flyteorg/flyte