Are retries for user errors supported for raw container tasks? The behavior we see seems to suggest no and I can understand since its not using

pyflyte-execute

but wanted to confirm since I don't think this is documented

it you write an error file, that will be consider as user error.

Thank you!

cc @full-toddler-99513 do you want to help?

Sorry, I have other tasks to handle at the moment, so I’m unavailable for now.

@glamorous-carpet-83516: I am little confused with the suggestion. Are you suggesting to write the error to

output_data_dir

path? There could be cases like exception, OOM where we can not write to output_data_dir. How to handle cases like this?

do you want to retry on OOM?

Yes, OOM or some random Exception or timeouts.

what’s the value of

MaxNodeRetriesOnSystemFailures

in your settings?https://github.com/flyteorg/flyte/blob/92f8abb2f34b648a3430dc6e2262b4fdf625ad39/flytepropeller/pkg/controller/config/config.go#L102

In our case we are sending the compute to Armada so we might have to get a little more specific about which failures are retry-able and which ones are not. However I'm not sure retrying on an OOM makes sense. Example code from our Armada plugin

func mapExecutionStateToPhaseInfo(state ExecutionState, cfg *Config, clock clock.Clock) core.PhaseInfo {
	t := clock.Now()
	taskInfo := constructTaskInfo(state, cfg, clock)
	reason := fmt.Sprintf("Armada job in state %s", state.JobState)

	switch state.JobState {
	case api.JobState_UNKNOWN, api.JobState_QUEUED, api.JobState_SUBMITTED:
		return core.PhaseInfoQueuedWithTaskInfo(t, core.DefaultPhaseVersion, reason, taskInfo)

	case api.JobState_PENDING, api.JobState_LEASED:
		return core.PhaseInfoInitializing(t, core.DefaultPhaseVersion, reason, taskInfo)

	case api.JobState_RUNNING:
		return core.PhaseInfoRunningWithReason(core.DefaultPhaseVersion, taskInfo, reason)

	case api.JobState_SUCCEEDED:
		return core.PhaseInfoSuccessWithReason(taskInfo, reason)

	case api.JobState_PREEMPTED, api.JobState_CANCELLED:
		// This likely means Armada or someone cancelled these jobs. Marking them as retryable.
		return core.PhaseInfoRetryableFailure(errors.DownstreamSystemError, reason, taskInfo)

	case api.JobState_FAILED, api.JobState_REJECTED:
		return core.PhaseInfoFailure(errors.TaskFailedUnknownError, "Job failed", taskInfo)
	default:
		reason = fmt.Sprintf("unhandled job state %s", state.JobState)
		return core.PhaseInfoFailure(errors.RuntimeFailure, reason, taskInfo)
	}
}

@glamorous-carpet-83516: We are observing that if a

ContainerTask

returns a

non-zero

error code, flyte is not honoring

TaskMetadata.retries

set in

ContainerTask

.

@worried-iron-1001 is that with the outputs enabled like we discussed internally would need to be set?

Yes. I added a dummy output, but now flyte expects all container tasks images to emit the output else it throws

Outputs not generated by task execution

.

We are observing that if a

ContainerTask

returns a

non-zero

error code, flyte is not honoring

TaskMetadata.retries

set in

ContainerTask

that’s because non-zero error becomes system error for some reasons.

TaskMetadata.retries

is used for the user error