#6695 [BUG] UserError marked as SystemError doesn't trigger the on_failure handler Issue created by davidmirror-ops ### Flyte & Flytekit version Both

1.16.4

### Describe the bug Running this

import flytekit
import typing
from flytekit import Environment, map_task, task, workflow
from flytekit.types.error.error import FlyteError

env = Environment(
    resources=flytekit.Resources(cpu="500m", mem="512Mi")
)

@env.task
def create_cluster(name: str):
    print(f"Creating cluster: {name}")

@task
def delete_cluster(name: str):
    print(f"Deleting cluster {name}")

@env.task
def t1(a: int, b: str) -> str:
    # raise ValueError("Fail!")
    print(f"{a} {b}")
    return "result"
    
@env.task
def t2(x: int, y: list[str]) -> int:
    print(f"x: {x}, y: {y}")
    return x * 2

@env.task
def clean_up(name: str, err: typing.Optional[FlyteError] = None):
    print(f"Cleaning up {name}")
    if err:
        print(f"Error: {err}")


@env.task
def deliveries() -> list[str]:
    return ["a", "b", "c"]


@env.dynamic
def subfw() -> None:
    dds = deliveries()
    map_task(t2)(x=[1,2,3], y=[dds] * 3)
    map_task(t2)(x=[1,2,3], y=dds)  # <--- this is the wrong code but it shouldn't break the whole pipeline like that


@workflow(on_failure=clean_up)
def wf_hangs_error(name: str):
    c = create_cluster(name=name)
    t = t1(a=1, b="2")

    subfw()

    d = delete_cluster(name=name)
    c >> t >> d

Results in the following behavior: 1. Workflow error - EXPECTED (although not sure why it appears twice):

Workflow[pipelines:development:wf_hangs_on_failure_not_called.wf_hangs_error] failed. RuntimeExecutionError: max number of system retry attempts [11/10] exhausted. Last known status message: 0: [User] malformed dynamic workflow, caused by: Collected Errors: 1
	Error 0: Code: MismatchingTypes, Node Id: dn2, Description: The output variable 'deliveries.o0' has type [collection_type:{simple:STRING}], but it's assigned to the input variable 'wf_hangs_on_failure_not_called.map_t2_6b3bd0353da5de6e84d7982921ead2b3-arraynode.y' which has type type [collection_type:{collection_type:{simple:STRING}}].

[User] malformed dynamic workflow, caused by: Collected Errors: 1
	Error 0: Code: MismatchingTypes, Node Id: dn2, Description: The output variable 'deliveries.o0' has type [collection_type:{simple:STRING}], but it's assigned to the input variable 'wf_hangs_on_failure_not_called.map_t2_6b3bd0353da5de6e84d7982921ead2b3-arraynode.y' which has type type [collection_type:{collection_type:{simple:STRING}}].

1. subfw node fails - EXPECTED 2.

t1

node marked Aborted even though it actually succeeded! - NOT EXPECTED 3. on_failure node (clean_up) is not being executed - NOT EXPECTED The core issue here is mutlifaceted: a. Something that is clearly a UserError "bubbles up" as SytemError and, hence, it triggers the system retry logic. Depending on the configuration for

base-delay

and

max-delay

this could fail quickly or take much longer. Using Flyte defaults for these values makes it fail fast but doesn't fix the core issues. b. When propeller runs out of system retries it aborts the node, which doesn't trigger the

on_failure

node. ### Additional context a. The error is initially categorized as UserError (ref) b. It transitions the node to DynamicNodePhaseFailing(ref) c. The failure is not marked as permanent hence it kicks off the retry logic (ref) d. The workflow eventually runs out of system retries and it gets Aborted which doesn't trigger the failure node (ref) ### Expected behavior 1. UserErrors in dynamic workflows should always be marked as permanent 2. For system errors, when exceeding the maximum number of system retries, Flyte should transition the workflow to a `FAILING `state so that the

on_failure

block is called and the workflow is aborted rather than immediately failing. ### Additional context to reproduce No response ### Screenshots [Image](https://private-user-images.githubusercontent.com/82604841/504686810-69fad210-f323-4bcc-97ce-6ae7b97a9de2.png?jwt=eyJ0eXAiOiJKV1QiLCJhbGciOiJIUzI1NiJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3NjEzNTE2MzQsIm5iZiI6MTc2MTM1MTMzNCwicGF0aCI6Ii84MjYwNDg0MS81MDQ2ODY4MTAtNjlmYWQyMTAtZjMyMy00YmNjLTk3Y2UtNmFlN2I5N2E5ZGUyLnBuZz9YLUFtei1BbGdvcml0aG09QVdTNC1ITUFDLVNIQTI1NiZYLUFtei1DcmVkZW50aWFsPUFLSUFWQ09EWUxTQTUzUFFLNFpBJTJGMjAyNTEwMjUlMkZ1cy1lYXN0LTElMkZzMyUyRmF3czRfcmVxdWVzdCZYLUFtei1EYXRlPTIwMjUxMDI1VDAwMTUzNFomWC1BbXotRXhwaXJlcz0zMDAmWC1BbXotU2lnbmF0dXJlPTczYTE1OGYwOWZiMDRiNWI1YWE4ZWRjNjcxYzU2NjQ2N2I5OWIxZjYwN2Y5MzY3NmE3MTEwNWQ0MmE1YjY0YWEmWC1BbXotU2lnbmVkSGVhZGVycz1ob3N0In0.ijAGjoHHTV-jzeNoHiYZFuW8wfU21sJzn7ffJtB8xzk) ### Are you sure this issue hasn't been raised already? • Yes ### Have you read the Code of Conduct? • Yes flyteorg/flyte