miniature-barista-44022
04/17/2023, 10:36 PMRuntimeError
. According to the docs, I thought this would be considered non-recoverable and thus not retried? I’l put the stack trace in the thread.miniature-barista-44022
04/17/2023, 10:37 PMTraceback (most recent call last):
File "/app/pip_pypi__flytekit/flytekit/exceptions/scopes.py", line 203, in user_entry_point
return wrapped(*args, **kwargs)
File "/app/atomwise/...", line 266, in run_block
block.validate_output(cmpds, result)
File "/app/atomwise/..., line 119, in validate_output
raise RuntimeError(
Message:
<blah>
User error.
glamorous-carpet-83516
04/17/2023, 11:52 PMuser
04/17/2023, 11:52 PMminiature-barista-44022
04/18/2023, 12:00 AMhallowed-mouse-14616
04/18/2023, 1:50 PMUSER
or SYSTEM
errors where SYSTEM
are retryable. Do we know how the RuntimeError
is handled here? My intuition is that Flyte is more conservative about labeling a failure as non-retryable and wouldn't map all python RuntimeError
instances as USER
errors and therefore non-retryable.glamorous-carpet-83516
04/18/2023, 9:38 PMRuntimeError
, flytekit will convert it to FlyteScopedUserException
(User error) here. it’s retryable right now. cc @thankful-minister-83577hallowed-mouse-14616
04/18/2023, 10:45 PMUSER
or SYSTEM
identifier only has to do with the number of times a failure is retried (which can be configured to mean one never retries). In this scenario it seems that flytekit will designate the specified error as retryable unless the the user specifically throws a non-recoverable exception. I imagine there have been many discussions about what the preferred behavior is here, regardless from the flytekit side, if you just raise a non-recoverable error it should have the intended behavior.
TL;DR @miniature-barista-44022 in the issue it seems that you're manually throwing a runtime exception. Is there any reason you could not just manually designate a Flyte non-recoverable exception instead?glamorous-carpet-83516
04/18/2023, 11:04 PMruntimeError
as retryable. To workaround it, you could catch that error, and throw a Flyte non-recoverable exception. Just discussed with Yee, all the non-flyte errors raised from user code should be non-recoverable. we’ll fix it in flytekit side.miniature-barista-44022
04/19/2023, 12:19 AMFlyteUserException
and got the same behavior. Is there another exception type you suggest?thankful-minister-83577
thankful-minister-83577
@task(retries=10)
def say_hello_error(a: int) -> str:
if a == 1:
print("Raising Error")
raise RuntimeError("my runtime error")
return "hi"
this task runs once for me.miniature-barista-44022
04/19/2023, 4:08 PM