https://flyte.org logo
Join Slack
Powered by
Flyte seems to not report stack trace of nested ex...
# flyte-support
f
Flyte seems to not report stack trace of nested exceptions for recoverable user exceptions. We've been observing this in our training jobs, and I reproduced it minimally via the following:
Copy code
from sys import exc_info
from flytekit.exceptions.user import FlyteRecoverableException
from flytekit.exceptions.scopes import FlyteScopedUserException
try:
	try:
		raise RuntimeError("a")
	except Exception as e:
		raise FlyteRecoverableException("b") from e
except Exception:
	ex = FlyteScopedUserException(*exc_info()) # this is how Flyte wraps up user exceptions

print(ex.verbose_message)
This produces:
Copy code
Traceback (most recent call last):

   File "<stdin>", line 5, in <module>

Message:

  FlyteRecoverableException: USER:Recoverable: error=b, cause=a

User error.
See how there is only a single traceback. Compare to:
Copy code
from flytekit.exceptions.user import FlyteRecoverableException
import traceback
try:
	try:
		raise RuntimeError("a")
	except Exception as e:
		raise FlyteRecoverableException("b") from e
except Exception:
	traceback.print_exc()
which correctly produces 2 stack traces:
Copy code
Traceback (most recent call last):
 File "<stdin>", line 3, in <module>
RuntimeError: a

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
 File "<stdin>", line 5, in <module>
flytekit.exceptions.user.FlyteRecoverableException: USER:Recoverable: error=b, cause=a
Seems like a problem with 
FlyteScopedUserException
.
t
which version are you on? 1.13.7?
f
1.13.1
Let me try 1.13.7
t
we’ve been toying around with how error messages are displayed in the most recent releases.
it may be worth it to try to see how the original error example looks rather than this repro
f
It still behaves the same.
Copy code
def get_traceback_str(e: Exception) -> str:
    if isinstance(e, FlyteUserRuntimeException):
        # If the exception is a user exception, we want to capture the traceback of the exception that was raised by the
        # user code, not the Flyte internals.
        tb = e.__cause__.__traceback__ if e.__cause__ else e.__traceback__
    else:
        tb = e.__traceback__
    lines = traceback.format_tb(tb)
    lines = [line.rstrip() for line in lines]
    tb_str = "\n    ".join(lines)
    format_str = "Traceback (most recent call last):\n" "\n    {traceback}\n" "\n" "Message:\n" "\n" "    {message}"

    value = e.value if isinstance(e, FlyteUserRuntimeException) else e
    return format_str.format(traceback=tb_str, message=f"{type(value).__name__}: {value}")
This is the new method I see in the latest version. With the new version, I think this method can be improved to print the nested cause.
Do you mind if I create a PR to improve this?
t
🙏 please do
f
Hi @thankful-minister-83577. Is someone working in this area around to take a look?
12 Views
Open in Slack
PreviousNext
Powered by