https://flyte.org logo
#ask-the-community
Title
# ask-the-community
l

Len Strnad

06/09/2023, 1:48 PM
Hey, all 👋 . I have encountered an issue where a single execution apparently has invalid utf-8 data and the console fails to 1.) show the workflow details on the ui and 2.) fails to summarize all executions in the workflow entirely. Two screenshots attached in 🧵 Is there a way to simply delete that workflow execution and all its metadata with flyteclt in order to hopefully restore the general workflow page?
Workflow page fails to summarize all executions because of one execution
I can see other executions, but only after going back from a previous (valid) execution.
k

Ketan (kumare3)

06/09/2023, 2:39 PM
This is a bug we should handle
I actually think it is the http get that’s failing
Can you share what is the bad data
We should catch it at creation time
l

Len Strnad

06/09/2023, 2:41 PM
Any suggestions of how to discover this bad data? I am not the one who deployed Flyte and am just getting familiar with all the components (fyi), but I do have permissions.
k

Ketan (kumare3)

06/09/2023, 2:43 PM
Try flytectl
Seems like versions
l

Len Strnad

06/09/2023, 2:48 PM
flytectl get workflow
with version?
I am looking at
Copy code
flytectl get execution --admin.endpoint $FLYTE_HOST -p project -d development a4gbfch8hmrg67gv2pql --output yaml
and the only culprit I see might be the error.message field.
Is it valid to take that error message and verbatim check if that is utf-8?
k

Ketan (kumare3)

06/09/2023, 3:02 PM
Ohh man this is bad, we should be fixing this - cc @Katrina Rogan / @Frank Flitton can either of you help
l

Len Strnad

06/09/2023, 3:08 PM
The error message seems to encode to utf-8 when I copy and paste and try to encode in python 🤷. cc’ing @Bernhard Stadlbauer since he might be able to dig deeper when he is back Monday.
The task that failed due to retried exhausted and OOMKilled fwiw
f

Frank Flitton

06/09/2023, 3:57 PM
The 500 error code implies it may be in the api layer to me. Can you share a screenshot of what the network tab in the chrome inspector looks like after a refresh. That will help us narrow things down.
l

Len Strnad

06/09/2023, 4:00 PM
Of the console? This is an error I see:
It looks like we get a 500 from this request URL:
Copy code
<https://flyte.labs.our><http://_company.com/api/v1/node_executions/project/development/a4gbfch8hmrg67gv2pql?filters=&limit=10000&sort_by.direction=ASCENDING&sort_by.key=created_at|_company.com/api/v1/node_executions/project/development/a4gbfch8hmrg67gv2pql?filters=&limit=10000&sort_by.direction=ASCENDING&sort_by.key=created_at>
Hope that helps. fwiw a colleague checked the table for the meta data and we couldn’t find any invalid utf-8 data.
b

Bernhard Stadlbauer

06/09/2023, 4:28 PM
Did a quick check and couldn’t find anything more than @Len Strnad found.
Copy code
flytectl get execution --admin.endpoint $FLYTE_HOST -p project -d development a4gbfch8hmrg67gv2pql --details
also fails. Had a look at the
node_executions
flyteadmin table for that particular execution, but most of the columns are of type
bytea
. Happy to run a query against the table in case it helps.
l

Len Strnad

06/09/2023, 6:01 PM
Ok, we archived the specific execution and all is well. Not ideal, but a decent workaround for now.
k

Katrina Rogan

06/09/2023, 6:43 PM
the latest flyteadmin actually has a fix for the utf-8 encoding, which version are you on?
l

Len Strnad

06/09/2023, 7:00 PM
helm chart v.1.6.1
k

Katrina Rogan

06/09/2023, 9:07 PM
ah that might be it! could you update and let us know if you still run into this?
l

Len Strnad

06/09/2023, 9:10 PM
Will do!
Thanks for all the help on this 🙏
b

Bernhard Stadlbauer

06/12/2023, 9:22 AM
@Katrina Rogan Updating to the latest flyteadmin has fixed things for us, thank you!