https://flyte.org
Join Slack
Hey, all :wave: . I have encountered an issue where a single execution apparently has invalid utf-8...
# flyte-support
b
Hey, all 👋 . I have encountered an issue where a single execution apparently has invalid utf-8 data and the console fails to 1.) show the workflow details on the ui and 2.) fails to summarize all executions in the workflow entirely. Two screenshots attached in 🧵 Is there a way to simply delete that workflow execution and all its metadata with flyteclt in order to hopefully restore the general workflow page?
Workflow page fails to summarize all executions because of one execution
I can see other executions, but only after going back from a previous (valid) execution.
f
This is a bug we should handle
I actually think it is the http get that’s failing
Can you share what is the bad data
We should catch it at creation time
b
Any suggestions of how to discover this bad data? I am not the one who deployed Flyte and am just getting familiar with all the components (fyi), but I do have permissions.
f
Try flytectl
Seems like versions
b
flytectl get workflow
with version?
I am looking at
Copy code
flytectl get execution --admin.endpoint $FLYTE_HOST -p project -d development a4gbfch8hmrg67gv2pql --output yaml
and the only culprit I see might be the error.message field.
Is it valid to take that error message and verbatim check if that is utf-8?
f
Ohh man this is bad, we should be fixing this - cc @bitter-finland-83254 / @some-raincoat-58190 can either of you help
b
The error message seems to encode to utf-8 when I copy and paste and try to encode in python 🤷. cc’ing @agreeable-kitchen-44189 since he might be able to dig deeper when he is back Monday.
The task that failed due to retried exhausted and OOMKilled fwiw
s
The 500 error code implies it may be in the api layer to me. Can you share a screenshot of what the network tab in the chrome inspector looks like after a refresh. That will help us narrow things down.
b
Of the console? This is an error I see:
It looks like we get a 500 from this request URL:
Copy code
<https://flyte.labs.our><http://_company.com/api/v1/node_executions/project/development/a4gbfch8hmrg67gv2pql?filters=&limit=10000&sort_by.direction=ASCENDING&sort_by.key=created_at|_company.com/api/v1/node_executions/project/development/a4gbfch8hmrg67gv2pql?filters=&limit=10000&sort_by.direction=ASCENDING&sort_by.key=created_at>
Hope that helps. fwiw a colleague checked the table for the meta data and we couldn’t find any invalid utf-8 data.
👍 1
a
Did a quick check and couldn’t find anything more than @bored-beard-89967 found.
Copy code
flytectl get execution --admin.endpoint $FLYTE_HOST -p project -d development a4gbfch8hmrg67gv2pql --details
also fails. Had a look at the 
node_executions
flyteadmin table for that particular execution, but most of the columns are of type 
bytea
. Happy to run a query against the table in case it helps.
b
Ok, we archived the specific execution and all is well. Not ideal, but a decent workaround for now.
b
the latest flyteadmin actually has a fix for the utf-8 encoding, which version are you on?
🙇 1
🔥 3
🙌 1
b
helm chart v.1.6.1
b
ah that might be it! could you update and let us know if you still run into this?
b
Will do!
Thanks for all the help on this 🙏
a
@bitter-finland-83254 Updating to the latest flyteadmin has fixed things for us, thank you!
👍 1
160 Views
Open in Slack
PreviousNext
Powered by

Flyte

Flyte enables production-grade orchestration for machine learning workflows and data processing created to accelerate local workflows to production.

Powered by