https://flyte.org logo
#ask-the-community
Title
# ask-the-community
g

Gerry Meixiong

03/10/2023, 10:57 PM
I’m also seeing several workflow executions abort with the error
Workflow execution not found in flyteadmin.
which I don’t fully understand.
k

Ketan (kumare3)

03/10/2023, 11:07 PM
When do they abort
Your db seems off
What postgres are you using
v

Viljem Skornik

03/10/2023, 11:08 PM
RDS 14.5
Looking at flyteadmin logs, the following statement returns 0 rows, then it goes onto the killing spree, and 15 seconds the row is back…
Copy code
0m[33m[0.752ms] [34;1m[rows:1][0m SELECT * FROM "executions" WHERE "executions"."execution_project" = 'flyte-dev' AND "executions"."execution_domain" = 'production' AND "executions"."execution_name" = 're123' LIMIT 1
Copy code
[0m[33m[7.721ms] [34;1m[rows:0][0m SELECT * FROM "executions" WHERE "executions"."execution_project" = 'flyte-dev' AND "executions"."execution_domain" = 'production' AND "executions"."execution_name" = 're123' LIMIT 1

Failed to find existing execution with id [project:"flyte-dev" domain:"production" name:"re123" ] with err: missing entity of type execution with identifier project:"flyte-dev" domain:"production" name:"re123" 

Failed to record task event [task_id:<resource_type:TASK project:"flyte-dev" domain:"production"....
Then we end up with KillTask invoked and Deletion Triggered fro re123. But then 4 minutes later. it’s back:
Copy code
[0m[33m[0.752ms] [34;1m[rows:1][0m SELECT * FROM "executions" WHERE "executions"."execution_project" = 'flyte-dev' AND "executions"."execution_domain" = 'production' AND "executions"."execution_name" = 're123' LIMIT
Flyte deployment is pretty stock via helm chart, with dedicated stock rds instance with logical_replication enabled and no redis in the mix.
k

Ketan (kumare3)

03/10/2023, 11:24 PM
That does don’t sound like RDS right
That sounds like a database correctness issue
We have never seen this
Don’t you think something is odd
k

Kevin Su

03/10/2023, 11:30 PM
do all the tasks get the same error or only this task?
g

Gerry Meixiong

03/10/2023, 11:32 PM
I’ll see tasks output
Some node execution failed, auto-abort.
and the workflow also states
Workflow execution not found in flyteadmin.
typically, all the tasks abort with the same auto-abort error
v

Viljem Skornik

03/10/2023, 11:39 PM
I’m happy to spin up a new db to test this out. Docs suggest Aurora with Postgres compatibility. Is that what you’d recommend?
k

Ketan (kumare3)

03/10/2023, 11:40 PM
Ya we have user aurora a lot and it works great
v

Viljem Skornik

03/10/2023, 11:42 PM
Sounds good I’ll spin up 14.6 and move the dbs over
k

Ketan (kumare3)

03/11/2023, 12:20 AM
cool, please let us know if you see this behavior
y

Yee

03/11/2023, 12:23 AM
can you search for this line in your logs?
Copy code
The node execution launched an execution but it does not exist
k

Ketan (kumare3)

03/11/2023, 12:23 AM
@Gerry Meixiong / @Viljem Skornik Wait - @Yee thinks this is red-herring
y

Yee

03/11/2023, 12:23 AM
(this would be admin logs if you’re running them as separate services)
g

Gerry Meixiong

03/11/2023, 12:28 AM
I don’t see that line, but I do see
/go/src/github.com/flyteorg/flyteadmin/pkg/repositories/gormimpl/task_execution_repo.go:55 record not found
in the admin logs
v

Viljem Skornik

03/11/2023, 1:02 AM
One thing I missed while focusing on the logs was that our propeller was ooming, upped the memory limit and we got a step further I believe.
d

David Espejo (he/him)

03/13/2023, 4:05 PM
@Viljem Skornik / @Gerry Meixiong what's the current status?
v

Viljem Skornik

03/13/2023, 7:10 PM
Stuck on:
Copy code
Attempt 01
aborted
Some node execution failed, auto-abort.
Cant find anything useful. Kube logs show pod getting terminated, nothing out of the ordinary, except that the pod is then gone vs left in Successful state. propeller logs show nothing other than handling Abort event…
And this happens on random nodes relaunching the same workflow
Welp, running two flyte’s in different namespaces was the problem. All problems gone by running just one flyte install cluster wide.
23 Views