I m also seeing several workflow executions abort with the e

Question

I m also seeing several workflow executions abort with the error `Workflow execution not found in flyteadmin ` which I don t fully understand

Ketan (kumare3) · Answer

When do they abort

Ketan (kumare3) · Answer

Your db seems off

Ketan (kumare3) · Answer

What postgres are you using

Viljem Skornik · Answer

RDS 14 5

Viljem Skornik · Answer

Looking at flyteadmin logs the following statement returns 0 rows then it goes onto the killing spree and 15 seconds the row is back ```0m 33m 0 752ms 34 1m rows 1 0m SELECT FROM executions WHERE executions execution project = flyte dev AND executions execution domain = production AND executions execution name = re123 LIMIT 1```

Viljem Skornik · Answer

``` 0m 33m 7 721ms 34 1m rows 0 0m SELECT FROM executions WHERE executions execution project = flyte dev AND executions execution domain = production AND executions execution name = re123 LIMIT 1 Failed to find existing execution with id project flyte dev domain production name re123 with err missing entity of type execution with identifier project flyte dev domain production name re123 Failed to record task event task id lt resource type TASK project flyte dev domain production ``` Then we end up with KillTask invoked and Deletion Triggered fro re123 But then 4 minutes later it s back ``` 0m 33m 0 752ms 34 1m rows 1 0m SELECT FROM executions WHERE executions execution project = flyte dev AND executions execution domain = production AND executions execution name = re123 LIMIT ```

Viljem Skornik · Answer

Flyte deployment is pretty stock via helm chart with dedicated stock rds instance with logical replication enabled and no redis in the mix

Ketan (kumare3) · Answer

That does don t sound like RDS right

Ketan (kumare3) · Answer

That sounds like a database correctness issue

Ketan (kumare3) · Answer

We have never seen this

Ketan (kumare3) · Answer

Don t you think something is odd

Kevin Su · Answer

do all the tasks get the same error or only this task

Gerry Meixiong · Answer

I ll see tasks output `Some node execution failed auto abort ` and the workflow also states `Workflow execution not found in flyteadmin `

Gerry Meixiong · Answer

typically all the tasks abort with the same auto abort error

Viljem Skornik · Answer

I m happy to spin up a new db to test this out Docs suggest Aurora with Postgres compatibility Is that what you d recommend

Ketan (kumare3) · Answer

Ya we have user aurora a lot and it works great

Viljem Skornik · Answer

Sounds good I ll spin up 14 6 and move the dbs over

Ketan (kumare3) · Answer

cool please let us know if you see this behavior

Yee · Answer

can you search for this line in your logs ```The node execution launched an execution but it does not exist```

Ketan (kumare3) · Answer

< Gerry Meixiong> < Viljem Skornik> Wait < Yee> thinks this is red herring

Yee · Answer

this would be admin logs if you re running them as separate services

Gerry Meixiong · Answer

I don t see that line but I do see ` go src github com flyteorg flyteadmin pkg repositories gormimpl task execution repo go 55 record not found` in the admin logs

Viljem Skornik · Answer

One thing I missed while focusing on the logs was that our propeller was ooming upped the memory limit and we got a step further I believe

David Espejo (he/him) · Answer

< Viljem Skornik> < Gerry Meixiong> what s the current status

Viljem Skornik · Answer

Stuck on ```Attempt 01 aborted Some node execution failed auto abort ``` Cant find anything useful Kube logs show pod getting terminated nothing out of the ordinary except that the pod is then gone vs left in Successful state propeller logs show nothing other than handling Abort event

Viljem Skornik · Answer

And this happens on random nodes relaunching the same workflow

Viljem Skornik · Answer

Welp running two flyte s in different namespaces was the problem All problems gone by running just one flyte install cluster wide