lively-engineer-47231
03/10/2023, 10:57 PMWorkflow execution not found in flyteadmin.
which I don’t fully understand.freezing-airport-6809
freezing-airport-6809
freezing-airport-6809
wooden-airline-96737
03/10/2023, 11:08 PMwooden-airline-96737
03/10/2023, 11:09 PM0m[33m[0.752ms] [34;1m[rows:1][0m SELECT * FROM "executions" WHERE "executions"."execution_project" = 'flyte-dev' AND "executions"."execution_domain" = 'production' AND "executions"."execution_name" = 're123' LIMIT 1
wooden-airline-96737
03/10/2023, 11:15 PM[0m[33m[7.721ms] [34;1m[rows:0][0m SELECT * FROM "executions" WHERE "executions"."execution_project" = 'flyte-dev' AND "executions"."execution_domain" = 'production' AND "executions"."execution_name" = 're123' LIMIT 1
Failed to find existing execution with id [project:"flyte-dev" domain:"production" name:"re123" ] with err: missing entity of type execution with identifier project:"flyte-dev" domain:"production" name:"re123"
Failed to record task event [task_id:<resource_type:TASK project:"flyte-dev" domain:"production"....
Then we end up with KillTask invoked and Deletion Triggered fro re123.
But then 4 minutes later. it’s back:
[0m[33m[0.752ms] [34;1m[rows:1][0m SELECT * FROM "executions" WHERE "executions"."execution_project" = 'flyte-dev' AND "executions"."execution_domain" = 'production' AND "executions"."execution_name" = 're123' LIMIT
wooden-airline-96737
03/10/2023, 11:20 PMfreezing-airport-6809
freezing-airport-6809
freezing-airport-6809
freezing-airport-6809
glamorous-carpet-83516
03/10/2023, 11:30 PMlively-engineer-47231
03/10/2023, 11:32 PMSome node execution failed, auto-abort.
and the workflow also states Workflow execution not found in flyteadmin.
lively-engineer-47231
03/10/2023, 11:33 PMwooden-airline-96737
03/10/2023, 11:39 PMfreezing-airport-6809
wooden-airline-96737
03/10/2023, 11:42 PMfreezing-airport-6809
thankful-minister-83577
The node execution launched an execution but it does not exist
freezing-airport-6809
thankful-minister-83577
lively-engineer-47231
03/11/2023, 12:28 AM/go/src/github.com/flyteorg/flyteadmin/pkg/repositories/gormimpl/task_execution_repo.go:55 record not found
in the admin logswooden-airline-96737
03/11/2023, 1:02 AMaverage-finland-92144
03/13/2023, 4:05 PMwooden-airline-96737
03/13/2023, 7:10 PMAttempt 01
aborted
Some node execution failed, auto-abort.
Cant find anything useful. Kube logs show pod getting terminated, nothing out of the ordinary, except that the pod is then gone vs left in Successful state.
propeller logs show nothing other than handling Abort event…wooden-airline-96737
03/13/2023, 7:12 PMwooden-airline-96737
03/13/2023, 8:50 PM