@Prafulla Mahindrakar, mind helping @Alex Pozimenko with this? He's trying to get metrics on terminated tasks/executions annotated with the phase they ended as. We merged this PR https://github.com/flyteorg/flyteadmin/pull/386/files and he's trying with admin version 0.6.131 (should include this PR) but can't see any phases under:

flyte-admin-task.execution.manager-task.executions.terminated.counter

Sure 👍

I don't see the phase in the metrics you posted... Can you also check the task and node terminations' metrics?

Task execution metrics

# HELP flyte:admin:admin:task_execution_manager:task_executions_terminated overall count of terminated workflow executions
# TYPE flyte:admin:admin:task_execution_manager:task_executions_terminated counter
flyte:admin:admin:task_execution_manager:task_executions_terminated{app_name="",domain="development",exec_id="fe533aa1e880546b59db",node="n0",project="flytesnacks",runtime_type="",runtime_version="",task="core.control_flow.map_task.mapper_a_mappable_task_0-0",tasktype="",wf=""} 1
flyte:admin:admin:task_execution_manager:task_executions_terminated{app_name="",domain="development",exec_id="fe533aa1e880546b59db",node="n1",project="flytesnacks",runtime_type="",runtime_version="",task="core.control_flow.map_task.coalesce-0",tasktype="",wf=""} 1
flyte:admin:admin:task_execution_manager:task_executions_terminated{app_name="",domain="development",exec_id="ffd9b80a79ca34186889",node="n0",project="flytesnacks",runtime_type="",runtime_version="",task="core.control_flow.map_task.mapper_a_mappable_task_0-0",tasktype="",wf=""} 1
flyte:admin:admin:task_execution_manager:task_executions_terminated{app_name="",domain="development",exec_id="ffd9b80a79ca34186889",node="n1",project="flytesnacks",runtime_type="",runtime_version="",task="core.control_flow.map_task.coalesce-0",tasktype="",wf=""} 1

Node execution metrics

flyte:admin:admin:node_execution_manager:node_executions_terminated{app_name="",domain="development",exec_id="fe533aa1e880546b59db",node="end-node",project="flytesnacks",runtime_type="",runtime_version="",task="",tasktype="",wf=""} 1
flyte:admin:admin:node_execution_manager:node_executions_terminated{app_name="",domain="development",exec_id="fe533aa1e880546b59db",node="n0",project="flytesnacks",runtime_type="",runtime_version="",task="",tasktype="",wf=""} 1
flyte:admin:admin:node_execution_manager:node_executions_terminated{app_name="",domain="development",exec_id="fe533aa1e880546b59db",node="n1",project="flytesnacks",runtime_type="",runtime_version="",task="",tasktype="",wf=""} 1
flyte:admin:admin:node_execution_manager:node_executions_terminated{app_name="",domain="development",exec_id="fe533aa1e880546b59db",node="start-node",project="flytesnacks",runtime_type="",runtime_version="",task="",tasktype="",wf=""} 1
flyte:admin:admin:node_execution_manager:node_executions_terminated{app_name="",domain="development",exec_id="ffd9b80a79ca34186889",node="end-node",project="flytesnacks",runtime_type="",runtime_version="",task="",tasktype="",wf=""} 1
flyte:admin:admin:node_execution_manager:node_executions_terminated{app_name="",domain="development",exec_id="ffd9b80a79ca34186889",node="n0",project="flytesnacks",runtime_type="",runtime_version="",task="",tasktype="",wf=""} 1
flyte:admin:admin:node_execution_manager:node_executions_terminated{app_name="",domain="development",exec_id="ffd9b80a79ca34186889",node="n1",project="flytesnacks",runtime_type="",runtime_version="",task="",tasktype="",wf=""} 1
flyte:admin:admin:node_execution_manager:node_executions_terminated{app_name="",domain="development",exec_id="ffd9b80a79ca34186889",node="start-node",project="flytesnacks",runtime_type="",runtime_version="",task="",tasktype="",wf=""}

I will check why the phase is not being emitted

I think the issue is that we don’t have phase in the metric keys here: https://github.com/flyteorg/flyteadmin/blob/0260c173cc169f37e750d462f7168bbb4152e7b7/cmd/entrypoints/serve.go#L50-L52

Ok that seems to have worked @Haytham Abuelfutuh Node executions

flyte:admin:admin:node_execution_manager:node_executions_terminated{app_name="",domain="development",exec_id="f2282f588a3b940cd99d",node="end-node",phase="SUCCEEDED",project="flytesnacks",runtime_type="",runtime_version="",task="",tasktype="",wf=""} 1
flyte:admin:admin:node_execution_manager:node_executions_terminated{app_name="",domain="development",exec_id="f2282f588a3b940cd99d",node="n0",phase="SUCCEEDED",project="flytesnacks",runtime_type="",runtime_version="",task="",tasktype="",wf=""} 1
flyte:admin:admin:node_execution_manager:node_executions_terminated{app_name="",domain="development",exec_id="f2282f588a3b940cd99d",node="n1",phase="SUCCEEDED",project="flytesnacks",runtime_type="",runtime_version="",task="",tasktype="",wf=""} 1
flyte:admin:admin:node_execution_manager:node_executions_terminated{app_name="",domain="development",exec_id="f2282f588a3b940cd99d",node="start-node",phase="SUCCEEDED",project="flytesnacks",runtime_type="",runtime_version="",task="",tasktype="",wf=""} 1

Task executions

flyte:admin:admin:task_execution_manager:task_executions_terminated{app_name="",domain="development",exec_id="f2282f588a3b940cd99d",node="n0",phase="SUCCEEDED",project="flytesnacks",runtime_type="",runtime_version="",task="core.control_flow.map_task.mapper_a_mappable_task_0-0",tasktype="",wf=""} 1
flyte:admin:admin:task_execution_manager:task_executions_terminated{app_name="",domain="development",exec_id="f2282f588a3b940cd99d",node="n1",phase="SUCCEEDED",project="flytesnacks",runtime_type="",runtime_version="",task="core.control_flow.map_task.coalesce-0",tasktype="",wf=""} 1

Executions

flyte:admin:admin:execution_manager:executions_terminated{app_name="",domain="development",exec_id="f2282f588a3b940cd99d",node="",phase="SUCCEEDED",project="flytesnacks",runtime_type="",runtime_version="",task="",tasktype="",wf=""} 1

@Prafulla Mahindrakar, thanks for the fix. Shall I try flyteadmin v0.6.145 ?

Sorry you seen to have hit a regression!! Where do you see that error? Does the execution start and the task fails with that? When it fails, can you check the pod created to see what service account was set on it? How do you launch the workflow? Is it through flytectl and you pass a k8s service account? @Prafulla Mahindrakar @katrina can you please help investigate this? We can't ship with that regression in a release @Eduardo Apolinario (eapolinario) @Yuvraj can we see how to add this scenario to the endtoend tests?

Hi Alex , knowing how are you launching this workflows would be helpful for further investigation

@Alex Pozimenko are you setting the auth role?

Where do you see that error? Does the execution start and the task fails with that? When it fails, can you check the pod created to see what service account was set on it? How do you launch the workflow? Is it through flytectl and you pass a k8s service account?

1. i launch from the console 2. the task starts, the error is coming from the container when it's trying to access other aws resources 3. i'll check the service account (need to upgrade my environment again 🙂 ) (@Haytham Abuelfutuh @Prafulla Mahindrakar)

hi @Alex Pozimenko thank you for reporting this! we've update the v0.19.4 release with a fix for reading the deprecated auth field in admin. do you mind updating your deployment to unblock?

cool, what version of the admin shall I use?

v0.6.147

is there a matrix that maps release versions to containers?

this was all @Prafulla Mahindrakar 🎉

thank you @Prafulla Mahindrakar!

is there a matrix that maps release versions to containers?

we should publish an image per release: https://github.com/flyteorg/flyteadmin/pkgs/container/flyteadmin/19159524?tag=v0.6.147

right, but how do I know that v0.19.4 is part of v0.6.147 release?

ah it gets retagged under the flyteadmin-release package: https://github.com/flyteorg/flyteadmin/pkgs/container/flyteadmin-release/19161969?tag=v0.19.4

got it!

Glad it worked out for you Alex. And hopefully the metrics issue is resolved as well ?

also we use the non-release "flyteadmin" package in the helm chart: https://github.com/flyteorg/flyte/blob/v0.19.4/charts/flyte-core/values.yaml#L19

now about the missing metric. Looks like the name changed from

flyte-admin-task.execution.manager-task.executions.terminated

to

flyte-admin-admin-execution.manager-executions.terminated

I think failed can mean user errors aswell as infra errors like permission issues

can a workflow be terminated in phase other than failed/succeeded ? or is it always one of the two?

Node execution phases

const (
	NodeExecution_UNDEFINED       NodeExecution_Phase = 0
	NodeExecution_QUEUED          NodeExecution_Phase = 1
	NodeExecution_RUNNING         NodeExecution_Phase = 2
	NodeExecution_SUCCEEDED       NodeExecution_Phase = 3
	NodeExecution_FAILING         NodeExecution_Phase = 4
	NodeExecution_FAILED          NodeExecution_Phase = 5
	NodeExecution_ABORTED         NodeExecution_Phase = 6
	NodeExecution_SKIPPED         NodeExecution_Phase = 7
	NodeExecution_TIMED_OUT       NodeExecution_Phase = 8
	NodeExecution_DYNAMIC_RUNNING NodeExecution_Phase = 9
	NodeExecution_RECOVERED       NodeExecution_Phase = 10
)

Task execution phases

const (
	TaskExecution_UNDEFINED TaskExecution_Phase = 0
	TaskExecution_QUEUED    TaskExecution_Phase = 1
	TaskExecution_RUNNING   TaskExecution_Phase = 2
	TaskExecution_SUCCEEDED TaskExecution_Phase = 3
	TaskExecution_ABORTED   TaskExecution_Phase = 4
	TaskExecution_FAILED    TaskExecution_Phase = 5
	// To indicate cases where task is initializing, like: ErrImagePull, ContainerCreating, PodInitializing
	TaskExecution_INITIALIZING TaskExecution_Phase = 6
	// To address cases, where underlying resource is not available: Backoff error, Resource quota exceeded
	TaskExecution_WAITING_FOR_RESOURCES TaskExecution_Phase = 7
)

Workflow execution phases

const (
	WorkflowExecution_UNDEFINED  WorkflowExecution_Phase = 0
	WorkflowExecution_QUEUED     WorkflowExecution_Phase = 1
	WorkflowExecution_RUNNING    WorkflowExecution_Phase = 2
	WorkflowExecution_SUCCEEDING WorkflowExecution_Phase = 3
	WorkflowExecution_SUCCEEDED  WorkflowExecution_Phase = 4
	WorkflowExecution_FAILING    WorkflowExecution_Phase = 5
	WorkflowExecution_FAILED     WorkflowExecution_Phase = 6
	WorkflowExecution_ABORTED    WorkflowExecution_Phase = 7
	WorkflowExecution_TIMED_OUT  WorkflowExecution_Phase = 8
	WorkflowExecution_ABORTING   WorkflowExecution_Phase = 9
)

Terminal Phases for all

var terminalExecutionPhases = map[core.WorkflowExecution_Phase]bool{
	core.WorkflowExecution_SUCCEEDED: true,
	core.WorkflowExecution_FAILED:    true,
	core.WorkflowExecution_TIMED_OUT: true,
	core.WorkflowExecution_ABORTED:   true,
}

var terminalNodeExecutionPhases = map[core.NodeExecution_Phase]bool{
	core.NodeExecution_SUCCEEDED: true,
	core.NodeExecution_FAILED:    true,
	core.NodeExecution_TIMED_OUT: true,
	core.NodeExecution_ABORTED:   true,
	core.NodeExecution_SKIPPED:   true,
	core.NodeExecution_RECOVERED: true,
}

var terminalTaskExecutionPhases = map[core.TaskExecution_Phase]bool{
	core.TaskExecution_SUCCEEDED: true,
	core.TaskExecution_FAILED:    true,
	core.TaskExecution_ABORTED:   true,
}

thanks, but this doesn't answer the question

@Alex Pozimenko a terminated workflow has a phase in: [succeeded, failed, aborted, timed out]

can we use any of these as an indication of infra failures vs user errors?

not from these metrics i believe. we distinguish between system and user failures in the individual errors output by failed workflows using the kind. we could modify flyteadmin to tag the failures by kind but propeller should also output system errors

so just to be clear, are you saying there's a metric output by propeller that has kind tag?

the propeller metric just has a total count of system errors. but if you want to open a pr to tag failures by system vs. user errors in flyteadmin that would be awesome!

i can try if you point me to the code

awesome! here is where we increment terminated executions with the phase. so for failed executions you can parse the error in the event and add the error kind