I also see a note from a meeting I did not attend where we have written down that you have an SLO for Flyte Execution Success rate. Do you have a query to share on how you calculate this and what metrics to use?

We also see no values for

flyte:propeller:all:workflow:completion_latency_ms_count

is there a way to get number of completions?

For each success the duration (in ms) is reported. I think using a count over these values is the correct way to count the total number of successes. It shouldn't be necessary to emit two separate metrics.

The completion latency metric refers to the time is takes to transition from the workflow end node start to workflow success . I don't think this is what you're looking for, you're looking for a total count of completed workflows correct?

Not sure on SLO - perhaps @freezing-airport-6809 / @high-park-82026 can touch on this?

A better way would be to use the db for these metrics

@acoustic-cpu-86019 created this issue to capture these numeric metrics https://github.com/flyteorg/flyte/issues/2079

@freezing-airport-6809 Ok, why does it not work to use these metrics we are using? We are currently doing this to figure out how often our canary workflow is succeeding

message has been deleted

I think the but seeing your dashboard I understand- you want an indicator and not an actual number

To create a percentage

Let me Think and compose an answer

We are also interested in the number of internal server errors for create_task, we only find this for create_workflow

flyte:admin:create_workflow:codes:Internal

. For create_task the errors doesn’t seem to distinguish between invalid_arg codes and errors

flyte:admin:create_task:errors

(also create_launch_plan)

To give some more information on how we calculate the values. This is our current SLO dashboard. We calculate the metrics like: 1. Workflow execution success rate > 99.5% of the month (Canary success rate) a. sum of 

flyte:propeller:all:workflow:accepted

 occurrences for our canary workflow the past 30 days -  sum of 

flyte:propeller:all:workflow:failure_duration_ms_count

  occurrences for our canary workflow the past 30 days/ a constant number on how many times we expect the workflow to run in 30 days b. TODO: Would be nice to exchange this constant number to sum of completions 2. Workflow execution success rate > 99.5% of the month (Create Execution RPC success rate) a. sum of 

flyte:admin:create_execution:codes:Internal

 the past 30 days / 

flyte:admin:create_execution:codes:OK

  the past 30 days + 

flyte:admin:create_execution:codes:Internal

 the past 30 days + 

flyte:admin:create_execution:codes:InvalidArgument

the past 30 days b. Haven’t decided which one of metric 1 or 2 to use 3. Register workflow success rate > 99.5% of the month a. Same as 2 but with

flyte:admin:create_workflow*

b. TODO: We want to do the same for create_task and create_launch_plan but there is no 

create_task:codes:Internal

Hi again and good morning! 👋 This is our final dashboard, would be interested to hear your thoughts on these metrics and approach or if you think we need to take some other approach to it. Also had a last question about our last SLO. We want to calculate the time flyte spends on executing user code vs the time flyte spends on “overhead”, e.g., how much the workflow is delayed. I thought that overhead time should vary depending on how many nodes a workflow has so used (1-(

flyte:propeller:all:node:queueing_latency_unlabeled_ms_sum

+

flyte:propeller:all:node:transition_latency_unlabeled_ms_sum

) /`flytepropellerallnodenode_exec_latency_unlabeled_us_sum`)*100 which I hope is calculating how much time a node spends in queuing and transition out of its total execution time. Does that make sense or am I doing something wrong?

for Part 2:

(1-(flyte:propeller:all:node:queueing_latency_unlabeled_ms_sum + flyte:propeller:all:node:transition_latency_unlabeled_ms_sum  ) /flyte:propeller:all:node:node_exec_latency_unlabeled_us_sum)

In the above query this metric math is potentially incorrect

flyte:propeller:all:node:queueing_latency_unlabeled_ms_sum + flyte:propeller:all:node:transition_latency_unlabeled_ms_sum

as you are not using

avg

or

sum

or some aggregation. So i am not sure if this is for one workflow or random workflows.

Now lets try to see how we can get good metrics

I think we should add a new metric here - For Node Overhead • https://github.com/flyteorg/flytepropeller/blob/master/pkg/controller/nodes/executor.go#L702 Which can be computed as follows as a percentage Queue Overhead: (nodeStatus.GetStartedAt() - nodeStatus.GetQueuedAt()) Maybe your query itself can be improved if you perform avg maybe?

(1-(avg(flyte:propeller:all:node:queueing_latency_unlabeled_ms_sum) + avg(flyte:propeller:all:node:transition_latency_unlabeled_ms_sum)  ) /avg(flyte:propeller:all:node:node_exec_latency_unlabeled_us_sum))

Also all these times are recorded in flyteadmin database