https://flyte.org
Join Slack
hey! is there a metric in propeller to measure failures from plugins? We use these recording rules c...
# announcements
c
hey! is there a metric in propeller to measure failures from plugins? We use these recording rules currently, but each time we add a new plugin we need to go and add it to the rules 🧵
Copy code
- record: computed:propeller:all:node:plugin:failure_unlabeled
  expr: |    label_replace(flyte:propeller:all:node:plugin:bigquery_ddl_create_dataset_task_bigquery_api_failure_unlabeled, "plugin", "bigquery_ddl_create_dataset_task_bigquery_api", "", "")

- record: computed:propeller:all:node:plugin:failure_unlabeled
  expr: |
    label_replace(flyte:propeller:all:node:plugin:flink_flink_failure_unlabeled, "plugin", "flink_flink", "", "")
a
hey @colossal-solstice-11091 what do you mean by failures from plugins?
c
I guess if a reconcile failure is due to any specific plugin having problems
a
this is where we do the plugin to task type resolution but it looks like we only log if we encounter issues. are you mostly trying to make sure if you configure a new plugin it will actually be used for a specific task type?
c
We would like to understand if problems in propeller or if a set of workflows are not progressing is due to a specific plugin. For example when handle is called on a specific plugin, if some exception comes from the part of the code of the plugin.
a
Are the existing error logs sufficient in this case?
c
yes it works! it would be awesome though if it was possible. we had some issues recently where all our metrics looked fine, but one plugin was misbehaving due to some change in the general propeller config, so it would have been nice to be able to alert on it
a
gotcha, please feel free to open a PR!
👍 1
166 Views
Open in Slack
PreviousNext
Powered by

Flyte

Flyte enables production-grade orchestration for machine learning workflows and data processing created to accelerate local workflows to production.

Powered by