{"id":13616354,"url":"https://github.com/alerta/prometheus-config","last_synced_at":"2025-10-09T16:21:15.861Z","repository":{"id":3258337,"uuid":"48889520","full_name":"alerta/prometheus-config","owner":"alerta","description":"Prometheus config for Alerta","archived":false,"fork":false,"pushed_at":"2024-06-28T11:10:49.000Z","size":945,"stargazers_count":258,"open_issues_count":5,"forks_count":102,"subscribers_count":8,"default_branch":"master","last_synced_at":"2025-03-04T14:34:13.728Z","etag":null,"topics":["alerta","alertmanager","prometheus"],"latest_commit_sha":null,"homepage":"","language":null,"has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/alerta.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2016-01-01T21:10:51.000Z","updated_at":"2025-02-19T14:22:37.000Z","dependencies_parsed_at":"2024-11-08T00:32:10.309Z","dependency_job_id":"1a140c9b-c9a1-4d34-af97-16bb0c05dd4f","html_url":"https://github.com/alerta/prometheus-config","commit_stats":{"total_commits":39,"total_committers":9,"mean_commits":4.333333333333333,"dds":0.641025641025641,"last_synced_commit":"ab8195d8dd9d2b789ee936fcb44aae480963bba3"},"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/alerta/prometheus-config","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/alerta%2Fprometheus-config","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/alerta%2Fprometheus-config/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/alerta%2Fprometheus-config/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/alerta%2Fprometheus-config/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/alerta","download_url":"https://codeload.github.com/alerta/prometheus-config/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/alerta%2Fprometheus-config/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":279001770,"owners_count":26083171,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-10-09T02:00:07.460Z","response_time":59,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["alerta","alertmanager","prometheus"],"created_at":"2024-08-01T20:01:27.395Z","updated_at":"2025-10-09T16:21:15.814Z","avatar_url":"https://github.com/alerta.png","language":null,"funding_links":[],"categories":["Others"],"sub_categories":[],"readme":"Prometheus Config for Alerta\n============================\n\nConsolidate alerts from [Prometheus](http://prometheus.io/) and other\ntools (like Nagios or Zabbix) into a single \"at-a-glance\" console.\n\nTransform this ...\n\n![alertmanager](/docs/images/prometheus-alertmanager.png?raw=true)\n\nInto this ...\n\n![alerta](/docs/images/alerta-webui-v7.jpg?raw=true)\n\nInstallation\n------------\n\nInstall the following:\n\n  * Prometheus 2.0\n  * Prometheus Alertmanager\n  * Alerta\n\nConfiguration - Alertmanager\n----------------------------\n\nThis integration takes advantage of configurable webhooks available with\nPrometheus [Alertmanager](http://prometheus.io/docs/alerting/alertmanager/).\n\nSupport for Prometheus is built-in to Alerta so no special configuration\nis required other than to ensure the webhook URL is correct in the\nAlertmanager config file.\n\n**Example `alertmanager.yml` receivers section**\n\n```\nreceivers:\n- name: \"alerta\"\n  webhook_configs:\n  - url: 'http://localhost:8080/webhooks/prometheus'\n    send_resolved: true\n```\n\n**Note:** If the [Docker container for Alerta](https://hub.docker.com/r/alerta/alerta-web/)\nis used then the webhook URL will use a host and port specific to\nyour environment and the URL path will be `/api/webhooks/prometheus`.\n\n### Authentication\n\nIf Alerta is configured to enforce authentication then the receivers\nsection should define BasicAuth username and password or the webhook\nURL should include an API key. Bearer tokens are not recommended for\nauthenticating external systems with Alerta.\n\n**Example `alertmanager.yml` receivers section with BasicAuth**\n\n```\nreceivers:\n- name: \"alerta\"\n  webhook_configs:\n  - url: 'http://alerta:8080/api/webhooks/prometheus'\n    send_resolved: true\n    http_config:\n      basic_auth:\n        username: admin@alerta.io\n        password: alerta\n```\n\n**Example `alertmanager.yml` receivers section with API Key**\n\n```\nreceivers:\n- name: \"alerta\"\n  webhook_configs:\n  - url: 'http://localhost:8080/webhooks/prometheus?api-key=QBPALlsFSkokm-XiOSupkbpK4SJdFBtStfrOjcdG'\n    send_resolved: true\n```\n\nConfiguration - Rules\n---------------------\n\nPrometheus rules define thresholds at which alerts should be triggered.\nThe following table illustrates how Prometheus notification data is\nused to populate Alerta attributes in those triggered alerts:\n\n| **Prometheus**         | Type          | **Alerta**   |\n-------------------------|---------------|---------------\n| instance (*)           | internal      | resource     |\n| event or alertname (*) | label/internal| event        |\n| environment            | label         | environment  |\n| customer               | label         | customer     |\n| severity               | label         | severity (+) |\n| correlate              | label         | correlate    |\n| service                | label         | service      |\n| group or job (*)       | label/internal| group        |\n| value                  | annotation    | value        |\n| description or summary | annotation    | text         |\n| unassigned labels      | label         | tags         |\n| unassigned annotations | annotation    | attributes   |\n| monitor (*)            | label         | origin       |\n| externalURL (*)        | internal      | externalUrl  |\n| generatorlURL (*)      | internal      | moreInfo     |\n| \"prometheusAlert\"      | n/a           | type         |\n| raw notification       | n/a           | rawData      |\n| timeout (\\*\\*)         | label         | timeout      |\n\nNote: `value` has changed from a `label` to an `annotation`.\n\nPrometheus labels marked with a star (*) are built-in and assignment\nto Alerta attributes happens automatically. All other labels or\nannotations are user-defined and completely optional as they have\nsensible defaults.\n\nPrometheus label timeout use default value 0 if not exists.\n\n### Missing Instance Labels\n\nNote that during configuration of complex alarms with sum\nand rate function the `instance` and `exported_instance` labels can\nbe missed from an alarm. These will need to be setup manually with:\n\n```\n- alert: rate_too_low\n  expr: sum by(host) (rate(foo[5m])) \u003c 10\n  for: 2m\n  labels:\n    environment: Production\n    instance: '{{$labels.host}}'  # \u003c== instance added to labels manually\n  annotations:\n    value: '{{$value}}'\n```\n\n### Using `event` or `alertname`\n\nBy default the `alertname` label is used to populate the `event` attribute. However,\nthere are times where this simple 1-to-1 mapping is not sufficient and the event\nneeds to include other information, like filesystem mountpoints, to ensure that\nalerts for different resources aren't incorrectly deduplicated.\n\nIt is therefore possible to define an `event` label which will be used instead of\nthe `alertname`. The `event` label can include the `alertname` plus other\ninformation if required.\n\n### Python Format Syntax in Rules\n\nRegretably it isn't possible to refer to `alertname`[[1]],\nexternal labels [[2]], or labels in annotations [[3]]\n [[4]]. These limitations are due to design decisions which\nare [not going to change in the forseeable future][5]\nyet make integration with Prometheus via generic webhooks very difficult\nbeyond extremely simple use cases [[6]] [[7]] [[8]] [[9]].\n\n[1]: https://github.com/prometheus/prometheus/issues/2818\n[2]: https://github.com/prometheus/prometheus/issues/3043\n[3]: https://github.com/prometheus/prometheus/issues/2454\n[4]: https://github.com/prometheus/prometheus/issues/2992\n[5]: https://github.com/prometheus/prometheus/issues/2818\n[6]: https://github.com/prometheus/alertmanager/issues/1496\n[7]: https://github.com/prometheus/alertmanager/pull/1422\n[8]: https://github.com/prometheus/alertmanager/issues/684\n[9]: https://github.com/prometheus/alertmanager/issues/1400\n\nThe work-around developed for Alerta is to allow Prometheus rules to contain\npython format syntax that can refer to labels anywhere in a rule, ie. in either\nanother label or an annotation, and then pre-parse the labels and annotations\nwhen receiving a Prometheus alert before processing it as normal. Like so:\n\n```\n  labels:\n    foo: bar\n    baz: {foo}\n  annotations:\n    quux: the foo is {foo} and the baz is {baz}\n```\n\nThis will be interpreted as:\n\n```\n  labels:\n    foo: bar\n    baz: bar\n  annotations:\n    quux: the foo is bar and the baz is bar\n```\n\nThis makes refering to labels in annotations to create things like dynamic\n\"runbook\" URLs very straight-forward. An example the does this can be found\nunder the heading \"Python Format Template Example\" below.\n\nExamples\n--------\n\nMany Prometheus values can be used to populate attributes in Alerta alerts\nto enrich the available alert information if reasonable values are assigned\nwhere possible.\n\nThis is demonstrated in the example alert rules below, which become increasingly\nmore informative.\n\n### Testing the examples\n\nUse the provided `prometheus.yml`, `prometheus.rules.yml` and `alertmanager.yml`\nfiles in the `/example` directory to start with and run `prometheus` and\n`alertmanager` as follows:\n\n    $ ./prometheus -config.file=prometheus.yml -alertmanager.url=http://localhost:9093\n    $ ./alertmanager -config.file=alertmanager.yml\n\nOr if you have Docker installed run:\n\n    $ docker-compose up -d\n\nPrometheus Web =\u003e http://localhost:9090\n\nAlertmanager Web =\u003e http://localhost:9093\n\n### Basic Example\n\nThe example rule below is the absolute minimum required to trigger a\n\"warning\" alert and a corresponding \"normal\" alert for forwarding to\nAlerta.\n\n```\n- alert: MinimalAlert\n  expr: metric \u003e 0\n```\n\n### Simple Example\n\nThis example sets the severity to `major` and defines a description to\nbe used as the alert text.\n\n```\n- alert: SimpleAlert\n  expr: metric \u003e 0\n  labels:\n    severity: major\n  annotations:\n    description: simple alert triggered at {{$value}}\n```\n\n### Python Format Template Example\n\nThis example uses [python string format syntax](https://docs.python.org/3.4/library/string.html#format-string-syntax)\n(ie. curly braces `{}`) to template the values for `app` (in `runbook`) and `alertname` (in `event` and `runbook`).\n\n```\n- alert: TemplateAlert\n  expr: metric \u003e 0\n  labels:\n    event: {alertname}:{{ $labels.mountpoint }}\n    severity: major\n  annotations:\n    description: simple alert triggered at {{$value}}\n    runbook: https://internal.myorg.net/wiki/alerts/{app}/{alertname}\n```\n\n### Complex Example\n\nA more complex example where `external_labels` defined globally\nare used to populate common alert attributes like `environment`,\n`service` and `monitor` (used by `origin`). The alert value is set\nusing the `$value` label.\n```\nglobal:\n  external_labels:\n    environment: Production\n    service: Prometheus\n    monitor: codelab\n```\n\n```\n- alert: CompleteAlert\n  expr: metric \u003e 0\n  labels:\n    severity: minor\n  annotations:\n    description: complete alert triggered at {{$value}}\n    value: '{{ humanize $value }} req/s'\n```\n\n### Complex Example using Correlation\n\n```\n- alert: NodeDown\n  expr: up == 0\n  labels:\n    severity: minor\n    correlate: NodeUp,NodeDown\n  annotations:\n    description: Node is down.\n- alert: NodeUp\n  expr: up == 1\n  labels:\n    severity: ok\n    correlate: NodeUp,NodeDown\n  annotations:\n    description: Node is up.\n```\n\nIt is desirable that the `prometheus.yml` and `prometheus.rules.yml`\nconfiguration files conform to an expected format but this is not\nmandatory.\n\nIt is possible to set global labels which will be used for all alerts\nthat are sent to Alerta. For instance, you can label your server with\n'Production' or 'Development'. You can also describe the service, like\n'Prometheus'.\n\nExample `prometheus.yml` Global section:\n\n```\nglobal:\n  external_labels:\n    environment: Production\n    service: Prometheus\n    monitor: codelab\n```\n\n### Complex Example using Correlation without Resolve\n\n**Example `alertmanager.yml` with `send_resolve` disabled**\n```\nreceivers:\n- name: \"alerta\"\n  webhook_configs:\n  - url: 'http://alerta:8080/api/webhooks/prometheus'\n    send_resolved: false\n    http_config:\n      basic_auth:\n        username: admin@alerta.io\n        password: alerta\n```\n\n**Example `prometheus.rules.yml` with explicit \"normal\" rule**\n```\n  # system load alert\n  - alert: load_vhigh\n    expr: node_load1 \u003e= 0.7\n    labels:\n      severity: major\n      correlate: load_vhigh,load_high,load_ok\n    annotations:\n      description: '{{ $labels.instance }} of job {{ $labels.job }} is under very high load.'\n      value: '{{ $value }}'\n  - alert: load_high\n    expr: node_load1 \u003e= 0.5 and node_load1 \u003c 0.7\n    labels:\n      severity: warning\n      correlate: load_vhigh,load_high,load_ok\n    annotations:\n      description: '{{ $labels.instance }} of job {{ $labels.job }} is under high load.'\n      value: '{{ $value }}'\n  - alert: load_ok\n    expr: node_load1 \u003c 0.5\n    labels:\n      severity: normal\n      correlate: load_vhigh,load_high,load_ok\n    annotations:\n      description: '{{ $labels.instance }} of job {{ $labels.job }} is under normal load.'\n      value: '{{ $value }}'\n```\n\n**Note:** The \"load_high\" rule expression needs to bracket the\n`node_load1` metric value between the \"load_vhigh\" and the \"load_ok\"\nvalues ie. `expr: node_load1 \u003e= 0.5 and node_load1 \u003c 0.7`\n\nMetrics\n-------\n\nAlerta exposes prometheus metrics natively on `/management/metrics` so\nalerts can be generated based on Alerta performance.\n\n[Counter, Gauge and Summary metrics](http://prometheus.io/docs/concepts/metric_types/)\nare exposed and all use `alerta` as the application prefix. Metrics are\ncreated lazily, so for example, a summary metric for the number of deleted\nalerts will not be present in the metric output if an alert has never\nbeen deleted. Note that counters and summaries are **not** reset when\nAlerta restarts.\n\n*Example Metrics*\n\n```\n# HELP alerta_alerts_total Total number of alerts in the database\n# TYPE alerta_alerts_total gauge\nalerta_alerts_total 1\n# HELP alerta_alerts_rejected Number of rejected alerts\n# TYPE alerta_alerts_rejected counter\nalerta_alerts_rejected_total 3\n# HELP alerta_alerts_duplicate Total time to process number of duplicate alerts\n# TYPE alerta_alerts_duplicate summary\nalerta_alerts_duplicate_count 339\nalerta_alerts_duplicate_sum 1378\n# HELP alerta_alerts_received Total time to process number of received alerts\n# TYPE alerta_alerts_received summary\nalerta_alerts_received_count 20\nalerta_alerts_received_sum 201\n# HELP alerta_plugins_prereceive Total number of pre-receive plugins\n# TYPE alerta_plugins_prereceive summary\nalerta_plugins_prereceive_count 390\nalerta_plugins_prereceive_sum 10\n# HELP alerta_plugins_postreceive Total number of post-receive plugins\n# TYPE alerta_plugins_postreceive summary\nalerta_plugins_postreceive_count 387\nalerta_plugins_postreceive_sum 3\n# HELP alerta_alerts_create Total time to process number of new alerts\n# TYPE alerta_alerts_create summary\nalerta_alerts_create_count 26\nalerta_alerts_create_sum 85\n# HELP alerta_alerts_queries Total time to process number of alert queries\n# TYPE alerta_alerts_queries summary\nalerta_alerts_queries_count 57357\nalerta_alerts_queries_sum 195402\n# HELP alerta_alerts_deleted Total time to process number of deleted alerts\n# TYPE alerta_alerts_deleted summary\nalerta_alerts_deleted_count 32\nalerta_alerts_deleted_sum 59\n# HELP alerta_alerts_tagged Total time to tag number of alerts\n# TYPE alerta_alerts_tagged summary\nalerta_alerts_tagged_count 1\nalerta_alerts_tagged_sum 4\n# HELP alerta_alerts_untagged Total time to un-tag number of alerts\n# TYPE alerta_alerts_untagged summary\nalerta_alerts_untagged_count 1\nalerta_alerts_untagged_sum 1\n# HELP alerta_alerts_status Total time and number of alerts with status changed\n# TYPE alerta_alerts_status summary\nalerta_alerts_status_count 2\nalerta_alerts_status_sum 3\n# HELP alerta_alerts_webhook Total time to process number of web hook alerts\n# TYPE alerta_alerts_webhook summary\nalerta_alerts_webhook_count 344\nalerta_alerts_webhook_sum 3081\n# HELP alerta_alerts_correlate Total time to process number of correlated alerts\n# TYPE alerta_alerts_correlate summary\nalerta_alerts_correlate_count 23\nalerta_alerts_correlate_sum 69\n```\n\n\nReferences\n----------\n\n* Kubernetes [namespaces](http://kubernetes.io/docs/user-guide/namespaces/)\n* Kubernetes [labels](http://kubernetes.io/docs/user-guide/labels/)\n* Kubernetes [annotations](http://kubernetes.io/docs/user-guide/annotations/)\n* Kubernetes [metadata](https://github.com/kubernetes/kubernetes/blob/master/docs/devel/api-conventions.md#metadata)\n\nLicense\n-------\n\nCopyright (c) 2016-2018 Nick Satterly. Available under the MIT License.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Falerta%2Fprometheus-config","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Falerta%2Fprometheus-config","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Falerta%2Fprometheus-config/lists"}