https://github.com/fgouteroux/atm
Alertmanager silences distributor multi-tenants
https://github.com/fgouteroux/atm
alertmanager amtool multi-tenants prometheus silences
Last synced: 4 months ago
JSON representation
Alertmanager silences distributor multi-tenants
- Host: GitHub
- URL: https://github.com/fgouteroux/atm
- Owner: fgouteroux
- License: apache-2.0
- Created: 2024-07-01T15:25:33.000Z (11 months ago)
- Default Branch: main
- Last Pushed: 2024-07-02T13:45:19.000Z (11 months ago)
- Last Synced: 2024-12-30T07:42:17.547Z (5 months ago)
- Topics: alertmanager, amtool, multi-tenants, prometheus, silences
- Language: Go
- Homepage:
- Size: 26.4 KB
- Stars: 1
- Watchers: 1
- Forks: 0
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
- Changelog: CHANGELOG.md
- License: LICENSE
Awesome Lists containing this project
README
# atm (alertmanager tenants mute)
...aka Alertmanager silences distributor
This a dirty copy of [amtool](https://github.com/prometheus/alertmanager?tab=readme-ov-file#amtool) which allow to create silence for multi-tenants.
It support the same config file format, but has only the `silence add` cmd.
## usage
```
usage: atm silence add [] [...]
Add a new alertmanager silence
atm uses a simplified Prometheus syntax to represent silences. The
non-option section of arguments constructs a list of "Matcher Groups"
that will be used to create a number of silences. The following examples
will attempt to show this behaviour in action:
atm silence add alertname=foo node=bar
This statement will add a silence that matches alerts with the
alertname=foo and node=bar label value pairs set.
atm silence add foo node=bar
If alertname is omitted and the first argument does not contain a '=' or a
'=~' then it will be assumed to be the value of the alertname pair.
atm silence add 'alertname=~foo.*'
As well as direct equality, regex matching is also supported. The '=~' syntax
(similar to Prometheus) is used to represent a regex match. Regex matching
can be used in combination with a direct match.
Flags:
-h, --[no-]help Show context-sensitive help (also try --help-long and --help-man).
--date.format="2006-01-02 15:04:05 MST"
Format of date output
--alertmanager.url=ALERTMANAGER.URL
Alertmanager to talk to
--timeout=30s Timeout for the executed command
--http.config.file=
HTTP client configuration file for atm to connect to Alertmanager.
--[no-]version Show application version.
-t, --tenant=TENANT tenant
--tenant.http-header="X-Scope-OrgID"
tenant HTTP Header
--tenant.file= tenant file location
-a, --author="fgouteroux" Username for CreatedBy field
-d, --duration="1h" Duration of silence
--max-duration="12h" Max Duration of silence
--start=START Set when the silence should start. RFC3339 format
2006-01-02T15:04:05-07:00
--end=END Set when the silence should end (overwrites duration). RFC3339 format
2006-01-02T15:04:05-07:00
-c, --comment=COMMENT A comment to help describe the silence
Args:
[] Query filter
```
## How it works
When using grafana mimir with multi-tenancy enabled, we need to pass the HTTP tenant ID header.
atm take a tenants list file and iterate over each tenant, set the expected header ( `X-Scope-OrgID` by default) and create the silence.
We add a default max duration of 12h by default to avoid muting all tenants alerts for a long time.
### Create multi-tenants silences
examples/tenants.conf
```
tenant-a
tenant-b
```
Add silence on all tenants in examples/tenants.conf file
```
atm silence add alertname="test" --comment test-alert --duration 10m --tenant.file examples/tenants.conf
Silence added for 'tenant-a' tenant: 1fb1199b-6aec-4575-b6d4-cc5631b77326
Silence added for 'tenant-b' tenant: 0fed624d-2e62-43b8-a940-a337f40e4f05
```
## Limitations
atm couldn't view/expire silences because a silence could be created multiple time with the same matcher, so it's hard to know which silence to expire.