Ecosyste.ms: Awesome
An open API service indexing awesome lists of open source software.
https://github.com/qubitproducts/exporter_exporter
A reverse proxy designed for Prometheus exporters
https://github.com/qubitproducts/exporter_exporter
cadvisor ceh infra nginx prometheus prometheus-exporter proxy
Last synced: 5 days ago
JSON representation
A reverse proxy designed for Prometheus exporters
- Host: GitHub
- URL: https://github.com/qubitproducts/exporter_exporter
- Owner: QubitProducts
- License: apache-2.0
- Created: 2016-09-30T14:51:45.000Z (over 8 years ago)
- Default Branch: master
- Last Pushed: 2024-01-25T20:55:57.000Z (11 months ago)
- Last Synced: 2024-12-21T22:12:58.334Z (12 days ago)
- Topics: cadvisor, ceh, infra, nginx, prometheus, prometheus-exporter, proxy
- Language: Go
- Homepage:
- Size: 213 KB
- Stars: 346
- Watchers: 19
- Forks: 55
- Open Issues: 10
-
Metadata Files:
- Readme: README.md
- License: LICENSE
- Authors: AUTHORS
Awesome Lists containing this project
README
# Exporter Exporter - prometheus exporter proxy
``` "you mean apache/nginx" - bbrazil ```
This provides a simple reverse proxy for prometheus exporters. It is intended as a
single binary alternative to nginx/apache for use in environments where opening multiple
TCP ports to all servers might be difficult (technically or politically)
The advantages are:
- A single port can be used to query multiple exporters (to ease firewall configuration concerns).
- Can provide TLS with optional client certificate authentication.
- Provides verification that the target is serving prometheus metrics.
- Can be used to execute scripts that produce prometheus metrics.
- _up_ behaviour is the same as for querying individual collectors.
- Small code size, minimal external depedencies, easily auditable.
The exporter has three endpoints.
- /: displays a list of all exporters with links to their metrics.
- Returns JSON if the header "Accept: application/json" is passed
- /proxy: which takes the following parameters:
- *module*: the name of the module from the configuration to execute. (a default
module can be selected using the defaultModule config option)
- *args*: (only for exec modules): additional arguments to the backend command.
- all other query string parameters are passed on to any http backend module.
(excluding the first *module* parameter value).
- /metrics: this exposes the metrics for the collector itself.
Features that will NOT be included:
- merging of module outputs into one query (this would break _up_ behaviour)
## Installation
You can build directly using a plain ```go get github.com/QubitProducts/exporter_exporter```.
The provided Makefile is primarily used for releases.
Pre-built binaries and a debian package are available on the [GitHub release page](https://github.com/QubitProducts/exporter_exporter/releases).
An ansible recipe as also [available](https://github.com/umanit/ansible-prometheus_exporter_exporter) (kindly provided by one of our users).
TODO:
- Config reload on HUP (or POST, or config file change?)
- route to a docker/rocket container by name
### Windows Service
The binary can be installed as a Windows service by supplying the `-winsvc install` arg.
All other arguments passed along with `-winsvc install` will be added to the service startup
and can only be changed by uninstalling/installing it again (or modifying the Windows registry directly).
## Configuration
In expexp.yaml list each exporter listening on localhost with its known port.
```
defaultModule: node # called if "module" param is not supplied
modules:
node:
method: http
http:
port: 9100
mtail:
method: http
http:
port: 3903
headers:
foo: bar
cadvisor:
method: http
http:
verify: false
port: 4194
netdata:
method: http
http:
port: 19999
path: '/api/v1/allmetrics?format=prometheus'
blackbox:
method: http
http:
port: 9115
path: '/probe'
somescript:
method: exec
timeout: 1s
exec:
command: /tmp/myscript.sh
args:
- "myarg1"
- "myarg2"
env:
THING: "1"
THING2: "2"
somefile:
method: file
file:
path: /tmp/myfile.prometheus.txt
```
In your prometheus configuration
```
scrape_configs:
- job_name: 'expexp_metrics'
scrape_interval: 1s
static_configs:
- targets: ['host:9999']
- job_name: 'cadvisor'
scrape_interval: 5s
metrics_path: /proxy
params:
module:
- cadvisor
static_configs:
- targets: ['host:9999']
- job_name: 'mtail'
scrape_interval: 5s
metrics_path: /proxy
params:
module:
- mtail
static_configs:
- targets: ['host:9999']
- job_name: 'somescript'
scrape_interval: 5s
metrics_path: /proxy
params:
module:
- somescript
static_configs:
- targets: ['host:9999']
- job_name: 'blackbox'
metrics_path: /proxy
params:
module:
- blackbox
- icmp_example
static_configs:
- targets:
- 8.8.8.8
- 8.8.4.4
relabel_configs:
- source_labels: [__address__]
target_label: __param_target
- source_labels: [__param_target]
target_label: instance
- target_label: __address__
replacement: host:9999
```
### Blackbox Exporter
The blackbox exporter also uses the "module" query string parameter. To query it via
exporter_exporter we rely on the stripping of the initial "module" parameter. For example
```
curl http://localhost:9999/proxy\?module\=blackbox\&module\=icmp_example\&target\=8.8.8.8
```
Will query the icmp_example module in your blackbox configuration.
## Directory-based configuration
You can also specify `-config.dirs` to break the configuration into separate
files. The module name is taken from the name of the file (minus the
yml/yaml extension), and the configuration for that module goes in at the
top level.
Note that if you want to use *only* this configuration method and not the file-based
configuration (`-config.file` option), you must provide an empty string for the file
option : `./exporter_exporter -config.file "" -config.dirs "/etc/exporter_exporter/"`
```
==> expexp.yaml <==
modules: {}
==> expexp.d/node.yaml <==
method: http
http:
port: 9100
==> expexp.d/mtail.yaml <==
method: http
http:
port: 3903
```
## TLS configuration
You can use exporter_exporter with TLS to encrypt the traffic, and at the
same time enforce strong mutual authentication between the nodes and the
prometheus server.
Note that `-web.tls.verify` will accept *any* certificate signed by the
`-web.tls.ca`, so you need to create a separate CA for this purpose - or use
a self-signed certificate, which acts as its own CA.
Here is a simple configuration example, using one key/cert for the
prometheus server and one key/cert shared between all the remote nodes.
Firstly, create the keys and certs:
```
openssl req -x509 -newkey rsa:2048 -keyout prom_node_key.pem -out prom_node_cert.pem -days 29220 -nodes -subj /commonName=prom_node/ -addext "subjectAltName=DNS:prom_node"
openssl req -x509 -newkey rsa:2048 -keyout prometheus_key.pem -out prometheus_cert.pem -days 29220 -nodes -subj /commonName=prometheus/ -addext "subjectAltName=DNS:prometheus"
```
Create an `/etc/prometheus/ssl/` directory on the prometheus server and all
the remote nodes. Install both cert.pem files everywhere. It is safe for
them to be world-readable.
Install `prom_node_key.pem` only on the nodes, and set file permissions to
protect it so that only exporter_exporter can read it. Similarly, install
`prometheus_key.pem` only on the prometheus server, and set permissions so
that only the prometheus process can read it.
Configuration for exporter_exporter on the nodes (here it also disables
plain HTTP):
```
EXPEXP_FLAGS='-web.listen-address= -web.tls.listen-address=:9998
-web.tls.cert=/etc/prometheus/ssl/prom_node_cert.pem
-web.tls.key=/etc/prometheus/ssl/prom_node_key.pem
-web.tls.ca=/etc/prometheus/ssl/prometheus_cert.pem
-web.tls.verify'
```
To test, use `curl` to make a scrape, replacing x.x.x.x with the IP address
of the target:
```
curl --cert /etc/prometheus/ssl/prometheus_cert.pem \
--key /etc/prometheus/ssl/prometheus_key.pem \
--cacert /etc/prometheus/ssl/prom_node_cert.pem \
--resolve prom_node:9998:x.x.x.x \
-v https://prom_node:9998/proxy?module=node
```
When this is working, configure your prometheus server to use https. Example:
```
- job_name: node
scrape_interval: 1m
scrape_timeout: 50s
file_sd_configs:
- files:
- /etc/prometheus/targets.d/node_targets.yml
scheme: https
tls_config:
# Verifying remote identity
ca_file: /etc/prometheus/ssl/prom_node_cert.pem
server_name: prom_node
# Asserting our identity
cert_file: /etc/prometheus/ssl/prometheus_cert.pem
key_file: /etc/prometheus/ssl/prometheus_key.pem
metrics_path: /proxy
params:
module: [ node ]
relabel_configs:
- source_labels: [__address__]
target_label: instance
- source_labels: [__address__]
regex: '([^:]+)'
target_label: __address__
replacement: '${1}:9998'
```
Example `/etc/prometheus/targets.d/node_targets.yml`:
```
- labels: []
targets:
- 192.0.2.1
- 192.0.2.2
```