Ecosyste.ms: Awesome
An open API service indexing awesome lists of open source software.
https://github.com/pambrose/prometheus-proxy
Prometheus Proxy
https://github.com/pambrose/prometheus-proxy
grpc kotlin prometheus prometheus-proxy
Last synced: 3 months ago
JSON representation
Prometheus Proxy
- Host: GitHub
- URL: https://github.com/pambrose/prometheus-proxy
- Owner: pambrose
- License: apache-2.0
- Created: 2017-04-08T04:45:11.000Z (almost 8 years ago)
- Default Branch: master
- Last Pushed: 2024-08-07T20:16:26.000Z (6 months ago)
- Last Synced: 2024-08-07T23:29:47.962Z (6 months ago)
- Topics: grpc, kotlin, prometheus, prometheus-proxy
- Language: Kotlin
- Homepage:
- Size: 37.4 MB
- Stars: 142
- Watchers: 5
- Forks: 23
- Open Issues: 6
-
Metadata Files:
- Readme: README.md
- License: License.txt
Awesome Lists containing this project
README
# Prometheus Proxy
[](https://jitpack.io/#pambrose/prometheus-proxy)
[](https://app.travis-ci.com/pambrose/prometheus-proxy)
[](https://codebeat.co/projects/github-com-pambrose-prometheus-proxy-master)
[](https://app.codacy.com/gh/pambrose/prometheus-proxy/dashboard)
[](https://codecov.io/gh/pambrose/prometheus-proxy)
[](https://coveralls.io/github/pambrose/prometheus-proxy?branch=master)
[](https://kotlinlang.org/)
[](https://pinterest.github.io/ktlint/)
[Prometheus](https://prometheus.io) is an excellent systems monitoring and alerting toolkit, which uses a pull model for
collecting metrics. The pull model is problematic when a firewall separates a Prometheus server and its metrics
endpoints.
[Prometheus Proxy](https://github.com/pambrose/prometheus-proxy) enables Prometheus to reach metrics endpoints running
behind a firewall and preserves the pull model.
The `prometheus-proxy` runtime comprises 2 services:
* `proxy`: runs in the same network domain as Prometheus server (outside the firewall) and proxies calls from Prometheus
to the `agent` behind the firewall.
* `agent`: runs in the same network domain as all the monitored hosts/services/apps (inside the firewall). It maps the
scraping queries coming from the `proxy` to the actual `/metrics` scraping endpoints of the hosts/services/apps.
Here's a simplified network diagram of how the deployed `proxy` and `agent` work:
Endpoints running behind a firewall require a `prometheus-agent` (the agent) to be run inside the firewall. An agent can
run as a stand-alone server, embedded in another java server, or as a java agent. Agents connect to
a `prometheus-proxy` (the proxy) and register the paths for which they will provide data. One proxy can work one or many
agents.
## Requirements
Requires Java 11 or newer.
## CLI Usage
Download the proxy and agent uber-jars from [here](https://github.com/pambrose/prometheus-proxy/releases).
Start a `proxy` with:
```bash
java -jar prometheus-proxy.jar
```
Start an `agent` with:
```bash
java -jar prometheus-agent.jar -Dagent.proxy.hostname=mymachine.local --config https://raw.githubusercontent.com/pambrose/prometheus-proxy/master/examples/myapps.conf
```
If the prometheus-proxy were running on a machine named *mymachine.local* and the
`agent.pathConfigs` value in
the [myapps.conf](https://raw.githubusercontent.com/pambrose/prometheus-proxy/master/examples/myapps.conf)
config file had the contents:
```hocon
agent {
pathConfigs: [
{
name: "App1 metrics"
path: app1_metrics
url: "http://app1.local:9100/metrics"
},
{
name: "App2 metrics"
path: app2_metrics
url: "http://app2.local:9100/metrics"
},
{
name: "App3 metrics"
path: app3_metrics
url: "http://app3.local:9100/metrics"
}
]
}
```
then the *prometheus.yml* scrape_config would target the three apps with:
* http://mymachine.local:8080/app1_metrics
* http://mymachine.local:8080/app2_metrics
* http://mymachine.local:8080/app3_metrics
If the endpoints were restricted with basic auth/bearer authentication, you could either include the basic-auth credentials in the URL with: `http://user:pass@hostname/metrics` or they could be configured with `basic_auth`/`bearer_token` in the scrape-config.
The `prometheus.yml` file would include:
```yaml
scrape_configs:
- job_name: 'app1 metrics'
metrics_path: '/app1_metrics'
bearer_token: 'eyJ....hH9rloA'
static_configs:
- targets: [ 'mymachine.local:8080' ]
- job_name: 'app2 metrics'
metrics_path: '/app2_metrics'
basic_auth:
username: 'user'
password: 's3cr3t'
static_configs:
- targets: [ 'mymachine.local:8080' ]
- job_name: 'app3 metrics'
metrics_path: '/app3_metrics'
static_configs:
- targets: [ 'mymachine.local:8080' ]
```
## Docker Usage
The docker images are available via:
```bash
docker pull pambrose/prometheus-proxy:1.22.0
docker pull pambrose/prometheus-agent:1.22.0
```
Start a proxy container with:
```bash
docker run --rm -p 8082:8082 -p 8092:8092 -p 50051:50051 -p 8080:8080 \
--env ADMIN_ENABLED=true \
--env METRICS_ENABLED=true \
pambrose/prometheus-proxy:1.22.0
```
Start an agent container with:
```bash
docker run --rm -p 8083:8083 -p 8093:8093 \
--env AGENT_CONFIG='https://raw.githubusercontent.com/pambrose/prometheus-proxy/master/examples/simple.conf' \
pambrose/prometheus-agent:1.22.0
```
Using the config
file [simple.conf](https://raw.githubusercontent.com/pambrose/prometheus-proxy/master/examples/simple.conf), the proxy
and the agent metrics would be available from the proxy on *localhost* at:
* http://localhost:8082/proxy_metrics
* http://localhost:8083/agent_metrics
If you want to use a local config file with a docker container (instead of the above HTTP-served config file), use the
docker [mount](https://docs.docker.com/storage/bind-mounts/) option. Assuming the config file `prom-agent.conf`
is in your current directory, run an agent container with:
```bash
docker run --rm -p 8083:8083 -p 8093:8093 \
--mount type=bind,source="$(pwd)"/prom-agent.conf,target=/app/prom-agent.conf \
--env AGENT_CONFIG=prom-agent.conf \
pambrose/prometheus-agent:1.22.0
```
**Note:** The `WORKDIR` of the proxy and agent images is `/app`, so make sure to use `/app` as the base directory in the
target for `--mount` options.
## Embedded Agent Usage
If you are running a JVM-based program, you can run with the agent embedded directly in your app and not use an external
agent:
```Java
EmbeddedAgentInfo agentInfo = startAsyncAgent("configFile.conf", true);
```
## Configuration
The proxy and agent use the [Typesafe Config](https://github.com/typesafehub/config) library for configuration.
Highlights include:
* support for files in three formats: Java properties, JSON, and a human-friendly JSON
superset ([HOCON](https://github.com/typesafehub/config#using-hocon-the-json-superset))
* config files can be files or urls
* config values can come from CLI options, environment vars, Java system properties, and/or config files.
* config files can reference environment variables
All the proxy and agent properties are
described [here](https://github.com/pambrose/prometheus-proxy/blob/master/etc/config/config.conf). The only required
argument is an agent config value, which should have an `agent.pathConfigs` value.
### Proxy CLI Options
| Options | ENV VAR
Property | Default | Description |
|--------------------|----------------------------------------------------------------------------|--------------------------|---------------------------------------------|
| --config, -c | PROXY_CONFIG | | Agent config file or url |
| --port, -p | PROXY_PORT
proxy.http.port | 8080 | Proxy listen port |
| --agent_port, -a | AGENT_PORT
proxy.agent.port | 50051 | gRPC listen port for agents |
| --admin, -r | ADMIN_ENABLED
proxy.admin.enabled | false | Enable admin servlets |
| --admin_port, -i | ADMIN_PORT
proxy.admin.port | 8092 | Admin servlets port |
| --debug, -b | DEBUG_ENABLED
proxy.admin.debugEnabled | false | Enable proxy debug servlet
on admin port |
| --metrics, -e | METRICS_ENABLED
proxy.metrics.enabled | false | Enable proxy metrics |
| --metrics_port, -m | METRICS_PORT
proxy.metrics.port | 8082 | Proxy metrics listen port |
| --sd_enabled | SD_ENABLED
proxy.service.discovery.enabled | false | Service discovery endpoint enabled |
| --sd_path | SD_PATH
proxy.service.discovery.path | "discovery" | Service discovery endpoint path |
| --sd_target_prefix | SD_TARGET_PREFIX
proxy.service.discovery.targetPrefix | "http://localhost:8080/" | Service discovery target prefix |
| --tf-disabled | TRANSPORT_FILTER_DISABLED
proxy.transportFilterDisabled | false | Transport filter disabled |
| --cert, -t | CERT_CHAIN_FILE_PATH
proxy.tls.certChainFilePath | | Certificate chain file path |
| --key, -k | PRIVATE_KEY_FILE_PATH
proxy.tls.privateKeyFilePath | | Private key file path |
| --trust, -s | TRUST_CERT_COLLECTION_FILE_PATH
proxy.tls.trustCertCollectionFilePath | | Trust certificate collection file path |
| --version, -v | | | Print version info and exit |
| --usage, -u | | | Print usage message and exit |
| -D | | | Dynamic property assignment |
### Agent CLI Options
| Options | ENV VAR
Property | Default | Description |
|:-------------------|:---------------------------------------------------------------------------|:--------|:-----------------------------------------------------------|
| --config, -c | AGENT_CONFIG | | Agent config file or url (required) |
| --proxy, -p | PROXY_HOSTNAME
agent.proxy.hostname | | Proxy hostname (can include :port) |
| --name, -n | AGENT_NAME
agent.name | | Agent name |
| --admin, -r | ADMIN_ENABLED
agent.admin.enabled | false | Enable admin servlets |
| --admin_port, -i | ADMIN_PORT
agent.admin.port | 8093 | Admin servlets port |
| --debug, -b | DEBUG_ENABLED
agent.admin.debugEnabled | false | Enable agent debug servlet
on admin port |
| --metrics, -e | METRICS_ENABLED
agent.metrics.enabled | false | Enable agent metrics |
| --metrics_port, -m | METRICS_PORT
agent.metrics.port | 8083 | Agent metrics listen port |
| --consolidated, -o | CONSOLIDATED
agent.consolidated | false | Enable multiple agents per registered path |
| --timeout | SCRAPE_TIMEOUT_SECS
agent.scrapeTimeoutSecs | 15 | Scrape timeout time (seconds) |
| --max_retries | SCRAPE_MAX_RETRIES
agent.scrapeMaxRetries | 0 | Scrape maximum retries (0 disables scrape retries) |
| --chunk | CHUNK_CONTENT_SIZE_KBS
agent.chunkContentSizeKbs | 32 | Threshold for chunking data to Proxy and buffer size (KBs) |
| --gzip | MIN_GZIP_SIZE_BYTES
agent.minGzipSizeBytes | 1024 | Minimum size for content to be gzipped (bytes) |
| --tf-disabled | TRANSPORT_FILTER_DISABLED
proxy.transportFilterDisabled | false | Transport filter disabled |
| --trust_all_x509 | TRUST_ALL_X509_CERTIFICATES
agent.http.enableTrustAllX509Certificates | false | Disable SSL verification for agent https endpoints |
| --cert, -t | CERT_CHAIN_FILE_PATH
agent.tls.certChainFilePath | | Certificate chain file path |
| --key, -k | PRIVATE_KEY_FILE_PATH
agent.tls.privateKeyFilePath | | Private key file path |
| --trust, -s | TRUST_CERT_COLLECTION_FILE_PATH
agent.tls.trustCertCollectionFilePath | | Trust certificate collection file path |
| --override | OVERRIDE_AUTHORITY
agent.tls.overrideAuthority | | Override authority (for testing) |
| --version, -v | | | Print version info and exit |
| --usage, -u | | | Print usage message and exit |
| -D | | | Dynamic property assignment |
Misc notes:
* If you want to customize the logging, include the java arg `-Dlogback.configurationFile=/path/to/logback.xml`
* JSON config files must have a *.json* suffix
* Java Properties config files must have a *.properties* or *.prop* suffix
* HOCON config files must have a *.conf* suffix
* Option values are evaluated in the order: CLI, environment vars, and finally config file vals
* Property values can be set as a java -D arg to or as a proxy or agent jar -D arg
* For more information about the proxy service discovery options, see the
Prometheus [documentation](https://prometheus.io/docs/prometheus/latest/http_sd/)
### Admin Servlets
These admin servlets are available when the admin servlet is enabled:
* /ping
* /threaddump
* /healthcheck
* /version
The admin servlets can be enabled with the `ADMIN_ENABLED` environment var, the `--admin` CLI option, or with the
`proxy.admin.enabled` and `agent.admin.enabled` properties.
The debug servlet can be enabled with the `DEBUG_ENABLED` environment var, the `--debug` CLI option , or with the
`proxy.admin.debugEnabled` and `agent.admin.debugEnabled` properties. The debug servlet requires that the admin servlets
are enabled. The debug servlet is at: `/debug` on the admin port.
Descriptions of the servlets are [here](http://metrics.dropwizard.io/3.2.2/manual/servlets.html). The path names can be
changed in the configuration file. To disable an admin servlet, assign its property path to "".
## Adding TLS to Agent-Proxy Connections
Agents connect to a proxy using [gRPC](https://grpc.io). gRPC supports TLS with or without mutual authentication. The
necessary certificate and key file paths can be specified via CLI args, environment variables and configuration file
settings.
The gRPC docs describe [how to set up TLS](https://github.com/grpc/grpc-java/tree/master/examples/example-tls).
The [repo](https://github.com/pambrose/prometheus-proxy/tree/master/testing/certs) includes the certificates and keys
necessary to test TLS support.
Running TLS without mutual authentication requires these settings:
* `certChainFilePath` and `privateKeyFilePath` on the proxy
* `trustCertCollectionFilePath` on the agent
Running TLS with mutual authentication requires these settings:
* `certChainFilePath`, `privateKeyFilePath` and `trustCertCollectionFilePath` on the proxy
* `certChainFilePath`, `privateKeyFilePath` and `trustCertCollectionFilePath` on the agent
### Running with TLS
Run a proxy and an agent with TLS (no mutual auth) using the included testing certs and keys with:
```bash
java -jar prometheus-proxy.jar --config examples/tls-no-mutual-auth.conf
java -jar prometheus-agent.jar --config examples/tls-no-mutual-auth.conf
```
Run a proxy and an agent docker container with TLS (no mutual auth) using the included testing certs and keys with:
```bash
docker run --rm -p 8082:8082 -p 8092:8092 -p 50440:50440 -p 8080:8080 \
--mount type=bind,source="$(pwd)"/testing/certs,target=/app/testing/certs \
--mount type=bind,source="$(pwd)"/examples/tls-no-mutual-auth.conf,target=/app/tls-no-mutual-auth.conf \
--env PROXY_CONFIG=tls-no-mutual-auth.conf \
--env ADMIN_ENABLED=true \
--env METRICS_ENABLED=true \
pambrose/prometheus-proxy:1.22.0
docker run --rm -p 8083:8083 -p 8093:8093 \
--mount type=bind,source="$(pwd)"/testing/certs,target=/app/testing/certs \
--mount type=bind,source="$(pwd)"/examples/tls-no-mutual-auth.conf,target=/app/tls-no-mutual-auth.conf \
--env AGENT_CONFIG=tls-no-mutual-auth.conf \
--env PROXY_HOSTNAME=mymachine.lan:50440 \
--name docker-agent \
pambrose/prometheus-agent:1.22.0
```
**Note:** The `WORKDIR` of the proxy and agent images is `/app`, so make sure to use `/app` as the base directory in the
target for `--mount` options.
## Scraping HTTPS Endpoints
Disable SSL verification for agent https endpoints with the `TRUST_ALL_X509_CERTIFICATES` environment var,
the `--trust_all_x509` CLI option, or the `agent.http.enableTrustAllX509Certificates` property.
## Scraping a Prometheus Instance
It's possible to scrape an existing prometheus server using the `/federate` endpoint.
This enables using the existing service discovery features already built into Prometheus.
An example config can be found in
[federate.conf](https://github.com/pambrose/prometheus-proxy/blob/master/examples/federate.conf).
## Nginx Support
To use the prometheus_proxy with nginx as a reverse proxy, disable the transport filter with the
`TRANSPORT_FILTER_DISABLED` environment var, the `--tf-disabled` CLI option, or the `agent.transportFilterDisabled`/
`proxy.transportFilterDisabled` properties. Agents and the Proxy must run with the same `transporFilterDisabled` value.
When using `transporFilterDisabled`, you will not see agent contexts immediately removed
from the proxy when agents are terminated. Instead, agent contexts will be removed from the proxy
after they age out from inactivity. The maximum age is controlled by the `proxy.internal.maxAgentInactivitySecs` value.
The default value is 1 minute.
An example nginx conf file is [here](https://github.com/pambrose/prometheus-proxy/tree/master/nginx/docker/nginx.conf)
and an example agent/proxy conf file
is [here](https://github.com/pambrose/prometheus-proxy/tree/master/nginx/nginx-proxy.conf)
## Grafana
[Grafana](https://grafana.com) dashboards for the proxy and agent
are [here](https://github.com/pambrose/prometheus-proxy/tree/master/grafana).
## Related Links
* [Prometheus.io](http://prometheus.io)
* [gRPC](http://grpc.io)
* [Typesafe Config](https://github.com/typesafehub/config)