https://github.com/localghost/hadoop-http-jmx-exporter
Prometheus exporter of Hadoop JMX metrics
https://github.com/localghost/hadoop-http-jmx-exporter
hadoop jmx jmx-exporter metrics prometheus-exporter
Last synced: about 2 months ago
JSON representation
Prometheus exporter of Hadoop JMX metrics
- Host: GitHub
- URL: https://github.com/localghost/hadoop-http-jmx-exporter
- Owner: localghost
- Created: 2024-03-24T11:59:00.000Z (about 1 year ago)
- Default Branch: main
- Last Pushed: 2024-03-31T07:00:21.000Z (about 1 year ago)
- Last Synced: 2024-04-15T03:35:16.772Z (about 1 year ago)
- Topics: hadoop, jmx, jmx-exporter, metrics, prometheus-exporter
- Language: Go
- Homepage:
- Size: 25.4 KB
- Stars: 0
- Watchers: 1
- Forks: 0
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
Awesome Lists containing this project
README
# hadoop-http-jmx-exporter
Scrapes [Hadoop
Metrics](https://hadoop.apache.org/docs/stable/hadoop-project-dist/hadoop-common/Metrics.html)
and exposes them in [Prometheus](https://prometheus.io/) format.
Supports both _simple_ and _SPNEGO_ authentication for the Hadoop endpoints.
> [!WARNING]
> Metrics are exposed over HTTP without any authentication.
```mermaid
flowchart LR
id1[Prometheus] --no auth--> id2[Hadoop-HTTP-JMX-Exporter]
id2[Hadoop-HTTP-JMX-Exporter] --no auth/SPNEGO--> id3[Hadoop Service]
id3[Hadoop Service]
```
## Configuration
Configuration can be provided ether via environment variable:
```bash
JMX_URLS="http://10.0.0.1:8088/jmx,http://10.0.0.1:8042/jmx"
KERBEROS_PRINCIPAL=HTTP/localhost
KERBEROS_REALM=SOME_REALM
KERBEROS_KEYTAB_PATH=/path/to/keytab/with/HTTP/principal
KERBEROS_CONFIG_PATH=/etc/krb5.conf
HTTP_CLIENT_TIMEOUT=10s # default
LISTEN_ADDRESS=0.0.0.0 # default
LISTEN_PORT=9100 # default
```
or a config YAML file passed as the only argument to the CLI:
```yaml
jmx_urls:
- http://10.0.0.1:8088/jmx
- http://10.0.0.1:8042/jmx
kerberos_principal: HTTP/localhost
kerberos_realm: SOME_REALM
kerberos_keytab_path: /path/to/keytab/with/HTTP/principal
kerberos_config_path: /etc/krb5.conf
http_client_timeout: 10s # default
listen_address: 0.0.0.0 # default
listen_port: 9100 # default
```
> [!NOTE]
> If _SPNEGO_ should not be used do not set configuration options starting with `kerberos`/`KERBEROS` prefix. The
> bare minimum is to provide `jmx_urls`/`JMX_URLS`.
## Exposed Metrics
Following [metrics](https://hadoop.apache.org/docs/stable/hadoop-project-dist/hadoop-common/Metrics.html) are currently exposed:
- JvmMetrics
- MemNonHeapUsedM
- MemNonHeapCommittedM
- MemNonHeapMaxM
- MemHeapUsedM
- MemHeapCommittedM
- MemHeapMaxM
- MemMaxM
- GcCount
- GcTimeMillis
- GcTotalExtraSleepTime
- datanode
- BytesWritten
- BytesRead
- HeartbeatsAvgTime
- ClusterMetrics
- NumLostNMs
- NumUnhealthyNMs
- NumRebootedNMs
- NodeManagerMetrics
- ContainersLaunched
- ContainersCompleted
- ContainersFailed
- ContainersKilled
- ContainersIniting
- ContainersRunning
- AllocatedContainers
- AllocatedGB
- AvailableGB
- AllocatedVCores
- AvailableVCores
- ContainerLaunchDuration
- NodeUsedMemGB
- NodeUsedVMemGB
- NodeCpuUtilization
- ContainerUsedMemGB
- ContainerUsedVMemGB
## Releases
Releases are acommpanied by a Docker image available
[here](https://hub.docker.com/r/zkostrzewa/hadoop-http-jmx-exporter).