{"id":20333963,"url":"https://github.com/braedon/prometheus-es-exporter","last_synced_at":"2025-04-04T07:04:58.447Z","repository":{"id":3468200,"uuid":"49691886","full_name":"braedon/prometheus-es-exporter","owner":"braedon","description":"Prometheus Elasticsearch Exporter","archived":false,"fork":false,"pushed_at":"2024-06-30T19:15:29.000Z","size":199,"stargazers_count":251,"open_issues_count":18,"forks_count":90,"subscribers_count":15,"default_branch":"master","last_synced_at":"2025-03-28T06:09:57.039Z","etag":null,"topics":["elasticsearch","prometheus","prometheus-exporter"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/braedon.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2016-01-15T02:45:32.000Z","updated_at":"2025-03-04T23:05:40.000Z","dependencies_parsed_at":"2024-12-13T16:09:06.931Z","dependency_job_id":"a542f49f-d093-4003-8c6b-3b22fff6e241","html_url":"https://github.com/braedon/prometheus-es-exporter","commit_stats":{"total_commits":181,"total_committers":13,"mean_commits":"13.923076923076923","dds":"0.16022099447513816","last_synced_commit":"f656788638fa1d856ef013915df7e6efd8b75f5e"},"previous_names":[],"tags_count":34,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/braedon%2Fprometheus-es-exporter","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/braedon%2Fprometheus-es-exporter/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/braedon%2Fprometheus-es-exporter/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/braedon%2Fprometheus-es-exporter/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/braedon","download_url":"https://codeload.github.com/braedon/prometheus-es-exporter/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":247135141,"owners_count":20889420,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["elasticsearch","prometheus","prometheus-exporter"],"created_at":"2024-11-14T20:35:25.341Z","updated_at":"2025-04-04T07:04:58.427Z","avatar_url":"https://github.com/braedon.png","language":"Python","funding_links":[],"categories":["Elasticsearch"],"sub_categories":["Tools"],"readme":"Prometheus Elasticsearch Exporter\n====\nThis Prometheus exporter collects metrics from queries run on an Elasticsearch cluster's data, and metrics about the cluster itself.\n\n[Source Code](https://github.com/braedon/prometheus-es-exporter) | [Python Package](https://pypi.org/project/prometheus-es-exporter) | [Docker Image](https://hub.docker.com/r/braedon/prometheus-es-exporter) | [Helm Chart](https://braedon.github.io/helm/prometheus-es-exporter)\n\n## Query Metrics\nThe exporter periodically runs configured queries against the Elasticsearch cluster and exports the results as Prometheus gauge metrics.\n\nValues are parsed out of the Elasticsearch results automatically, with the path through the JSON to the value being used to construct metric names.\n\nMetrics are only extracted from aggregation results, with the exception of the query `hits.total` count (exposed as `hits`) and `took` time (exposed as `took_milliseconds`). The keys of any buckets are converted to labels, rather than being inserted into the metric name.\n\n### Supported Aggregations\nA limited set of aggregations are explicitly supported with tests. See [tests/test_parser.py](tests/test_parser.py) for example queries using these aggregations, and the metrics they produce. Most other aggregations should also work, so long as their result format is similar in structure to one of the explicitly supported aggregations.\n\nIf you would like to use a particular aggregation but it is not working correctly (and it isn't explicitly unsupported), please raise an issue or PR.\n\n### Unsupported Aggregations\nSome aggregations are explicitly unsupported - they don't work correctly, and this can't/won't be fixed for some reason.\n\n#### `top_hits`\nThe `top_hits` aggregation returns documents, not metrics about documents. Extracting metrics from arbitrary documents is out of scope for this exporter due to the complexities involved.\n\n## Cluster Metrics\nThe exporter queries the Elasticsearch cluster's `_cluster/health`, `_nodes/stats`, and `_stats` endpoints whenever its metrics endpoint is called, and exports the results as Prometheus gauge metrics.\n\nEndpoint responses are parsed into metrics as generically as possible so that (hopefully) all versions of Elasticsearch (past and future) can be reasonably supported with the same code. This results in less than ideal metrics in some cases - e.g. redundancy between some metrics, no distinction between gauges and counters (everything's a gauge). If you spot something you think can be reasonably improved let me know via a Github issue (or better yet - a PR).\n\nSee [tests/test_cluster_health_parser.py](tests/test_cluster_health_parser.py), [tests/test_nodes_stats_parser.py](tests/test_nodes_stats_parser.py), and [tests/test_indices_stats_parser.py](tests/test_indices_stats_parser.py) for examples of responses and the metrics produced.\n\nThe exporter also produces the following metrics:\n\n### `es_indices_aliases_alias{index, alias}` (gauge)\nIndex aliases and the indices they point to. Parsed from the `_alias` endpoint.\n\nMetrics for current aliases have the value `1`. If an alias is removed, or changes the index it points too, old metrics will be dropped immediately.\n\nNote that Prometheus may [keep returning dropped metrics for a short period](https://prometheus.io/docs/prometheus/latest/querying/basics/#staleness) (usually 5 minutes). In the future, old metrics may return `0` for a period before being dropped to mitigate this issue. As such, it's best to check the metric value, not just its existence.\n\nFilter other index metrics by alias using the `and` operator and this metric, e.g.:\n```\nes_indices_mappings_field_count and on(index) (es_indices_aliases_alias{alias=\"logstash-today\"} == 1)\n```\n\n### `es_indices_mappings_field_count{index, field_type}` (gauge)\nThe number of mapped fields of a given type in an index. Parsed from the `_mappings` endpoint.\n\nSum by index to calculate the total fields per index. Useful for checking if an index is at risk of reaching a field limit, e.g.\n```\nsum(es_indices_mappings_field_count) by(index) \u003e 900\n```\nNote that these counts don't include system fields (ones prefixed with `_`, e.g. `_id`), so may be slightly lower than the field count used by Elasticsearch to check the field limit.\n\n# Installation\nThe exporter requires Python 3 and Pip 3 to be installed.\n\nTo install the latest published version via Pip, run:\n```bash\n\u003e pip3 install prometheus-es-exporter\n```\nNote that you may need to add the start script location (see pip output) to your `PATH`.\n\n# Usage\nOnce installed, you can run the exporter with the `prometheus-es-exporter` command.\n\nBy default, it will bind to port 9206, query Elasticsearch on `localhost:9200` and run queries configured in a file `exporter.cfg` in the working directory. You can change these defaults as required by passing in options:\n```bash\n\u003e prometheus-es-exporter -p \u003cport\u003e -e \u003celasticsearch nodes\u003e -c \u003cpath to query config file\u003e\n```\nRun with the `-h` flag to see details on all the available options.\n\nNote that all options can be set via environment variables. The environment variable names are prefixed with `ES_EXPORTER`, e.g. `ES_EXPORTER_BASIC_USER=fred` is equivalent to `--basic-user fred`. CLI options take precedence over environment variables.\n\nCommand line options can also be set from a configuration file, by passing `--config FILE`. The format of the file should be [Configobj's unrepre mode](https://configobj.readthedocs.io/en/latest/configobj.html#unrepr-mode), so instead of `--basic-user fred` you could use a configuration file `config_file` with `basic-user=\"fred\"` in it, and pass `--config config_file`. CLI options and environment variables take precedence over configuration files.\n\nCLI options, environment variables, and configuration files all override any default options. The full resolution order for a given option is: CLI \u003e Environment \u003e Configuration file \u003e Default.\n\nSee the provided [exporter.cfg](exporter.cfg) file for query configuration examples and explanation.\n\n# Docker\nDocker images for released versions can be found on Docker Hub (note that no `latest` version is provided):\n```bash\n\u003e sudo docker pull braedon/prometheus-es-exporter:\u003cversion\u003e\n```\nTo run a container successfully, you will need to mount a query config file to `/usr/src/app/exporter.cfg` and map container port 9206 to a port on the host. Any options placed after the image name (`prometheus-es-exporter`) will be passed to the process inside the container. For example, you will need to use this to configure the elasticsearch node(s) using `-e`.\n```bash\n\u003e sudo docker run --rm --name exporter \\\n    -v \u003cpath to query config file\u003e:/usr/src/app/exporter.cfg \\\n    -p \u003chost port\u003e:9206 \\\n    braedon/prometheus-es-exporter:\u003cversion\u003e -e \u003celasticsearch nodes\u003e\n```\nIf you don't want to mount the query config file in at run time, you could extend an existing image with your own Dockerfile that copies the config file in at build time.\n\n# Helm\nA Helm chart is available from the Helm repo at [https://braedon.github.io/helm](https://braedon.github.io/helm/).\n```bash\n\u003e helm repo add braedon https://braedon.github.io/helm\n\u003e helm repo update\n\n\u003e helm install braedon/prometheus-es-exporter --name \u003crelease name\u003e \\\n                                              --set elasticsearch.cluster=\u003celasticsearch nodes\u003e \\\n                                              --set image.tag=\u003cimage tag\u003e\n```\nSee the [`prometheus-es-exporter` chart README](https://braedon.github.io/helm/prometheus-es-exporter/) for more details on how to configure the chart.\n\n# Development\nTo install directly from the git repo, run the following in the root project directory:\n```bash\n\u003e pip3 install .\n```\nThe exporter can be installed in \"editable\" mode, using pip's `-e` flag. This allows you to test out changes without having to re-install.\n```bash\n\u003e pip3 install -e .\n```\nTo run tests (as usual, from the root project directory), use:\n```bash\n\u003e python3 -m unittest\n```\nNote that these tests currently only cover the response parsing functionality - there are no automated system tests as of yet.\n\nTo build a docker image directly from the git repo, run the following in the root project directory:\n```bash\n\u003e sudo docker build -t \u003cyour repository name and tag\u003e .\n```\n\nTo develop in a docker container, first build the image, and then run the following in the root project directory:\n```bash\n\u003e sudo docker run --rm -it --name exporter --entrypoint bash -v $(pwd):/usr/src/app \u003cyour repository name and tag\u003e\n```\nThis will mount all the files inside the container, so editing tests or application code will be synced live. You can run the tests with `python -m unittest`. You may need to run `pip install -e .` again after running the container if you get an error like\n```\npkg_resources.DistributionNotFound: The 'prometheus-es-exporter' distribution was not found and is required by the application\n```\n\nSend me a PR if you have a change you want to contribute!\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fbraedon%2Fprometheus-es-exporter","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fbraedon%2Fprometheus-es-exporter","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fbraedon%2Fprometheus-es-exporter/lists"}