Ecosyste.ms: Awesome

An open API service indexing awesome lists of open source software.

Awesome Lists | Featured Topics | Projects

https://github.com/peekjef72/exalead_exporter

Exalead Prometheus exporter
https://github.com/peekjef72/exalead_exporter

Last synced: about 2 months ago
JSON representation

Exalead Prometheus exporter

Host: GitHub
URL: https://github.com/peekjef72/exalead_exporter
Owner: peekjef72
Created: 2022-05-09T17:11:57.000Z (over 2 years ago)
Default Branch: master
Last Pushed: 2022-10-23T16:53:13.000Z (over 2 years ago)
Last Synced: 2024-10-28T17:37:58.721Z (3 months ago)
Language: Python
Size: 581 KB
Stars: 0
Watchers: 1
Forks: 0
Open Issues: 0
Metadata Files:
- Readme: README.md
- Changelog: CHANGELOG.md

Awesome Lists containing this project

README

# exalead_exporter

## Description
Prometheus exporter for Exalead solution

This exporter collect metrics from Exalead HTTP API.

It is a python HTTP server that exposes metrics to http (default port 9271) that can be then scrapped by [Prometheus](https://github.com/prometheus).

![exporter_diagram](screenshots/exalead_licences.png)

Several Exalead servers can be polled by adding them to the YAML config file, by adding a host section:

**Config**: (see config.yml)

```yaml
exaleads:
- host: host.domain
port: 9398
# protocol: https
# verify_ssl: false
# timeout: 20
# keep_session: true # default
# default_labels:
# - name: veeam_em
# value: my_veeam_em_server.domain
# proxy:
# url: http://my.proxy.domain:port/
# protocol: https

weblisten:
address: 0.0.0.0
port: 9247

logger:
level: info
facility: syslog

metrics_file: "metrics/*_metrics.yml"
```

## Usage

The exporter may run as a unix command with module installation or as standalone python script without instalation.
Usage as a system command

the easiest way is to install from pip:

```shell
pip3 install --upgrade exalead-exporter
```

then you can use the entry point create by the installer of the module in /usr/local/bin/exalead_exporter or in [venv]/bin/exalead_exporter for venv context.
The commanded usage is in venv.

Usage as a Python Script

To use the exporter, few packages needs to be installed. This can be done using:

```shell
pip3 install -r pip_requirements.txt
```

Contents of requirements.txt

```python
xmltodict==0.12.0
tenacity==6.2.0
requests>=2.20.0
Jinja2==3.0.3
urllib3==1.24.2
prometheus_client==0.14.1
PyYAML>=5.3.1
python-dateutil>=2.7.0
```

+ Consider, to extract the archiv file in /tmp folder; this will generate a folder /tmp/exalead_exporter_[version].
+ create a directory where you want by example /opt/exalead_exporter_[version],
+ move the /tmp/exalead_exporter_[version]/exalead_exporter_package directory to /opt/eeam_exporter_[version]
+ create a command file to launch the exporter in dir /opt/exalead_exporter_[version]
```shell
vi /opt/exalead_exporter_X.Y.Z/exalead_exporter_cmd
#!/usr/libexec/platform-python
# -*- coding: utf-8 -*-
import re
import sys

from exalead_exporter.exalead_exporter import main

if __name__ == '__main__':
sys.argv[0] = re.sub(r'(-script\.pyw?|\.exe)?$', '', sys.argv[0])
sys.exit(main())

```
+ Then edit the conf/config.yml file and add your settings.
+ Try your config by executing the command in try mode

example with the default dumy config file:
```shell
python3 exalead_exporter_cmd -n -v

exalead_exporter[227114]: level=INFO - exalead_exporter 0.0.2 starting....
exalead_exporter[227114]: level=DEBUG - config is {'exaleads': [{'host': 'host.domain', 'port': 9398, 'verify_ssl': False, 'timeout': 20}], 'weblisten': {'address': '0.0.0.0', 'port': 9271}, 'logger': {'level': 'info', 'facility': 'syslog'}, 'metrics_file': 'metrics/*.yml'}
exalead_exporter[227114]: level=WARNING - [Errno 2] No such file or directory: '/home/users/d107684/python/exalead-exporter-package/exalead_exporter/metrics'
exalead_exporter[227114]: level=ERROR - no metrics found
exalead_exporter[227114]: level=INFO - exalead_exporter 0.0.2 stopped.
[ /home/users/d107684/python/exalead-exporter-package ]d107684@dal-v-survdadc $ python3 cli.py -v -n
exalead_exporter[228511]: level=INFO - exalead_exporter 0.0.2 starting....
exalead_exporter[228511]: level=DEBUG - config is {'exaleads': [{'host': 'host.domain', 'port': 9398, 'verify_ssl': False, 'timeout': 20}], 'weblisten': {'address': '0.0.0.0', 'port': 9271}, 'logger': {'level': 'info', 'facility': 'syslog'}, 'metrics_file': 'conf/metrics/*.yml'}
exalead_exporter[228511]: level=ERROR - Connection Exception: Host host.domain: HTTPConnectionPool(host='host.domain', port=9398): Max retries exceeded with url: /mami/ (Caused by NewConnectionError(': Failed to establish a new connection: [Errno -2] Name or service not known',))
exalead_exporter[228511]: level=DEBUG - # HELP exalead_up probe success login status: 0 Down / 1 Up
exalead_exporter[228511]: level=DEBUG - # TYPE exalead_up gauge
exalead_exporter[228511]: level=DEBUG - exalead_up 0.0
exalead_exporter[228511]: level=INFO - exalead_exporter 0.0.2 stopped
```

## exporter command line options

to start the exporter:

```shell
./exalead_exporter &
```

By default, it will load the file config.yml to perform action.

Detail options

```shell

Usage: exalead_exporter [-h] [-b BASE_PATH] [-c CONFIG_FILE]
[-f LOGGER.FACILITY] [-l {error,warning,info,debug}]
[-o METRICS_FILE] [-m METRIC] [-n] [-t TARGET]
[-w WEB.LISTEN_ADDRESS] [-V] [-v]

collector for exalead.

optional arguments:
-h, --help show this help message and exit
-b BASE_PATH, --base_path BASE_PATH
set base directory to find default files.
-c CONFIG_FILE, --config_file CONFIG_FILE
path to config files.
-f LOGGER.FACILITY, --logger.facility LOGGER.FACILITY
logger facility (syslog or file path).
-l {error,warning,info,debug}, --logger.level {error,warning,info,debug}
logger level.
-o METRICS_FILE, --metrics_file METRICS_FILE
collect the metrics from the specified file instead of
config.
-m METRIC, --metric METRIC
collect only the specified metric name from the
metrics_file.
-n , --dry_mode collect the metrics then exit; display results to
stdout.
-t TARGET, --target TARGET
In dry_mode collect metrics on specified target.i
Default first from config file.
-w WEB.LISTEN_ADDRESS, --web.listen-address WEB.LISTEN_ADDRESS
Address to listen on for web interface and telemetry.
-V, --version display program version and exit..
-v , --verbose verbose mode; display log message to stdout.
```

To test your configuration you can launch the exporter in dry_mode:

```shell
./exalead_exporter -v -n -t host.domain
```

This command will try to connect to the 'host.domain' exaleas server with parameters specified in config.yml, expose the collected metrics, and eventually the warning or errors, then exits.

## Prometheus config

Since several exalead servers can be set in the exporter, Prometheus addresses each server by adding a target parameter in the url. The "target" must be the same (lexically) that in exporter config file.

```yaml
- job_name: "exalead"
scrape_interval: 120s
scrape_timeout: 60s
metrics_path: /metrics

static_configs:
- targets: [ exaleadhost.domain ]
labels:
environment: "PROD"
# file_sd_configs:
# - files: [ "/etc/prometheus/exalead_exp/*.yml" ]
relabel_configs:
- source_labels: [__address__]
target_label: __param_target
- source_labels: [__param_target]
target_label: instance
- target_label: __address__
replacement: "exalead-exporter-hostname.domain:9247" # The exalead exporter's real hostname.

```
## Metrics

The collected metrics are defined in separeted files positionned the folder conf/metrics.
All Values, computations, labels are defined in the metrics files, meaning that the exporter doesn't nothing internally on values. The configuration fully drive how values are rendered.

### Collected Metrics

All metrics are defined in the configuration file (conf/metrics/*.yml). You can retrive all metric names here. Most of them have help text too.
All metrics are prefixed with "exaled_".

file | domain | metrics
---- | ------ | -------
none | default | exalead_up : 0 or 1.
define if exalead server can be reached or not.
licence_status.yml | general licences elements | prefix: exalead_license_ .

licence status,

expiration,

token usage,

licence components activation.
(see [licences dashboard](screenshots/exalead_licences.png))
deploymentsStatus.yml | general indexing processes | prefix: exalead_process_.

status labeled by process name,

start_timestamp"backedup",

unexpected restart count,

loop crashing.
(see [processes dashboard](screenshots/exalead_processes.png))
connectorStatus.yml | indexation | Prefix: exalead_connectors_.
gauge by connector name:

active_documents,

added,

deleted,

failed deleted,

indexed documents,

partial update,

replaced,

total

scan_status,

scan_retries,

scan_duration,

scan_deleted_objects,

scan_pushed_objects
(see [processes dashboard](screenshots/exalead_connectors.png))

## Extending metrics

Exported metrics, are defined the YAML config file. The value can use Jinja2 templating language. The format of the configuration is inspired from Ansible task representation.
So a metric configuration file, consists in a list of action to perform.

There are five possible actions:

- url: to collect metrics from HTTP API
- set_fact: to assign vlaue to variables
- actions: to perform a list of (sub-)actions
- metrics: to define metrics to expose/return to Prometheus
- debug: to display debug text to logger.

All actions have default "attributes":

- name: the name of action or metric counter for metrics action.
- vars: to set vars to global symbols' table.
- with_items: to loop on current action with a list of items.
- loop_var: to set the name of the variable that will receive the current value in the loop. Default is 'item'.
- when: a list of condition (and) that must be check and be true to perform the action.

The "attributes" are analyzed in the order specified in previous table; it means that you can't use "item" var (obtained from 'with_items' directive) in the vars section because it is not yet defined when the 'vars' section is evaluated. If you need that feature, you will have to consider 'with_items' in an 'actions' section (see metrics/connectorStatus.yml).

action | parameter | description | remark
------ | ----------- | ------ | ------
url | |a string that's representing the entity to collect without '/api' | http://host.domain:port/api**[url]**. e.g.: /reports/summary/overview
| var_name |the name of the variable to store the results. Default is '_root' meaning that the resulting JSON object is directly store in symbols table. |
| | |
set_fact | | list of variable to define |
| var_name: value|
| | |
metrics | | define the list of metrics to expose
| metric_prefix | a prefix to add to all metric name | final name will be [metric_prefix]_[metric_name]
'a metric' | name | the name of the metric
| help | the help message added to the metric (and displayed in grafana explorer)
| type 'gauge' or 'counter' | the type of the prometheus metric |
| value | the numeric value itself |
| labels | a list of name value pairs to qualify the metric |