{"id":14110335,"url":"https://github.com/ovh/beamium","last_synced_at":"2025-08-01T10:33:15.973Z","repository":{"id":45960573,"uuid":"77493486","full_name":"ovh/beamium","owner":"ovh","description":"Prometheus to Warp10 metrics forwarder","archived":true,"fork":false,"pushed_at":"2021-11-24T10:03:20.000Z","size":472,"stargazers_count":84,"open_issues_count":20,"forks_count":21,"subscribers_count":20,"default_branch":"master","last_synced_at":"2024-08-14T10:15:55.926Z","etag":null,"topics":["metrics","monitoring","prometheus","reliable","sensision","simple","warp10"],"latest_commit_sha":null,"homepage":"","language":"Rust","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"other","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/ovh.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":"CODE_OF_CONDUCT.md","threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2016-12-28T01:32:55.000Z","updated_at":"2024-06-08T00:06:04.000Z","dependencies_parsed_at":"2022-09-13T08:41:58.984Z","dependency_job_id":null,"html_url":"https://github.com/ovh/beamium","commit_stats":null,"previous_names":[],"tags_count":26,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ovh%2Fbeamium","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ovh%2Fbeamium/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ovh%2Fbeamium/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ovh%2Fbeamium/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/ovh","download_url":"https://codeload.github.com/ovh/beamium/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":228369183,"owners_count":17909204,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["metrics","monitoring","prometheus","reliable","sensision","simple","warp10"],"created_at":"2024-08-14T10:02:47.955Z","updated_at":"2024-12-05T20:30:37.538Z","avatar_url":"https://github.com/ovh.png","language":"Rust","funding_links":[],"categories":["Rust"],"sub_categories":[],"readme":"# Beamium - metrics scraper for Warp10 \u0026 Prometheus\n\n[![GitHub release](https://img.shields.io/github/release/ovh/beamium.svg)]()\n[![Build Status](https://travis-ci.org/ovh/beamium.svg?branch=master)](https://travis-ci.org/ovh/beamium)\n\nBeamium collect metrics from HTTP endpoints like http://127.0.0.1/metrics and supports Prometheus and Warp10/Sensision format. Once scraped, Beamium can filter and forward data to a Warp10 Time Series platform. While acquiring metrics, Beamium uses DFO (Disk Fail Over) to prevent metrics loss due to eventual network issues or unavailable service.\n\nBeamium is written in Rust to ensure efficiency, a very low footprint and deterministic performances.\n\nBeamium key points:\n - **Simple**: Beamium is a single binary that does one thing : scraping then pushing.\n - **Integration**: Beamium fetch Prometheus metrics and so benefits from a large community.\n - **Reliable**: Beamium handle network failure. Never loose data. We guarantee void proof graph ;)\n - **Versatile**: Beamium can also scrape metrics from a directory.\n - **Powerful**: Beamium is able to filter metrics and send them to multiple Warp10 platforms.\n\n## How it works?\n\nScraper (optionals) will collect metrics from defined endpoints. They will store them into the source_dir.\nBeamium will read files inside source_dir, and will fan out them according to the provided selector into sink_dir.\nFinaly Beamium will push files from the sink_dir to the defined sinks.\n\nThe pipeline can be describe this way :\n\n    HTTP /metrics endpoint --scrape--\u003e source_dir --route--\u003e sink_dir --forward--\u003e warp10\n\nIt also means that given your need, you could produce metrics directly to source/sink directory, example :\n\n    $ TS=`date +%s` \u0026\u0026 echo $TS\"000000// metrics{} T\" \u003e\u003e /opt/beamium/data/sources/prefix-$TS.metrics\n\n## Status\n\nBeamium is not under development. We are moving toward [prometheus in agent mode](https://github.com/prometheus/prometheus)\n\n## Install\n\n### Debian\n\nWe provide deb packages for Beamium!\n```\nsudo apt-get install apt-transport-https\nsudo lsb_release -a | grep Codename | awk '{print \"deb https://last-public-ovh-metrics.snap.mirrors.ovh.net/debian/ \" $2 \" main\"}' \u003e\u003e /etc/apt/sources.list.d/beamium.list\ncurl https://last-public-ovh-metrics.snap.mirrors.ovh.net/pub.key | sudo apt-key add -\nsudo apt-get update\nsudo apt-get install beamium\n```\n\n### Kubernetes\n\nWe are providing an [example yaml](deploy/kubernetes/beamium.yaml) file to deploy Beamium within Kubernetes.\n\n```bash\nkubectl apply -f deploy/kubernetes\n```\n\n## Building\nBeamium is pretty easy to build.\n - Clone the repository\n - Setup a minimal working config (see below)\n - Install rust compile tools with `curl https://sh.rustup.rs -sSf | sh`\n - Then `source ~/.cargo/env`\n - Build with `cargo build`\n - Finally, run `cargo run`\n\nIf you have already rust:\n - `cargo install --git https://github.com/ovh/beamium`\n\n## Configuration\nBeamium come with a [sample config file](config.sample.yaml). Simply copy the sample to *config.yaml*, replace `WARP10_ENDPOINT` and `WARP10_TOKEN`, launch Beamiun and you are ready to go!\n\nSince the release 2.x, Beamium will look for configuration in those directories/files:\n- /etc/beamium.d/\n- /etc/beamium/config.yaml\n- $HOME/beamium.d/\n- $HOME/beamium/config.yaml\n\nIn addition, it will recursively discover configuration files in `beamium.d` directories. Then it will merge all discovered configuration files.\n\nFurthermore, Beamium support multiple formats for configuration files which are `hjson`, `json`, `toml`, `yaml`, `yml` or `ini`.\n\n\nAlso, Beamium can be started with several labels put as env vars, they must be prefixed by `BEAMIUM_LABEL`.\n\nEx:\n```sh\nBEAMIUM_LABEL_HOST=myhost ./beamium -v\n```\n\nThis is also available per scraper\n\nEx:\n```sh\nBEAMIUM_SCRAPPER1_LABEL_HOST=myhost ./beamium -v\n```\n\n### Hot reload\n\nBeamium now supports hot reloading of his configuration. There is no specific thing to do to enable this feature. Actually, this support all features excepted those in relation with the logger.\n\nBesides, beamium debounced file-system event in an interval of two seconds. So, it may appears that the reload of beamium is not released at the same time of the configuration.\n\n### Definitions\nConfig is composed of four parts:\n\n#### Scrapers\nBeamium can have none to many Prometheus or Warp10/Sensision endpoints. A *scraper* is defined as follow:\n``` yaml\nscrapers:                              # Scrapers definitions (Optional)\n  scraper1:                            # Source name                  (Required)\n    url: http://127.0.0.1:9100/metrics # Prometheus endpoint          (Required)\n    period: 60s                        # Polling interval             (Required)\n    format: prometheus                 # Polling format               (Optional, default: prometheus, value: [prometheus, sensision])\n    labels:                            # Labels definitions           (Optional)\n      label_name: label_value          # Label definition             (Required)\n      another: env:USER                # label values can be resolved from env vars\n    filtered_labels:                   # filtered labels              (optional)\n      - jobid                          # key label which is removed   (required)\n    metrics:                           # filter fetched metrics       (optional)\n      - node.*                         # regex used to select metrics (required)\n    headers:                           # Add custom header on request (Optional)\n      X-Toto: tata                     # list of headers to add       (Optional)\n      Authorization: Basic XXXXXXXX\n    pool: 1                            # Number of threads allocated for the scraper (Optionnal)\n```\n\n#### Sinks\nBeamium can have none to many Warp10 endpoints. A *sink* is defined as follow:\n``` yaml\nsinks: # Sinks definitions (Optional)\n  source1:                             # Sink name                                (Required)\n    url: https://warp.io/api/v0/update # Warp10 endpoint                          (Required)\n    token: mywarp10token               # Warp10 write token                       (Required)\n    token-header: X-Custom-Token       # Warp10 token header name                 (Optional, default: X-Warp10-Token)\n    selector: metrics.*                # Regex used to filter metrics             (Optional, default: None)\n    ttl: 1h                            # Discard file older than ttl              (Optional, default: 3600)\n    size: 100Gb                        # Discard old file if sink size is greater (Optional, default: 1073741824)\n    parallel: 1                        # Send parallelism                         (Optional, default: 1)\n    keep-alive: 1                      # Use keep alive                           (Optional, default: 1)\n```\n\n#### Labels\nBeamium can add static labels to collected metrics. A *label* is defined as follow:\n``` yaml\nlabels: # Labels definitions (Optional)\n  label_name: label_value # Label definition             (Required)\n  another: env:USER       # label values can be resolved from env vars\n```\n\n#### Parameters\nBeamium can be customized through parameters. See available parameters bellow:\n``` yaml\nparameters: # Parameters definitions                                                                  (Optional)\n  source-dir: sources     # Beamer data source directory                                                  (Optional, default: sources)\n  sink-dir: sinks         # Beamer data sink directory                                                    (Optional, default: sinks)\n  scan-period: 1s         # Delay(ms) between source/sink scan                                            (Optional, default: 1000)\n  batch-count: 250        # Maximum number of files to process in a batch                                 (Optional, default: 250)\n  batch-size: 2Kb         # Maximum batch size                                                            (Optional, default: 200000)\n  log-file: beamium.log   # Log file                                                                      (Optional, default: beamium.log)\n  log-level: 4            # Log level                                                                     (Optional, default: info)\n  timeout: 500            # Http timeout                                                                  (Optional, default: 500)\n  router-parallel: 1      # Routing threads                                                               (Optional, default: 1)\n  metrics: 127.0.0.1:9110 # Open a server on the given address and expose a prometheus /metrics endpoint  (Optional, default: none)\n  filesystem-threads: 100 # Set the maximum number of threads use for blocking treatment per scraper, sink and router (Optional, default: 100)\n  backoff:                # Backoff configuration - slow down push on errors                              (Optional)\n    initial: 500ms          # Initial interval                                                              (Optional, default: 500ms)\n    max: 1m                 # Max interval                                                                  (Optional, default: 1m)\n    multiplier: 1.5         # Interval multiplier                                                           (Optional, default: 1.5)\n    randomization: 0.3      # Randomization factor - delay = interval * 0.3                                 (Optional, default: 0.3)\n```\n\n#### Test\nIn order to know if the configuration is healthy, you can use the following command:\n```bash\n$ beamium -t [--config \u003c/path/to/file\u003e]\n```\n\nThis will output if the configuration is healthy.\n\nTo print the configuration you can use the `-v` flag.\n\n```bash\n$ beamium -t -vvvvv [--config \u003c/path/to/file\u003e]\n```\n\nThis will output if the configuration is healthy and the configuration loaded.\n\n## Metrics\nBeamium can expose metrics about his usage:\n\n| name                     | labels       | type    | description                      |\n| ------------------------ | ------------ | ------- | -------------------------------- |\n| beamium_directory_files  | directory    | gauge   | Number of files in the directory |\n| beamium_fetch_datapoints | scraper      | counter | Number of datapoints fetched     |\n| beamium_fetch_errors     | scraper      | counter | Number of fetch errors           |\n| beamium_push_datapoints  | sink         | counter | Number of datapoints pushed      |\n| beamium_push_http_status | sink, status | counter | Push response http status code   |\n| beamium_push_errors      | sink         | counter | Number of push error             |\n| beamium_reload_count     |              | counter | Number of global reloads         |\n\n## Contributing\nInstructions on how to contribute to Beamium are available on the [Contributing][Contributing] page.\n\n## Get in touch\n\n- Twitter: [@notd33d33](https://twitter.com/notd33d33)\n\n[Contributing]: CONTRIBUTING.md\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fovh%2Fbeamium","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fovh%2Fbeamium","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fovh%2Fbeamium/lists"}