{"id":16900783,"url":"https://github.com/benoit74/benchmarking","last_synced_at":"2026-05-04T20:37:17.435Z","repository":{"id":181824813,"uuid":"667406819","full_name":"benoit74/benchmarking","owner":"benoit74","description":"A small benchmarking stack to monitor a host with Docker containers","archived":false,"fork":false,"pushed_at":"2023-11-28T13:57:42.000Z","size":563,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-01-25T13:11:12.979Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"gpl-3.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/benoit74.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2023-07-17T12:38:49.000Z","updated_at":"2023-09-05T14:44:58.000Z","dependencies_parsed_at":"2023-11-28T14:56:27.974Z","dependency_job_id":null,"html_url":"https://github.com/benoit74/benchmarking","commit_stats":null,"previous_names":["benoit74/benchmarking"],"tags_count":1,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/benoit74%2Fbenchmarking","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/benoit74%2Fbenchmarking/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/benoit74%2Fbenchmarking/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/benoit74%2Fbenchmarking/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/benoit74","download_url":"https://codeload.github.com/benoit74/benchmarking/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":244618535,"owners_count":20482331,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-10-13T17:55:57.584Z","updated_at":"2026-05-04T20:37:12.404Z","avatar_url":"https://github.com/benoit74.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Benchmarking\n\n## Purpose\n\nThis project is a benchmarking solution to:\n- record system and docker metrics on a node (named `monitored_node`) \n  - this node has limited computing / memory (and maybe no LAN / Internet connectivity)\n- exploit those metrics on another node (named `analysis_node`) with dashboards for manual interpretation\n\n![Sample Host dashboard](host_dashboard.png)\n\n![Sample Docker dashboard](docker_dashboard.png)\n\nMany use-case are possible, our first one was to record monitoring data on a Pi 3 (`monitored_node`)\nrunning our offpost, while performing various activities on the web interface (browse Zims, ...).\nSince the Pi 3 has limited computing power, we need a stack which is lightweight and capable to\nrecord data for further analysis on any other machine with bigger CPU / RAM (`analysis_node`). This\nis made even more important by the fact that we don't wan't to assume that the Pi has LAN connectivity\nsince we might need to use this benchmarking stack on the field.\n\n\n## Overview\n\nElastic `metricbeat` has been selected to record system and docker activity on the `monitoring_node`. It has the advantage\nto be able to output recorded data in `ndjson` files, which can be saved on local storage for persistence and transfered afterwards to the `analysis_node`. It is also very efficient in terms of computing / memory requirements.\n\nWe had to compliment the `monitored_node` with a `compactor` which is responsible to compact rotated\n`ndjson` files. Elastic `metricbeat` is capable to rotate files but does not compact rotated ones automatically.\nThese files are however very good candidates for compaction, achieving up to x10 compaction rate with default\nGZip compaction.\n\nOn the `analysis_node` side, `elasticsearch` and `kibana` are the natural solution to easily exploit `metricbeat` data. Dashboards are ready to use to interpret most `metricbeat` datasets, and feeding `ndjson` files into `elasticsearch` is almost straightforward.\n\n## Howto\n\nThis section describes how to use the stack.\n\n### Retrieve source code\n\nRetrieve this repo source code on `monitored_node` and `analysis_node` via git clone, curl a zip release, ...\n\nIf `monitored_node` has no Internet access, it is not straightforward. In our case, we simply get the SD Card from the Pi Zero and plug it to another Pi (3/4/...) with Internet access. Then we retrieve what is needed (**do not forget also to pull Docker images**) and re-plug the SD Card back into the Pi Zero.\n\n### Start the stack on `monitored_node`\n\nRun the docker-compose stack:\n```\ncd monitored_node\ndocker compose -p bench up -d\n```\n\nBy default, Metricbeat recorded data will be placed in `monitored_node/output`, and compressed by `compactor` in an `monitored_node/output/compressed`. Feel free to change this in `monitored_node/docker-compose.yml`.\n\n### Add systemctl services\n\nIf you want the stack to restart everytime the host starts, you might add systemctl services:\n\n```\ncp systemctl/benchmarking-compressor.service /etc/systemd/system/benchmarking-compressor.service\ncp systemctl/benchmarking-metricbeat.service /etc/systemd/system/benchmarking-metricbeat.service\n```\n\nStart them:\n```\nsystemctl start benchmarking-compressor.service\nsystemctl start benchmarking-metricbeat.service\n```\n\nCheck their status:\n```\nsystemctl status benchmarking-compressor.service\nsystemctl status benchmarking-metricbeat.service\n```\n\nEnable them for execution at every startup:\n```\nsystemctl enable benchmarking-compressor.service\nsystemctl enable benchmarking-metricbeat.service\n```\n\n### Stop the stack on `monitored_node`\n\nOnce monitoring is finished, stop the docker-compose stack:\n```\ncd monitored_node\ndocker compose -p bench down\n```\n\nor if you have installed systemctl services:\n\n```\nsystemctl stop benchmarking-compressor.service\nsystemctl stop benchmarking-metricbeat.service\n```\n\n### Start the stack on `analysis_node`\n\nChange to appropriate directory:\n```\ncd analysis_node\n```\n\nAdapt environment variables in `analysis_node/.env` (e.g. passwords).\n\nLoad it into your shell session:\n```\nset -a; source .env; set +a \n```\n\nRun the docker-compose stack:\n```\ndocker compose -p bench up -d\n```\n\nThis will start:\n- `es01`: an elasticsearch node, accessible on http://localhost:9200\n- `kibana`: a kibana node, accessible on http://localhost:5601 (use the `elastic` user with appropriate password)\n- some setup containers:\n  - `manual-setup`: container responsible to create the `kibana_system` user and password\n  - `metricbeat-setup`: container responsible to setup metricbeat datastream and index template in ElasticSearch and dashboards in Kibana (this is done via metricbeat source code)\n  - `wait-for-me`: _hack_ container to not consider that docker-compose stack is up until all others containers are healthy or succesfully completed\n- `uploader`: a helper container which will upload metricbeat data placed in the `analysis_node/input` folder to elasticsearch through the appropriate datastream\n\n### Upload data\n\nOnce the stack is up and running, you have to load data from `monitoring_node` to Elasticsearch. Simply retrieve this data from `monitored_node/output` and copy it to the `analysis_node/input` folder. After at most 10 secs, `uploader` will start to push it to Elasticsearch. Everytime a file is uploaded, a companion file with the `.done` suffix is created, so that this file is not processed again.\n\n### Investigate dashboards\n\nYou can now investigate monitoring data through Kibana dashboards on http://localhost:5601 (use the `elastic` user with appropriate password).\n\nMost usefull dashboard with default `metricbeat` configuration from this repo are:\n- `[Metricbeat System] Overview ECS`: if you have multiple hosts that have been monitored, you will be able to focus on only one host\n- `[Metricbeat System] Host overview ECS`: view details about one host activity\n- `[Metricbeat Docker] Overview ECS`: view details about Docker containers activity\n\n### Stop the stack on `analysis_node`\n\nOnce monitoring is finished, stop the docker-compose stack:\n```\ncd analysis_node\ndocker compose -p bench down\n```\n\nor if you have installed systemctl services:\n\n```\nsystemctl stop benchmarking-compressor.service\nsystemctl stop benchmarking-metricbeat.service\n```\n\n### Delete all Docker data on `analysis_node`\n\n```\ndocker volume rm bench_esdata01\ndocker volume rm bench_kibanadata\n```","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fbenoit74%2Fbenchmarking","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fbenoit74%2Fbenchmarking","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fbenoit74%2Fbenchmarking/lists"}