{"id":25624000,"url":"https://github.com/metricq/metricq","last_synced_at":"2026-03-16T07:03:22.957Z","repository":{"id":54174897,"uuid":"141130512","full_name":"metricq/metricq","owner":"metricq","description":"🐼 A highly-scalable, distributed metric data processing framework based on RabbitMQ","archived":false,"fork":false,"pushed_at":"2024-09-03T22:19:50.000Z","size":861,"stargazers_count":27,"open_issues_count":13,"forks_count":4,"subscribers_count":4,"default_branch":"master","last_synced_at":"2025-03-27T21:11:10.822Z","etag":null,"topics":["cpp","library","message-broker","metrics","python","python3","rabbitmq","time-series"],"latest_commit_sha":null,"homepage":"","language":"Shell","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"bsd-3-clause","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/metricq.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null}},"created_at":"2018-07-16T11:38:15.000Z","updated_at":"2024-05-27T10:38:17.000Z","dependencies_parsed_at":"2023-02-17T08:45:40.240Z","dependency_job_id":"ab8fdbf1-c027-4d4d-a823-b340c558f1b9","html_url":"https://github.com/metricq/metricq","commit_stats":null,"previous_names":[],"tags_count":5,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/metricq%2Fmetricq","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/metricq%2Fmetricq/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/metricq%2Fmetricq/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/metricq%2Fmetricq/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/metricq","download_url":"https://codeload.github.com/metricq/metricq/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248839899,"owners_count":21169891,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["cpp","library","message-broker","metrics","python","python3","rabbitmq","time-series"],"created_at":"2025-02-22T12:17:27.049Z","updated_at":"2026-03-16T07:03:17.912Z","avatar_url":"https://github.com/metricq.png","language":"Shell","funding_links":[],"categories":[],"sub_categories":[],"readme":"![BSD 3-clause](https://img.shields.io/badge/license-BSD%203--clause-blue.svg)\n[![PyPI](https://img.shields.io/pypi/v/metricq)](https://pypi.org/project/metricq/)\n\n# metricq\n\nMetricQ is a highly-scalable, distributed metric data processing framework based on RabbitMQ.\nThis repository used to be the central repository, but has since been splitted into several other\nrepositories.\n\nThe different MetricQ language implementations can be found here:\n\n- [C++](https://github.com/metricq/metricq-cpp)\n- [Python](https://github.com/metricq/metricq-python)\n\nThe proto files of the used Protobuf definitions can be found [here](https://github.com/metricq/metricq-python).\n\n## Documentation\n\nGiven the distributed architecture of MetricQ, the documentation is scattered over several repositories and webpages:\n\n- [The documentation of the Python implementation is a good entry point](https://metricq.github.io/metricq-python/)\n- [The Wiki in this repository still contains some information](https://github.com/metricq/metricq/wiki)\n- [The HTA README.md talks about some details of persistence layer](https://github.com/metricq/hta/blob/master/README.md)\n- [The Management protocol documentation lists all RPC's used](https://metricq.github.io/metricq-rpc-docs/)\n\nThere are also a lot of client implementations available:\n\n- [Tools using the Python implementation](https://github.com/metricq/metricq-tools)\n- [Examples for the Python implementation](https://github.com/metricq/metricq-python/tree/master/examples)\n- [Tools using the C++ implementation](https://github.com/metricq/metricq-cpp/tree/master/tools)\n- [Examples for the C++ implementation](https://github.com/metricq/metricq-cpp/tree/master/examples)\n- [All MetricQ clients in the MetricQ GitHub organization](https://github.com/metricq?q=topic%3Ametricq-client\u0026type=\u0026language=)\n- [MetricQ client implementations on GitHub](https://github.com/search?p=2\u0026q=metricq-client\u0026type=Repositories)\n\n## Setup development environment with `docker-compose`\n\n\u003e Note: During the startup, especially on the first one, errors and restarts of some services are normal! Please be patient.\n\nJust run:\n\n```\ndocker-compose -f docker-compose-development.yml up\n```\n\nThis will setup:\n\n- [Grafana server](http://localhost:3001) (port 3000 forwarded to localhost:3001)\n- [CouchDB server](http://localhost:5984/_utils) (port 5984 forwarded to localhost)\n- [RabbitMQ server](http://localhost:15672/) (port 5672 and 15672 forwarded to localhost)\n- [MetricQ Wizard](http://localhost:3000/) (port 3000 forwarded to localhost)\n- [MetricQ Webview](http://localhost:3002/webview/) (port 80 forwarded to localhost:3002)\n- [MetricQ Explorer](http://localhost:3004/) (port 80 forwarded to localhost:3004)\n- MetricQ Wizard backend (port 8000 forwarded to localhost)\n- metricq-sink-websocket (port 3000 forwarded to localhost:3003)\n- MetricQ Manager\n- `metricq-grafana` (port 4000 forwarded to localhost)\n- C++ example source generating a metric called `dummy.source`\n- `metricq-rabbitmq-source` providing `metricq.rabbitmq.[...]` performance metrics for the running RabbitMQ server\n- `metricq-source-sysinfo` providing `loalhost.[...]` performance metrics for the docker host\n- `metricq-db-hta` database that stores the metrics\n- `metricq-example-combinator` a combinator that can combine metrics into new metrics\n\nBy default, all logins are `admin` / `admin`. Do not use this dockerfile for production use!\n\nTo run it in the background append `-d`:\n\n```\ndocker-compose -f docker-compose-development.yml up -d\n```\n\nTo stop everything run:\n\n```\ndocker-compose -f docker-compose-development.yml stop\n```\n\nTo stop and remove everything run\n\n```\ndocker-compose -f docker-compose-development.yml down\n```\n\n### Connecting to the MetricQ network\n\nYou can now connect to the network with `amqp://admin:admin@localhost` as url and `dummy.source` as a metric. Using the examples from [metricq-python](https://github.com/metricq/metricq-python).\n\n```\npip install \".[examples]\"\n./examples/metricq_sink.py --server amqp://admin:admin@localhost -m dummy.source\n```\n\n## Setup clustered development environment with `docker-compose`\n\nIf you follow the steps from above instead with `docker-compose-cluster.yml`,\nthree RabbitMQ nodes will be set up.\nOn start, they will automatically form a cluster.\n\nThe container names will be (might be different for your specific setup):\n\n- metricq_rabbitmq-server-node0_1\n- metricq_rabbitmq-server-node1_1\n- metricq_rabbitmq-server-node2_1\n\nBy default, all MetricQ agents started from the compose file will connect to\n`rabbitmq-server`, which resolves to any of the three nodes.\n\n\u003e Note: You need to make sure to use the new BuildKit by for instance setting the ENV variable\n\u003e COMPOSE_DOCKER_CLI_BUILD to 1, or use docker-compose newer than 1.28.0-rc3.\n\n### Configure like live Cluster\n\n- Create a user-policy with\n  - Name: ManagementAsHA\n  - Pattern: `management`\n  - Definition: `ha-mode: all`\n\n### Connecting to nodes from docker network\n\nUse the hostname `rabbitmq-server` and the client will connect to random node in the cluster.\n\nFor specific nodes, use the hostnames `rabbitmq-node0`, `rabbitmq-node1`, or `rabbitmq-node2`.\n\n### Connecting to nodes from host or remotely\n\nThe different RabbitMQ nodes are listening on the network interface of their host.\n\n- rabbitmq-node0: 5671 / 15671\n- rabbitmq-node1: 5672 / 15672\n- rabbitmq-node2: 5673 / 15673\n\n# Acknowledgements\n\nThis work is supported in part by the German Research Foundation (DFG) within the CRC 912 - HAEC.\n\n# Primary Reference\n\nThomas Ilsche, Daniel Hackenberg, Robert Schöne, Mario Bielert, Franz Höpfner and Wolfgang E. Nagel: [MetricQ: A Scalable Infrastructure for Processing High-Resolution Time Series Data 📕](https://tu-dresden.de/zih/forschung/ressourcen/dateien/projekte/haec/metricq-a-scalable-infrastructure-for-processing-high-resolution-time-series-data) 2019 IEEE/ACM Industry/University Joint International Workshop on Data-center Automation, Analytics, and Control (DAAC), Denver, CO, USA, 2019, pp. 7-12, DOI: [`10.1109/DAAC49578.2019.00007`](https://doi.org/10.1109/DAAC49578.2019.00007).\n\n# Additional Reference\n\nThomas Ilsche: [Energy Measurements of High Performance Computing Systems: From Instrumentation to Analysis 📕](https://tud.qucosa.de/api/qucosa%3A71600/attachment/ATT-0/) 2020 Doctoral dissertation TU Dresden, URN: [`urn:nbn:de:bsz:14-qucosa2-716000`](https://nbn-resolving.org/urn:nbn:de:bsz:14-qucosa2-716000)\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmetricq%2Fmetricq","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fmetricq%2Fmetricq","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmetricq%2Fmetricq/lists"}