{"id":13696038,"url":"https://github.com/Percona-Lab/PromHouse","last_synced_at":"2025-05-03T14:30:56.834Z","repository":{"id":37451214,"uuid":"99583898","full_name":"Percona-Lab/PromHouse","owner":"Percona-Lab","description":"PromHouse is a long-term remote storage with built-in clustering and downsampling for Prometheus 2.x on top of ClickHouse.","archived":false,"fork":false,"pushed_at":"2023-02-09T00:21:43.000Z","size":14956,"stargazers_count":259,"open_issues_count":12,"forks_count":32,"subscribers_count":40,"default_branch":"master","last_synced_at":"2024-08-03T18:20:27.347Z","etag":null,"topics":["clickhouse","prometheus"],"latest_commit_sha":null,"homepage":"","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/Percona-Lab.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":".github/CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":".github/CODEOWNERS","security":null,"support":null,"governance":null}},"created_at":"2017-08-07T13:59:44.000Z","updated_at":"2024-07-24T07:25:04.000Z","dependencies_parsed_at":"2022-07-20T12:17:42.546Z","dependency_job_id":"f90e15ad-dc78-4ea8-9d7b-04a25d819a2c","html_url":"https://github.com/Percona-Lab/PromHouse","commit_stats":null,"previous_names":[],"tags_count":1,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Percona-Lab%2FPromHouse","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Percona-Lab%2FPromHouse/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Percona-Lab%2FPromHouse/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Percona-Lab%2FPromHouse/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/Percona-Lab","download_url":"https://codeload.github.com/Percona-Lab/PromHouse/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":224364596,"owners_count":17299105,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["clickhouse","prometheus"],"created_at":"2024-08-02T18:00:35.782Z","updated_at":"2024-11-12T23:31:48.990Z","avatar_url":"https://github.com/Percona-Lab.png","language":"Go","readme":"# PromHouse\n\n[![Build Status](https://travis-ci.org/Percona-Lab/PromHouse.svg?branch=master)](https://travis-ci.org/Percona-Lab/PromHouse)\n[![codecov](https://codecov.io/gh/Percona-Lab/PromHouse/branch/master/graph/badge.svg)](https://codecov.io/gh/Percona-Lab/PromHouse)\n[![Go Report Card](https://goreportcard.com/badge/github.com/Percona-Lab/PromHouse)](https://goreportcard.com/report/github.com/Percona-Lab/PromHouse)\n[![CLA assistant](https://cla-assistant.percona.com/readme/badge/Percona-Lab/PromHouse)](https://cla-assistant.percona.com/Percona-Lab/PromHouse)\n\nPromHouse is a long-term remote storage with built-in clustering and downsampling for 2.x on top of\n[ClickHouse](https://clickhouse.yandex). Or, rather, it will be someday.\nFeel free to ~~like, share, retweet,~~ star and watch it, but **do not use it in production** yet.\n\n## Database Schema\n\n```sql\nCREATE TABLE time_series (\n    date Date CODEC(Delta),\n    fingerprint UInt64,\n    labels String\n)\nENGINE = ReplacingMergeTree\n    PARTITION BY date\n    ORDER BY fingerprint;\n\nCREATE TABLE samples (\n    fingerprint UInt64,\n    timestamp_ms Int64 CODEC(Delta),\n    value Float64 CODEC(Delta)\n)\nENGINE = MergeTree\n    PARTITION BY toDate(timestamp_ms / 1000)\n    ORDER BY (fingerprint, timestamp_ms);\n```\n\n```sql\nSELECT * FROM time_series WHERE fingerprint = 7975981685167825999;\n```\n\n```\n┌───────date─┬─────────fingerprint─┬─labels─────────────────────────────────────────────────────────────────────────────────┐\n│ 2017-12-31 │ 7975981685167825999 │ {\"__name__\":\"up\",\"instance\":\"promhouse_clickhouse_exporter_1:9116\",\"job\":\"clickhouse\"} │\n└────────────┴─────────────────────┴────────────────────────────────────────────────────────────────────────────────────────┘\n```\n\n```sql\nSELECT * FROM samples WHERE fingerprint = 7975981685167825999 LIMIT 3;\n```\n\n```\n┌─────────fingerprint─┬──timestamp_ms─┬─value─┐\n│ 7975981685167825999 │ 1514730532900 │     0 │\n│ 7975981685167825999 │ 1514730533901 │     1 │\n│ 7975981685167825999 │ 1514730534901 │     1 │\n└─────────────────────┴───────────────┴───────┘\n```\n\nTime series in Prometheus are identified by label name/value pairs, including `__name__` label, which stores metric\nname. Hash of those pairs is called a fingerprint. PromHouse uses the same hash algorithm as Prometheus to simplify data\nmigration. During the operation, all fingerprints and label name/value pairs a kept in memory for fast access. The new\ntime series are written to ClickHouse for persistence. They are also periodically read from it for discovering new time\nseries written by other ClickHouse instances. `ReplacingMergeTree` ensures there are no duplicates if several ClickHouses\nwrote the same time series at the same time.\n\nPromHouse currently stores 24 bytes per sample: 8 bytes for UInt64 fingerprint, 8 bytes for Int64 timestamp, and 8 bytes\nfor Float64 value. The actual compression rate is about 4.5:1, that's about 24/4.5 = 5.3 bytes per sample. Prometheus\nlocal storage compresses 16 bytes (timestamp and value) per sample to [1.37](https://coreos.com/blog/prometheus-2.0-storage-layer-optimization), that's 12:1.\n\nSince ClickHouse v19.3.3 it is possible to use delta and double delta for compression, which should make storage almost as efficient as TSDB's one.\n\n## Outstanding features in the roadmap\n\n- Downsampling (become possible since ClickHouse v18.12.14)\n- Query Hints (become possible since [prometheus PR 4122](https://github.com/prometheus/prometheus/pull/4122), help wanted [issue #24](https://github.com/Percona-Lab/PromHouse/issues/24))\n\n## SQL queries\n\nThe largest jobs and instances by time series count:\n\n```sql\nSELECT\n    job,\n    instance,\n    COUNT(*) AS value\nFROM time_series\nGROUP BY\n    visitParamExtractString(labels, 'job') AS job,\n    visitParamExtractString(labels, 'instance') AS instance\nORDER BY value DESC LIMIT 10\n```\n\nThe largest metrics by time series count (cardinality):\n\n```sql\nSELECT\n    name,\n    COUNT(*) AS value\nFROM time_series\nGROUP BY\n    visitParamExtractString(labels, '__name__') AS name\nORDER BY value DESC LIMIT 10\n```\n\nThe largest time series by samples count:\n\n```sql\nSELECT\n    labels,\n    value\nFROM time_series\nANY INNER JOIN\n(\n    SELECT\n        fingerprint,\n        COUNT(*) AS value\n    FROM samples\n    GROUP BY fingerprint\n    ORDER BY value DESC\n    LIMIT 10\n) USING (fingerprint)\n```\n","funding_links":[],"categories":["Integrations"],"sub_categories":["Metrics and Monitoring"],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FPercona-Lab%2FPromHouse","html_url":"https://awesome.ecosyste.ms/projects/github.com%2FPercona-Lab%2FPromHouse","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FPercona-Lab%2FPromHouse/lists"}