{"id":49594001,"url":"https://github.com/cockroachdb/goodhistogram","last_synced_at":"2026-05-04T03:04:21.436Z","repository":{"id":351164247,"uuid":"1209809208","full_name":"cockroachdb/goodhistogram","owner":"cockroachdb","description":"A prometheus-native compatible, performant-everywhere histogram implementation.","archived":false,"fork":false,"pushed_at":"2026-04-29T14:42:43.000Z","size":66,"stargazers_count":1,"open_issues_count":2,"forks_count":1,"subscribers_count":0,"default_branch":"main","last_synced_at":"2026-04-29T16:22:44.997Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/cockroachdb.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2026-04-13T19:59:05.000Z","updated_at":"2026-04-29T14:32:13.000Z","dependencies_parsed_at":null,"dependency_job_id":null,"html_url":"https://github.com/cockroachdb/goodhistogram","commit_stats":null,"previous_names":["cockroachdb/goodhistogram"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/cockroachdb/goodhistogram","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/cockroachdb%2Fgoodhistogram","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/cockroachdb%2Fgoodhistogram/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/cockroachdb%2Fgoodhistogram/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/cockroachdb%2Fgoodhistogram/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/cockroachdb","download_url":"https://codeload.github.com/cockroachdb/goodhistogram/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/cockroachdb%2Fgoodhistogram/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":32592720,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-03T22:12:39.696Z","status":"online","status_checked_at":"2026-05-04T02:00:06.625Z","response_time":58,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2026-05-04T03:04:20.635Z","updated_at":"2026-05-04T03:04:21.427Z","avatar_url":"https://github.com/cockroachdb.png","language":"Go","funding_links":[],"categories":[],"sub_categories":[],"readme":"# goodhistogram\n\nA fast, lock-free exponential histogram for Go that produces\nPrometheus native histogram output with bounded relative error\nand trapezoidal quantile estimation.\n\n## Performance vs Prometheus native histograms\n\nSchema 2, range [500ns, 60s]. Speed benchmarks on Intel Xeon @ 2.80GHz, 24\ncores, x86_64, Linux. Accuracy across 12 distributions, 100k observations each.\n\n| | goodhistogram | Prometheus native |\n|---|---:|---:|\n| Memory per histogram | 0.85 KB | 19 KB |\n| Record (1 thread) | 20 ns/op | 132 ns/op |\n| Mean quantile error | 0.35% | 1.00% |\n\nThe full results of benchmarks on a Macbook and GCE worker can be seen in\n[reports/benchmark_comparison.txt](reports/benchmark_comparison.txt).\n\n## Motivation\n\nCockroachDB maintains hundreds of histograms per node for\nlatency distributions, operation sizes (log entries, compaction bytes), and\nresource utilization. These histograms sit on the hot path of every SQL\nstatement, every KV request, and every Raft proposal.\n\nWe'd been using Prometheus' classic histogram implementation to track and\nreport on this data with good results, albeit with a little more overhead than\nwe would have liked. Recent tests however to move Prometheus' new native\nhistogram formats once again surfaced these overhead concerns and the necessity\nof them came into question. \n\nAs a result of this thinking, we designed goodhistogram. goodhistogram is a\npurpose-built replacement that:\n\n- Matches Prometheus native bucket and export format.\n- Strips memory and compute requirements to the bare minimum required.\n- Introduces new recording and quantile computation improvements.\n\nIt's designed to be lightweight, fast and accurate as can be to meet the needs\nof a performance sensitive database. Additionally, goodhistograms are mergeable,\ncompressible, and thread safe, making them ideal for high throughput\ndistributed systems.\n\n## Getting started\n\n### Install\n\n```\ngo get github.com/cockroachdb/goodhistogram\n```\n\n### Create and record\n\n```go\nh := goodhistogram.New(goodhistogram.Params{\n    Lo:           500,    // lower bound of tracked range (e.g. 500ns)\n    Hi:           60e9,   // upper bound (e.g. 60s in nanoseconds)\n    ErrorBound: 0.10,   // 10% relative error → schema 2\n})\n\nh.Record(1_500_000)  // 1.5ms\nh.Record(42_000)     // 42µs\n```\n\n### Take a snapshot and compute quantiles\n\n```go\nsnap := h.Snapshot()\n\np99 := snap.ValueAtQuantile(0.99)\nmean := snap.Mean()\ncount, sum := snap.Total()\n```\n\n### Export to Prometheus\n\n```go\nph := snap.ToPrometheusHistogram() // *prometheusgo.Histogram\n```\n\nThe returned proto includes both conventional cumulative buckets (for\nbackward compatibility with classic Prometheus) and native histogram\nsparse fields (schema, spans, deltas). Because the internal bucket\nindices are Prometheus bucket keys by construction, export is a direct\ncopy with no remapping.\n\n\n## How it works\n\ngoodhistogram takes inspiration from [Prometheus native\nhistograms](https://prometheus.io/docs/specs/native_histograms/),\n[DDSketch](https://www.datadoghq.com/blog/engineering/computing-accurate-percentiles-with-ddsketch/),\nand [base2histogram](https://blog.openacid.com/algo/histogram/). At a high\nlevel, it borrows the exponential bucketing scheme from Prometheus and\nDDSketch, whose error bounds act as the input to the constructor, it uses the\nbucket grouping concept found both in base2histogram and prometheus for fast\nrecording. It then uses the trapezoidal interpolation described by\nbase2histogram to significantly improve quantile estimation.\n\nWhat makes it different from these libraries is the following:\n - It uses an array for optimal runtime operations, while preserving the ability for sparse output.\n - It performs constant time lookups with a log-linear bucketing function.\n - It allows configuring a precise error bound, which determines bucket density.\n - It only allocates buckets within a parameterized expected range, minimizing memory.\n\n### Bucket layout\n\ngoodhistogram uses the same exponential bucket scheme as Prometheus\nnative histograms. For a given schema *s*, each power-of-two octave is\ndivided into 2^*s* buckets with boundaries at 2^(*j*/2^*s*). The\nschema is chosen as the coarsest one whose relative error\n(*gamma* - 1)/(*gamma* + 1) is at or below the requested `ErrorBound`,\nwhere *gamma* = 2^(2^(-*s*)).\n\nAt construction, a fixed array of `atomic.Uint64` counters is allocated\ncovering the range `[Lo, Hi]`. The number of buckets is determined by\nthe Prometheus bucket keys of `Lo` and `Hi`:\n\n```\nnumBuckets = promBucketKey(Hi, schema) - promBucketKey(Lo, schema) + 1\n```\n\n### Recording (the hot path)\n\n`Record(v)` maps an `int64` to a bucket index in O(1) without locks or\nallocations:\n\n1. Convert to `float64` and extract the IEEE 754 bits.\n2. Compute the exponent from the biased exponent field: `exp = bits\u003e\u003e52 - 1022`.\n3. Look up the bucket index from a precomputed 256-entry table indexed by the\n   top 8 bits of the mantissa.\n4. Combine: `key = bucket + (exp-1) * bucketsPerGroup`, then\n   `idx = key - minKey`.\n5. `counts[idx].Add(1)`.\n\nThe novelty of this implementation is in the precomputed bucket lookup table.\nThis enables both constant-time bucket lookup, in addition to prometheus\ncompatible export, which serves as a helpful collection of features from\nincompatible implementations.\n\n### Quantile estimation\n\n`ValueAtQuantile` uses trapezoidal interpolation rather than the\nuniform-density (log-linear) interpolation used by Prometheus:\n\n1. Compute average density in each bucket: count / width.\n2. Estimate density at each boundary by averaging adjacent buckets.\n3. Within the target bucket, model the density as varying linearly from\n   left to right (a trapezoid). The CDF is the integral of this linear\n   function, which is a quadratic.\n4. Solve the quadratic to find the value at the target rank.\n\nThis produces more accurate estimates when the true density is not uniform\nwithin a bucket, which is nearly always the case for real-world distributions.\nThis approach is described well in the base2histogram link cited above.\n\n### Sacrifices in Accuracy\n\nAs a final note on the design, goodhistogram takes two tracks which\ntheoretically affect quantile accuracy. The first is that it does not lock on\nsnapshot, which means that snapshots can be inconsistent. Inconsistent in this\ncase means that the `TotalSum` seen for the histogram can be greater than the sum\nof its parts. In practice, the likelihood of this is low enough, and the order\nof the effect small enough, that no detectable difference would be seen.\n\nBucket lookup is imprecise as well. Because of the techniques used to make\nbucket lookup constant, ~0.45% of the samples will be incorrectly rounded to a\nneighboring bucket. The resulting distortion however is small (.02%) compared\nto the performance gains that it allows for. The distortions seen are similar\nto the ones seen in non-linear bucketing formats, which too optimize for lookup\nperformance.\n\n## Performance Notes\n\nIt should be noted that the performance on this structure is likely to look\nmuch closer to the single-threaded example than the concurrently accessed one.\nThis is because the benchmarks operate on a throughput that is much higher than\nwould be seen on any histogram in CockroachDB. This means the likelihood of any\ntwo requests entering the critical region simultaneously is exceedingly low, at\nleast with the database as written today.\n\nI should also note that these results are not the actual floor observed. The\nbenchmarks above were run on x86, as it is today the most common chip\narchitecture found on cloud machines. If one re-runs the test on ARM-based\nmachines the differences between goodhistogram and Prometheus widen:\n\n| | goodhistogram | Prometheus native |\n|---|---:|---:|\n| Record (1 thread) | 12.9 ns/op | 103 ns/op |\n\nThis is the result of ARM's optimistic atomic instructions, which allow\nincrements without fencing. The gap widens even further on a Mac ARM\nmachine:\n\n| | goodhistogram | Prometheus native |\n|---|---:|---:|\n| Record (1 thread) | 2.6 ns/op | 58.1 ns/op |\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcockroachdb%2Fgoodhistogram","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fcockroachdb%2Fgoodhistogram","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcockroachdb%2Fgoodhistogram/lists"}