{"id":19439073,"url":"https://github.com/spacejam/loghisto","last_synced_at":"2025-07-16T21:38:17.527Z","repository":{"id":24484122,"uuid":"27888485","full_name":"spacejam/loghisto","owner":"spacejam","description":"counters and logarithmically bucketed histograms for distributed systems","archived":false,"fork":false,"pushed_at":"2017-09-03T15:31:32.000Z","size":34,"stargazers_count":85,"open_issues_count":3,"forks_count":5,"subscribers_count":6,"default_branch":"master","last_synced_at":"2025-07-09T12:38:58.340Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/spacejam.png","metadata":{"files":{"readme":"readme.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2014-12-11T19:43:45.000Z","updated_at":"2025-05-12T13:02:00.000Z","dependencies_parsed_at":"2022-08-22T18:00:19.103Z","dependency_job_id":null,"html_url":"https://github.com/spacejam/loghisto","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/spacejam/loghisto","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/spacejam%2Floghisto","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/spacejam%2Floghisto/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/spacejam%2Floghisto/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/spacejam%2Floghisto/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/spacejam","download_url":"https://codeload.github.com/spacejam/loghisto/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/spacejam%2Floghisto/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":265218751,"owners_count":23729530,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-10T15:21:09.987Z","updated_at":"2025-07-16T21:38:17.510Z","avatar_url":"https://github.com/spacejam.png","language":"Go","funding_links":[],"categories":[],"sub_categories":[],"readme":"loghisto\n============\n[![Build Status](https://travis-ci.org/spacejam/loghisto.svg)](https://travis-ci.org/spacejam/loghisto)\n\nA metric system for high performance counters and histograms.  Unlike popular metric systems today, this does not destroy the accuracy of histograms by sampling.  Instead, a logarithmic bucketing function compresses values, generally within 1% of their true value (although between 0 and 1 the precision loss may not be within this boundary).  This allows for extreme compression, which allows us to calculate arbitrarily high percentiles with no loss of accuracy - just a small amount of precision.  This is particularly useful for highly-clustered events that are tolerant of a small precision loss, but for which you REALLY care about what the tail looks like, such as measuring latency across a distributed system.\n\nCopied out of my work for the CockroachDB metrics system.  Based on an algorithm created by Keith Frost.\n\n\n### running a print benchmark for quick analysis\n```go\npackage main\n\nimport (\n  \"runtime\"\n  \"github.com/spacejam/loghisto\"\n)\n\nfunc benchmark() {\n  // do some stuff\n}\n\nfunc main() {\n  numCPU := runtime.NumCPU()\n  runtime.GOMAXPROCS(numCPU)\n\n  desiredConcurrency := uint(100)\n  loghisto.PrintBenchmark(\"benchmark1234\", desiredConcurrency, benchmark)\n}\n```\nresults in something like this printed to stdout each second:\n```\n2014-12-11 21:41:45 -0500 EST\nbenchmark1234_count:     2.0171025e+07\nbenchmark1234_max:       2.4642914167480484e+07\nbenchmark1234_99.99:     4913.768840299134\nbenchmark1234_99.9:      1001.2472422902518\nbenchmark1234_99:        71.24044000732538\nbenchmark1234_95:        67.03348428941965\nbenchmark1234_90:        65.68633104092515\nbenchmark1234_75:        63.07152259993664\nbenchmark1234_50:        58.739891704145194\nbenchmark1234_min:       -657.5233632152207           // Corollary: time.Since(time.Now()) is often \u003c 0\nbenchmark1234_sum:       1.648051169322668e+09\nbenchmark1234_avg:       81.70388809307748\nbenchmark1234_agg_avg:   89\nbenchmark1234_agg_count: 6.0962226e+07\nbenchmark1234_agg_sum:   5.454779078e+09\nsys.Alloc:               1.132672e+06\nsys.NumGC:               5741\nsys.PauseTotalNs:        1.569390954e+09\nsys.NumGoroutine:        113\n```\n### adding an embedded metric system to your code\n```go\nimport (\n  \"time\"\n  \"fmt\"\n  \"github.com/spacejam/loghisto\"\n)\nfunc ExampleMetricSystem() {\n  // Create metric system that reports once a minute, and includes stats\n  // about goroutines, memory usage and GC.\n  includeGoProcessStats := true\n  ms := loghisto.NewMetricSystem(time.Minute, includeGoProcessStats)\n  ms.Start()\n\n  // create a channel that subscribes to metrics as they are produced once\n  // per minute.\n  // NOTE: if you allow this channel to fill up, the metric system will NOT\n  // block, and  will FORGET about your channel if you fail to unblock the\n  // channel after 3 configured intervals (in this case 3 minutes) rather\n  // than causing a memory leak.\n  myMetricStream := make(chan *loghisto.ProcessedMetricSet, 2)\n  ms.SubscribeToProcessedMetrics(myMetricStream)\n\n  // create some metrics\n  timeToken := ms.StartTimer(\"time for creating a counter and histo\")\n  ms.Counter(\"some event\", 1)\n  ms.Histogram(\"some measured thing\", 123)\n  timeToken.Stop()\n\n  for m := range myMetricStream {\n    fmt.Printf(\"number of goroutines: %f\\n\", m.Metrics[\"sys.NumGoroutine\"])\n  }\n\n  // if you want to manually unsubscribe from the metric stream\n  ms.UnsubscribeFromProcessedMetrics(myMetricStream)\n\n  // to stop and clean up your metric system\n  ms.Stop()\n}\n```\n### automatically sending your metrics to OpenTSDB, KairosDB or Graphite\n```go\nfunc ExampleExternalSubmitter() {\n  includeGoProcessStats := true\n  ms := NewMetricSystem(time.Minute, includeGoProcessStats)\n  ms.Start()\n  // graphite\n  s := NewSubmitter(ms, GraphiteProtocol, \"tcp\", \"localhost:7777\")\n  s.Start()\n\n  // opentsdb / kairosdb\n  s := NewSubmitter(ms, OpenTSDBProtocol, \"tcp\", \"localhost:7777\")\n  s.Start()\n\n  // to tear down:\n  s.Shutdown()\n}\n```\n\nSee code for the Graphite/OpenTSDB protocols for adding your own output plugins, it's pretty simple.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fspacejam%2Floghisto","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fspacejam%2Floghisto","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fspacejam%2Floghisto/lists"}