{"id":16788440,"url":"https://github.com/sasha-s/go-hll","last_synced_at":"2025-04-10T23:13:13.663Z","repository":{"id":57491170,"uuid":"60131175","full_name":"sasha-s/go-hll","owner":"sasha-s","description":"HyperLogLog in golang ","archived":false,"fork":false,"pushed_at":"2018-05-22T06:52:23.000Z","size":56,"stargazers_count":25,"open_issues_count":0,"forks_count":3,"subscribers_count":2,"default_branch":"master","last_synced_at":"2025-04-10T23:13:08.731Z","etag":null,"topics":["golang","hll","hyperloglog"],"latest_commit_sha":null,"homepage":null,"language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/sasha-s.png","metadata":{"files":{"readme":"Readme.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2016-05-31T23:46:24.000Z","updated_at":"2024-03-07T23:00:12.000Z","dependencies_parsed_at":"2022-08-29T20:31:41.895Z","dependency_job_id":null,"html_url":"https://github.com/sasha-s/go-hll","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sasha-s%2Fgo-hll","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sasha-s%2Fgo-hll/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sasha-s%2Fgo-hll/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sasha-s%2Fgo-hll/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/sasha-s","download_url":"https://codeload.github.com/sasha-s/go-hll/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248312135,"owners_count":21082638,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["golang","hll","hyperloglog"],"created_at":"2024-10-13T08:17:54.215Z","updated_at":"2025-04-10T23:13:13.642Z","avatar_url":"https://github.com/sasha-s.png","language":"Go","funding_links":[],"categories":[],"sub_categories":[],"readme":"# HyperLogLog in golang. [Docs](https://godoc.org/github.com/sasha-s/go-hll). [![Build Status](https://travis-ci.org/sasha-s/go-hll.svg?branch=master)](https://travis-ci.org/sasha-s/go-hll) [![codecov](https://codecov.io/gh/sasha-s/go-hll/branch/master/graph/badge.svg)](https://codecov.io/gh/sasha-s/go-hll)\n## What\nA go implementation of HypeLogLog data structure with a twist.\nSee [HyperLogLog in Practice](http://research.google.com/pubs/pub40671.html) paper by Stefan Heule, Marc Nunkesser, Alex Hall.\n\n## Ø-serialization\nThere is no need to serialize/deserialize hll.\nEverything is stored in a byte slice, which can be memory mapped, passed around over the network as is etc.\n\n## Differences from the paper:\n* sparse representation. this implementation does exact counting for small sets.\n* fixed memory usage (even for empty HLL). HLL of a given precision P uses fixed (8 + 3*2^(P-2), 8 byte header + 6 bits per register) size in bytes.\n* thresholds are tuned. different from [Sub-Algorithm Threshold](https://docs.google.com/document/d/1gyjfMHy43U9OWBXxfaeG-3MjGzejW1dlpyMwEYAAWEI/view?fullscreen#heading=h.nd379k1fxnux).\n\n## Why\nI wanted an HLL implementation that is\n\n* simple\n* reasonably fast\n* (almost) non-allocating\n* exact when number of unique elements is small\n* memory-mapped file friendly\n* well tested (90+% coverage)\n\n## Usage\nGet go-hll:\n```sh\ngo get github.com/sasha-s/go-hll\n```\n\nUse it:\n```go\ns, err := SizeByP(16)\nif err != nil {\n\tlog.Panicln(err)\n}\nh := make(HLL, s)\n...\nfor _, x := range []string{\"alpha\", \"beta\"} {\n\th.Add(siphash.Hash(2, 57, []byte(x)))\n}\nlog.Println(h.EstimateCardinality())\n```\n\nUse good hash (otherwise accuracy would be poor). Some options:\n\n* [MurmurHash3](https://github.com/spaolacci/murmur3)\n* [HighwayHash](https://github.com/dgryski/go-highway)\n* [Siphash](https://github.com/dchest/siphash)\n* [SpookyHash](https://github.com/dgryski/go-spooky)\n\n## Speed\n\nBenchmark results on my MacBook Pro (Mid 2014).\n\n```\nAdd-8            9.68ns ± 1%\nEstimate-8       27.3µs ± 1%\nMerge-8          38.0µs ± 1%\n\nAddDense-8       6.73ns ± 3%\nMergeDense-8     37.9µs ± 1%\nEstimateDense-8  22.9µs ± 1%\nSort-8            108µs ± 1%\nAddSparse-8      10.2ns ± 3%\n```\n\nMerge/Estimate etc. are done for P=14.\n\n## Other implementations (in no particular order)\n\n* [HLL++ by Micha Gorelick @mynameisfiber](https://github.com/mynameisfiber/gohll)\n* [Probably (has regular HLL) by Dustin Sallings @dustin](https://github.com/dustin/go-probably)\n* [HLL++ by lytrics](https://github.com/lytics/hll)\n* [HLL/HLL++ by Clark DuVall @clarkduvall](https://github.com/clarkduvall/hyperloglog)\n* [Redis has regular HLL](http://download.redis.io/redis-stable/src/hyperloglog.c). Also see [blogpost](http://antirez.com/news/75).\n* [hllpp by Muir Manders @muirrn/retailnext](https://github.com/retailnext/hllpp)\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsasha-s%2Fgo-hll","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fsasha-s%2Fgo-hll","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsasha-s%2Fgo-hll/lists"}