https://github.com/oertl/hyperloglog-sketch-estimation-paper
Paper about the estimation of cardinalities from HyperLogLog sketches
https://github.com/oertl/hyperloglog-sketch-estimation-paper
cardinality-estimation data-sketches hyperloglog hyperloglog-sketches sketch-data-structures
Last synced: 3 months ago
JSON representation
Paper about the estimation of cardinalities from HyperLogLog sketches
- Host: GitHub
- URL: https://github.com/oertl/hyperloglog-sketch-estimation-paper
- Owner: oertl
- Created: 2016-04-15T18:28:22.000Z (about 10 years ago)
- Default Branch: master
- Last Pushed: 2021-05-18T06:17:44.000Z (about 5 years ago)
- Last Synced: 2025-10-04T20:42:01.023Z (8 months ago)
- Topics: cardinality-estimation, data-sketches, hyperloglog, hyperloglog-sketches, sketch-data-structures
- Language: TeX
- Homepage: http://oertl.github.io/hyperloglog-sketch-estimation-paper
- Size: 51.6 MB
- Stars: 62
- Watchers: 6
- Forks: 6
- Open Issues: 1
-
Metadata Files:
- Readme: README.md
Awesome Lists containing this project
README
# New cardinality estimation algorithms for HyperLogLog sketches
[Paper](http://oertl.github.io/hyperloglog-sketch-estimation-paper/paper/paper.pdf) about the estimation of cardinalities from HyperLogLog sketches.
## Abstract
This paper presents new methods to estimate the cardinalities of data sets recorded by HyperLogLog sketches. A theoretically motivated extension to the original estimator is presented that eliminates the bias for small and large cardinalities. Based on the maximum likelihood principle a second unbiased method is derived together with a robust and efficient numerical algorithm to calculate the estimate. The maximum likelihood approach can also be applied to more than a single HyperLogLog sketch. In particular, it is shown that it gives more precise cardinality estimates for union, intersection, or relative complements of two sets that are both represented by HyperLogLog sketches compared to the conventional technique using the inclusion-exclusion principle. All the new methods are demonstrated and verified by extensive simulations.
## See also
The paper [SetSketch: Filling the Gap between MinHash and HyperLogLog](https://arxiv.org/abs/2101.00314) and corresponding [GitHub project](https://github.com/dynatrace-research/set-sketch-paper) present a new data structure, which can be seen as a generalization of HyperLogLog, and whose estimators can therefore also be applied to HyperLogLog sketches.