{"id":13465154,"url":"https://github.com/Callidon/bloom-filters","last_synced_at":"2025-03-25T13:33:05.602Z","repository":{"id":41380777,"uuid":"79781970","full_name":"Callidon/bloom-filters","owner":"Callidon","description":"JS implementation of probabilistic data structures: Bloom Filter (and its derived), HyperLogLog, Count-Min Sketch, Top-K and MinHash","archived":false,"fork":false,"pushed_at":"2024-11-21T20:31:07.000Z","size":9574,"stargazers_count":394,"open_issues_count":12,"forks_count":47,"subscribers_count":3,"default_branch":"master","last_synced_at":"2025-03-23T00:36:33.254Z","etag":null,"topics":["bloom-filter","count-min-sketch","cuckoo-filter","hyperloglog","invertible-bloom-filter","javascript","minhash","probabilistic","topk"],"latest_commit_sha":null,"homepage":"https://callidon.github.io/bloom-filters/","language":"TypeScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/Callidon.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2017-01-23T07:47:16.000Z","updated_at":"2025-03-19T09:13:58.000Z","dependencies_parsed_at":"2024-05-06T22:27:56.208Z","dependency_job_id":"553a9c90-ffdf-4b83-9fbb-88178e84f461","html_url":"https://github.com/Callidon/bloom-filters","commit_stats":{"total_commits":196,"total_committers":11,"mean_commits":"17.818181818181817","dds":0.5102040816326531,"last_synced_commit":"a795cdb5f86663103c5fc201dc13ca69136c19f2"},"previous_names":[],"tags_count":35,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Callidon%2Fbloom-filters","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Callidon%2Fbloom-filters/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Callidon%2Fbloom-filters/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Callidon%2Fbloom-filters/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/Callidon","download_url":"https://codeload.github.com/Callidon/bloom-filters/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":245471286,"owners_count":20620913,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["bloom-filter","count-min-sketch","cuckoo-filter","hyperloglog","invertible-bloom-filter","javascript","minhash","probabilistic","topk"],"created_at":"2024-07-31T14:01:01.748Z","updated_at":"2025-03-25T13:33:05.221Z","avatar_url":"https://github.com/Callidon.png","language":"TypeScript","funding_links":[],"categories":["TypeScript"],"sub_categories":[],"readme":"# Bloom-Filters [![Master](https://github.com/callidon/bloom-filters/actions/workflows/npm_test_doc.yml/badge.svg)](https://github.com/Callidon/bloom-filters/actions)\n\nJavaScript/TypeScript implementation of probabilistic data structures: Bloom Filter (and its derived), HyperLogLog, Count-Min Sketch, Top-K and MinHash.\n**This package relies on [non-cryptographic hash functions](https://cyan4973.github.io/xxHash/)**.\n\n📕[Online documentation](https://callidon.github.io/bloom-filters/)\n\n**Keywords:** _bloom filter, cuckoo filter, KyperLogLog, MinHash, Top-K, probabilistic data-structures, XOR-Filter._\n\n❗️**Compatibility**❗️\n\n- Be carefull when migrating from a version to another.\n- Bug fixes were introduced in `1.3.7` and from `1.3.9` to `2.0.0+` for hashing and indexing data. Then, you **must re-build completely your filters from start** to be compatible with the new versions.\n- To keep the `breaking changes` rule of npm versions we will make now new `majored versions` since 1.3.9 whenever a modification is done on the hashing/indexing system or breaks the current API.\n\n# Table of contents\n\n- [Installation](#installation)\n- [Data structures](#data-structures)\n  - [Classic Bloom Filter](#classic-bloom-filter)\n  - [Partitioned Bloom Filter](#partitioned-bloom-filter)\n  - [Scalable Bloom Filter](#scalable-bloom-filter)\n  - [Cuckoo Filter](#cuckoo-filter)\n  - [Counting Bloom Filter](#counting-bloom-filter)\n  - [Count Min Sketch](#count-min-sketch)\n  - [HyperLogLog](#hyperloglog)\n  - [MinHash](#minhash)\n  - [Top-K](#top-k)\n  - [Invertible Bloom Filters](#invertible-bloom-filters)\n  - [XOR-Filter](#xor-filter)\n- [Export and import](#export-and-import)\n- [Seeding and Hashing](#seeding-and-hashing)\n- [Documentation](#documentation)\n- [Tests](#tests-and-development)\n- [References](#references)\n- [Changelog](#changelog)\n- [License](#license)\n\n## Installation\n\n```bash\nnpm install bloom-filters --save\n```\n\n**Supported platforms**\n\n- [Node.js](https://nodejs.org): _v4.0.0_ or higher\n- [Google Chrome](https://www.google.com/intl/en/chrome/): _v41_ or higher\n- [Mozilla Firefox](https://www.mozilla.org/en-US/firefox/new/): _v34_ or higher\n- [Microsoft Edge](https://www.microsoft.com/en-US/edge): _v12_ or higher\n\n## Data structures\n\n### Classic Bloom Filter\n\nA Bloom filter is a space-efficient probabilistic data structure, conceived by Burton Howard Bloom in 1970,\nthat is used to test whether an element is a member of a set. False positive matches are possible, but false negatives are not.\n\n**Reference:** Bloom, B. H. (1970). _Space/time trade-offs in hash coding with allowable errors_. Communications of the ACM, 13(7), 422-426.\n([Full text article](http://crystal.uta.edu/~mcguigan/cse6350/papers/Bloom.pdf))\n\n#### Methods\n\n- `add(element: HashableInput) -\u003e void`: add an element into the filter.\n- `has(element: HashableInput) -\u003e boolean`: Test an element for membership, returning False if the element is definitively not in the filter and True is the element might be in the filter.\n- `equals(other: BloomFilter) -\u003e boolean`: Test if two filters are equals.\n- `rate() -\u003e number`: compute the filter's false positive rate (or error rate).\n\n```javascript\nconst {BloomFilter} = require('bloom-filters')\n// create a Bloom Filter with a size of 10 and 4 hash functions\nlet filter = new BloomFilter(10, 4)\n// insert data\nfilter.add('alice')\nfilter.add('bob')\n\n// lookup for some data\nconsole.log(filter.has('bob')) // output: true\nconsole.log(filter.has('daniel')) // output: false\n\n// print the error rate\nconsole.log(filter.rate())\n\n// alternatively, create a bloom filter optimal for a number of items and a desired error rate\nconst items = ['alice', 'bob']\nconst errorRate = 0.04 // 4 % error rate\nfilter = BloomFilter.create(items.length, errorRate)\n\n// or create a bloom filter optimal for a collections of items and a desired error rate\nfilter = BloomFilter.from(items, errorRate)\n```\n\n### Partitioned Bloom Filter\n\nA Partitioned Bloom Filter is a variation of a classic Bloom Filter.\n\nThis filter works by partitioning the M-sized bit array into k slices of size `m = M/k` bits, `k = nb of hash functions` in the filter.\nEach hash function produces an index over `m` for its respective slice.\nThus, each element is described by exactly `k` bits, meaning the distribution of false positives is uniform across all elements.\n\nBe careful, as a Partitioned Bloom Filter have much higher collison risks that a classic Bloom Filter on small sets of data.\n\n**Reference:** Chang, F., Feng, W. C., \u0026 Li, K. (2004, March). _Approximate caches for packet classification._ In INFOCOM 2004. Twenty-third AnnualJoint Conference of the IEEE Computer and Communications Societies (Vol. 4, pp. 2196-2207). IEEE.\n([Full text article](http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.153.6902\u0026rep=rep1\u0026type=pdf))\n\n#### Methods\n\n- `add(element: HashableInput) -\u003e void`: add an element into the filter.\n- `has(element: HashableInput) -\u003e boolean`: Test an element for membership, returning False if the element is definitively not in the filter and True is the element might be in the filter.\n- `equals(other: PartitionedBloomFilter) -\u003e boolean`: Test if two filters are equals.\n- `rate() -\u003e number`: compute the filter's false positive rate (or error rate).\n\n```javascript\nconst {PartitionedBloomFilter} = require('bloom-filters')\n\n// create a PartitionedBloomFilter of size 10, with 5 hash functions and a load factor of 0.5\nconst filter = new PartitionedBloomFilter(10, 5, 0.5)\n\n// add some value in the filter\nfilter.add('alice')\nfilter.add('bob')\n\n// lookup for some data\nconsole.log(filter.has('bob')) // output: true\nconsole.log(filter.has('daniel')) // output: false\n\n// now use it like a classic bloom filter!\n// ...\n\n// alternatively, create a PartitionedBloomFilter optimal for a number of items and a desired error rate\nconst items = ['alice', 'bob']\nconst errorRate = 0.04 // 4 % error rate\nfilter = PartitionedBloomFilter.create(items.length, errorRate)\n\n// or create a PartitionedBloomFilter optimal for a collections of items and a desired error rate\nfilter = PartitionedBloomFilter.from(items, errorRate)\n```\n\n### Scalable Bloom Filter\n\nA Scalable Bloom Filter is a variant of Bloom Filters that can adapt dynamically to the\nnumber of elements stored, while assuring a maximum false positive probability\n\n**Reference:** ALMEIDA, Paulo Sérgio, BAQUERO, Carlos, PREGUIÇA, Nuno, et al. Scalable bloom filters. Information Processing Letters, 2007, vol. 101, no 6, p. 255-261.\n([Full text article](https://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.725.390\u0026rep=rep1\u0026type=pdf))\n\nThis filter use internally [Paritionned Bloom Filters](#partitioned-bloom-filter).\n\n#### Methods\n\n- `add(element: HashableInput) -\u003e void`: add an element into the filter.\n- `has(element: HashableInput) -\u003e boolean`: Test an element for membership, returning False if the element is definitively not in the filter and True is the element might be in the filter.\n- `equals(other: ScalableBloomFilter) -\u003e boolean`: Test if two filters are equals.\n- `capacity():number` -\u003e return the total capacity of this filter\n- `rate() -\u003e number`: compute the filter's false positive rate (or error rate).\n\n```javascript\nconst {ScalableBloomFilter} = require('bloom-filters')\n\n// by default it creates an ideally scalable bloom filter for 8 elements with an error rate of 0.01 and a load factor of 0.5\nconst filter = new ScalableBloomFilter()\nfilter.add('alice')\nfilter.add('bob')\nfilter.add('carl')\nfor (let i = 0; i \u003c 10000; i++) {\n  filter.add('elem:' + i)\n}\nfilter.has('somethingwrong') // false\n\nfilter.capacity() // total capacity\nfilter.rate() // current rate of the current internal filter used\n```\n\n### Cuckoo Filter\n\nCuckoo filters improve on Bloom filters by supporting deletion, limited counting, and bounded False positive rate with similar storage efficiency as a standard Bloom Filter.\n\n**Reference:** Fan, B., Andersen, D. G., Kaminsky, M., \u0026 Mitzenmacher, M. D. (2014, December). _Cuckoo filter: Practically better than bloom._ In Proceedings of the 10th ACM International on Conference on emerging Networking Experiments and Technologies (pp. 75-88). ACM.\n([Full text article](https://www.cs.cmu.edu/~dga/papers/cuckoo-conext2014.pdf))\n\n#### Methods\n\n- `add(element: HashableInput) -\u003e void`: add an element into the filter.\n- `remove(element: HashableInput) -\u003e boolean`: delete an element from the filter, returning True if the deletion was a success and False otherwise.\n- `has(element: HashableInput) -\u003e boolean`: Test an element for membership, returning False if the element is definitively not in the filter and True is the element might be in the filter.\n- `equals(other: CuckooFilter) -\u003e boolean`: Test if two filters are equals.\n- `rate() -\u003e number`: compute the filter's false positive rate (or error rate).\n\n```javascript\nconst {CuckooFilter} = require('bloom-filters')\n\n// create a Cuckoo Filter with size = 15, fingerprint length = 3 and bucket size = 2\nconst filter = new CuckooFilter(15, 3, 2)\nfilter.add('alice')\nfilter.add('bob')\n\n// lookup for some data\nconsole.log(filter.has('bob')) // output: true\nconsole.log(filter.has('daniel')) // output: false\n\n// remove something\nfilter.remove('bob')\nconsole.log(filter.has('bob')) // output: false\n\n// alternatively, create a Cuckoo Filter optimal for a number of items and a desired error rate\nconst items = ['alice', 'bob']\nconst errorRate = 0.04 // 4 % error rate\nfilter = CuckooFilter.create(items.length, errorRate)\n\n// or create a Cuckoo Filter optimal for a collections of items and a desired error rate\nfilter = CuckooFilter.from(items, errorRate)\n```\n\n**WARNING**: The error rate cannot be higher than `1 * 10^-18`. Above this value, you will get an exception stating that the fingerprint length is higher than the hash length.\n\n### Counting Bloom Filter\n\nA Counting Bloom filter works in a similar manner as a regular Bloom filter; however, it is able to keep track of insertions and deletions. In a counting Bloom filter, each entry in the Bloom filter is a small counter associated with a basic Bloom filter bit.\n\n**Reference:** F. Bonomi, M. Mitzenmacher, R. Panigrahy, S. Singh, and G. Varghese, “An Improved Construction for Counting Bloom Filters,” in 14th Annual European Symposium on Algorithms, LNCS 4168, 2006\n\n#### Methods\n\n- `add(element: HashableInput) -\u003e void`: add an element into the filter.\n- `remove(element: HashableInput) -\u003e boolean`: delete an element from the filter, returning True if the deletion was a success and False otherwise.\n- `has(element: HashableInput) -\u003e boolean`: Test an element for membership, returning False if the element is definitively not in the filter and True is the element might be in the filter.\n- `equals(other: CountingBloomFilter) -\u003e boolean`: Test if two filters are equals.\n- `rate() -\u003e number`: compute the filter's false positive rate (or error rate).\n\n```javascript\nconst CountingBloomFilter = require('bloom-filters').CountingBloomFilter\n\n// create a Bloom Filter with capacity = 15 and 4 hash functions\nlet filter = new CountingBloomFilter(15, 4)\n\n// add some value in the filter\nfilter.add('alice')\nfilter.add('bob')\nfilter.add('carole')\n\n// remove some value\nfilter.remove('carole')\n\n// lookup for some data\nconsole.log(filter.has('bob')) // output: true\nconsole.log(filter.has('carole')) // output: false\nconsole.log(filter.has('daniel')) // output: false\n\n// print false positive rate (around 0.1)\nconsole.log(filter.rate())\n\n// alternatively, create a Counting Bloom Filter optimal for a number of items and a desired error rate\nconst items = ['alice', 'bob']\nconst errorRate = 0.04 // 4 % error rate\nfilter = CountingBloomFilter.create(items.length, errorRate)\n\n// or create a Counting Bloom Filter optimal for a collections of items and a desired error rate\nfilter = CountingBloomFilter.from(items, errorRate)\n```\n\n### Count Min Sketch\n\nThe Count Min Sketch (CM sketch) is a probabilistic data structure that serves as a frequency table of events in a stream of data.\nIt uses hash functions to map events to frequencies, but unlike a hash table uses only sub-linear space, at the expense of overcounting some events due to collisions.\n\n**Reference:** Cormode, G., \u0026 Muthukrishnan, S. (2005). _An improved data stream summary: the count-min sketch and its applications._ Journal of Algorithms, 55(1), 58-75.\n([Full text article](http://dimacs.rutgers.edu/~graham/pubs/papers/cm-full.pdf))\n\n#### Methods\n\n- `update(element: HashableInput, count = 1) -\u003e void`: add `count` occurences of an element into the sketch.\n- `count(element: HashableInput) -\u003e number`: estimate the number of occurences of an element.\n- `merge(other: CountMinSketch) -\u003e CountMinSketch`: merge occurences of two sketches.\n- `equals(other: CountMinSketch) -\u003e boolean`: Test if two sketchs are equals.\n- `clone(): CountMinSketch`: Clone the sketch.\n\n```javascript\nconst {CountMinSketch} = require('bloom-filters')\n\n// create a new Count Min sketch with 2048 columns and 1 row\nconst sketch = new CountMinSketch(2048, 1)\n\n// push some occurrences in the sketch\nsketch.update('alice')\nsketch.update('alice')\nsketch.update('bob')\n\n// count occurrences\nconsole.log(sketch.count('alice')) // output: 2\nconsole.log(sketch.count('bob')) // output: 1\nconsole.log(sketch.count('daniel')) // output: 0\n\n// alternatively, create a Count Min sketch optimal for a target error rate and probability of accuracy\nconst items = ['alice', 'bob']\nconst errorRate = 0.04 // 4 % error rate\nconst accuracy = 0.99 // 99% accuracy\nsketch = CountMinSketch.create(errorRate, accuracy)\n\n// or create a Count Min Sketch optimal for a collections of items,\n// a target error rate and probability of accuracy\nsketch = CountMinSketch.from(items, errorRate, accuracy)\n```\n\n### HyperLogLog\n\nHyperLogLog is an algorithm for the count-distinct problem, approximating the number of distinct elements in a multiset. Calculating the exact cardinality of a multiset requires an amount of memory proportional to the cardinality, which is impractical for very large data sets. Probabilistic cardinality estimators, such as the HyperLogLog algorithm, use significantly less memory than this, at the cost of obtaining only an approximation of the cardinality.\nThe HyperLogLog algorithm is able to estimate cardinalities greather than `10e9` with a typical accuracy (standard error) of `2%`, using around 1.5 kB of memory (see reference).\n\n**Reference:** Philippe Flajolet, Éric Fusy, Olivier Gandouet and Frédéric Meunier (2007). _\"Hyperloglog: The analysis of a near-optimal cardinality estimation algorithm\"_. Discrete Mathematics and Theoretical Computer Science Proceedings.\n([Full text article](http://algo.inria.fr/flajolet/Publications/FlFuGaMe07.pdf))\n\n#### Methods\n\n- `update(element: HashableInput) -\u003e void`: add a new occurence of an element to the sketch.\n- `count() -\u003e number`: estimate the number of distinct elements in the sketch.\n- `merge(other: HyperLogLog) -\u003e HyperLogLog`: merge occurences of two sketches.\n- `equals(other: HyperLogLog) -\u003e boolean`: Test if two sketchs are equals.\n\n```javascript\nconst {HyperLogLog} = require('bloom-filters')\n\n// create a new HyperLogLog with 100 registers\nconst sketch = new HyperLogLog(100)\n\n// push some occurrences in the sketch\nsketch.update('alice')\nsketch.update('alice')\nsketch.update('bob')\n\n// count occurrences\nconsole.log(sketch.count())\n\n// print accuracy\nconsole.log(sketch.accuracy())\n```\n\n### MinHash\n\n**MinHash** (or the min-wise independent permutations locality sensitive hashing scheme) is a technique for quickly estimating how similar two sets are.\nThe goal of MinHash is to estimate the _Jaccard similarity coefficient_, a commonly used indicator of the similarity between two sets, without explicitly computing the intersection and union of the two sets.\nIt does so by computing fixed sized signatures for a set of numbers using randomly generated hash functions.\n\n❗️**WARNINGS**❗\n\n- A `MinHash` class only accepts `numbers` (integers and floats) as inputs.\n- Two MinHash can be compared **only if they share the same set of randomly generated hash functions**. To ease the creation of MinHash sets, we introduce a `MinHashFactory` class that is able to create MinHash structures that _share the same set of hash functions_. We recommend most users **to rely on the factory**, but the `MinHash` class remains importable for advanced usage.\n\n**Reference:** Andrei Z. Broder, _\"On the resemblance and containment of documents\"_, in Compression and Complexity of Sequences: Proceedings (1997).\n([Full text article](https://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.24.779\u0026rep=rep1\u0026type=pdf))\n\n#### `MinHashFactory` methods\n\n- `create() -\u003e MinHash`: create a new empty MinHash structure, using the parameters of the factory.\n\n#### `MinHash` methods\n\n- `add(element: number) -\u003e void`: add a new element to the set.\n- `bulkLoad(elements: number[]) -\u003e void`: efficently add several new elements to the set.\n- `isEmpty() -\u003e boolean`: test if the signature of the MinHash is empty.\n- `compareWith(other: MinHash) -\u003e number`: estimate the Jaccard similarity coefficient with another MinHash set.\n\n```javascript\nconst {MinHashFactory} = require('bloom-filters')\n\n// create the MinHashFactory, to create several comparable MinHash sets\n// it uses 10 random hash functions and expect to see a maximum value of 999\nconst factory = new MinHashFactory(10, 999)\n\n// create two empty MinHash\nconst fistSet = factory.create()\nconst secondSet = factory.create()\n\n// push some occurrences in the first set\nfistSet.add(1)\nfistSet.add(2)\n\n// the MinHash class also supports bulk loading\nsecondSet.bulkLoad([1, 3, 4])\n\n// estimate the jaccard similarity between the two sets\nconst jaccardSim = fistSet.compareWith(secondSet)\nconsole.log(`The estimated Jaccard similarity is ${jaccardSim}`)\n```\n\n### Top-K\n\nGiven a multiset of elements, the **Top-K problem** is to compute the ranking of these elements (by an arbitrary score) and returns the `k` results with the highest scores.\nThis package provides an implementation of the Top-K problem that sort items based on their estimated cardinality in the multiset. It is based on a Count Min Sketch, for estimating the cardinality of items, and a MinHeap, for implementing a sliding window over the `k` results with the highest scores.\n\nItems produced by the `TopK` class are JavaScript objects with the following content (shown in Typescript notation).\n\n```typescript\ninterface TopkElement {\n  // The element's value\n  value: string\n  // The element's frequency\n  frequency: number\n  // The element's rank in the TopK, ranging from 1 to k\n  rank: number\n}\n```\n\n#### Methods\n\n- `add(element: string, count: number = 1) -\u003e void`: add one or more new occurences of an element to the sketch.\n- `values() -\u003e Array\u003cTopkElement\u003e`: get the top-k values as an array of objects.\n- `iterator() -\u003e Iterator\u003cTopkElement\u003e`: get the top-k values as an iterator that yields objects.\n\n```javascript\nconst {TopK} = require('bloom-filters')\n\n// create a new TopK with k = 10, an error rate of 0.001 and an accuracy of 0.99\nconst topk = new TopK(10, 0.001, 0.99)\n\n// push occurrences one-at-a-time in the multiset\ntopk.add('alice')\ntopk.add('bob')\ntopk.add('alice')\n\n// or, equally, push multiple occurrences at-once in the multiset\n// topk.add('alice', 2)\n// topk.add('bob', 1)\n\n// print the top k values\nfor (let item of topk.values()) {\n  console.log(\n    `Item \"${item.value}\" is in position ${item.rank} with an estimated frequency of ${item.frequency}`\n  )\n}\n// Output:\n// Item \"alice\" is in position 1 with an estimated frequency of 2\n// Item \"bob\" is in position 2 with an estimated frequency of 1\n```\n\n### Invertible Bloom Filters\n\nAn Invertible Bloom Filters (IBLT), also called Invertible Bloom Lookup Table, is a space-efficient and probabilistic data-structure for solving the set-difference problem efficiently without the use of logs or other prior context. It computes the set difference with communication proportional to the size of the difference between the sets being compared.\nThey can simultaneously calculate D(A−B) and D(B−A) using O(d) space. This data structure encodes sets in a fashion that is similar in spirit to Tornado codes’ construction, in that it randomly combines elements using the XOR function.\n\n❗️**WARNING**❗️ An IBLT only accepts [`Buffer`](https://nodejs.org/api/buffer.html) as inputs. If you are using `bloom-filters` in a Web browser, you might consider using the [`feros/buffer`](https://www.npmjs.com/package/buffer) package, which provides a polyfill for `Buffer` in a browser.\n\n**Reference:** Eppstein, D., Goodrich, M. T., Uyeda, F., \u0026 Varghese, G. (2011). _What's the difference?: efficient set reconciliation without prior context._ ACM SIGCOMM Computer Communication Review, 41(4), 218-229.\n([Full text article](http://www.sysnet.ucsd.edu/sysnet/miscpapers/EppGooUye-SIGCOMM-11.pdf))\n\n#### Methods\n\n- `add(element: Buffer) -\u003e void`: add an element into the filter.\n- `remove(element: Buffer) -\u003e void`: delete an element from the filter, returning True if the deletion was a success and False otherwise.\n- `has(element: Buffer) -\u003e boolean`: Test an element for membership, returning False if the element is definitively not in the filter and True is the element might be in the filter.\n- `equals(other: InvertibleBloomFilter) -\u003e boolean`: Test if two filters are equals.\n- `substract(remote: InvertibleBloomFilter)`: peform the XOR substraction of two IBLTs.\n- `decode() -\u003e {additional: Buffer[], missing: Buffer[]} `: decode an IBLT.\n- `listEntries() -\u003e Generator\u003cBuffer, number, void\u003e`: list all entries in the IBLT using a Generator.\n\n```javascript\nconst {InvertibleBloomFilter} = require('bloom-filters')\n\nconst hashcount = 3\nconst size = 50\nconst iblt = new InvertibleBloomFilter(size, hashcount)\n\n// push some data in the IBLT\niblt.add(Buffer.from('alice'))\niblt.add(Buffer.from('42'))\niblt.add(Buffer.from('help'))\niblt.add(Buffer.from('meow'))\niblt.add(Buffer.from('json'))\n\nconsole.log(ilbt.has(Buffer.from('alice'))) // output: true\nconsole.log(ilbt.has(Buffer.from('daniel'))) // output: false\n\niblt.remove(Buffer.from('alice'))\nconsole.log(ilbt.has(Buffer.from('alice'))) // output: false\n\n// Now, let's demonstrate the decoding power of IBLT!\nconst remote = new InvertibleBloomFilter(size, hashcount)\nremote.add(Buffer.from('alice'))\nremote.add(Buffer.from('car'))\nremote.add(Buffer.from('meow'))\nremote.add(Buffer.from('help'))\n\n// decode the difference between the two filters\nconst result = iblt.substract(remote).decode()\n\nconsole.log(\n  `Did we successfully decode the subtracted iblts? ${result.success}. Why? $${result.reason}`\n)\nconsole.log(\n  `Elements of iblt missing elements from remote: ${result.additional}`\n)\nconsole.log(`Elements of remote missing elements from iblt: ${result.missing}`)\n\n// alternatively, create an IBLT optimal for a number of items and a desired error rate\nconst items = [Buffer.from('alice'), Buffer.from('bob')]\nconst errorRate = 0.04 // 4 % error rate\nfilter = InvertibleBloomFilter.create(items.length, errorRate)\n\n// or create an IBLT optimal for a collections of items and a desired error rate\nfilter = InvertibleBloomFilter.from(items, errorRate)\n```\n\n**Tuning the IBLT** We recommend to use at least a **hashcount** of 3 and an **alpha** of 1.5 for at least 50 differences, which equals to 1.5\\*50 = 75 cells. Then, if you insert a huge number of values in there, the decoding will work (whatever the number of differences less than 50) but testing the presence of a value is still probabilistic, based on the number of elements inserted (Even for the functions like listEntries). For more details, you should read the seminal research paper on IBLTs ([full-text article](http://www.sysnet.ucsd.edu/sysnet/miscpapers/EppGooUye-SIGCOMM-11.pdf)).\n\n### XOR Filter\n\n**Available as 8-bits and 16-bits fingerprint length**\n\nA XOR Filter is a better space-efficient probabilistic data structure than Bloom Filters.\nVery usefull for space efficiency of readonly sets.\n\n**Reference:** Graf, Thomas Mueller, and Daniel Lemire. \"Xor filters: Faster and smaller than bloom and cuckoo filters.\" Journal of Experimental Algorithmics (JEA) 25 (2020): 1-16.\n([Full text article](https://arxiv.org/abs/1912.08258))\n\n#### Methods\n\n- `add(elements: XorHashableInput[]) -\u003e void`: Add elements to the filter. Calling more than once this methods will override the current filter with the new elements.\n- `has(element: XorHashableInput) -\u003e boolean`: true/false whether the element is in the set or not.\n\n---\n\n- Extended input types: `type XorHashableInput = HashableInput | Long`\n- We use Buffers internally which are exported/imported to/from `base64` strings.\n\n```javascript\nconst {XorFilter} = require('bloom-filters')\nconst xor8 = new XorFilter(1)\nxor8.add(['a'])\nxor8.has('a') // true\nxor8.has('b') // false\n// or the combined\nconst filter = XorFilter.create(['a'])\nfilter.has('a') // true\n// using 16-bits fingerprint length\nXorFilter.create(['a'], 16).has('a') // true\nconst a = new XorFilter(1, 16)\na.add(['a'])\na.has('a') // true\n```\n\n## Export and import\n\nAll data structures exposed by this package can be **exported and imported to/from JSON**:\n\n- Use the method `saveAsJSON()` to export any data structures into a JSON object.\n- Use the static method `fromJSON(json)` to load a data structure from a JSON object.\n\n```javascript\nconst {BloomFilter} = require('bloom-filters')\n\nconst filter = new BloomFilter(15, 0.01)\nfilter.add('alice')\n\n// export a bloom filter to JSON\nconst exported = filter.saveAsJSON()\n\n// do something with the JSON object (save it as file, send it to a server, etc)\n// ...\n\n// import the same filter from its JSON export\nconst importedFilter = BloomFilter.fromJSON(exported)\nconsole.log(filter.has('alice')) // output: true\nconsole.log(filter.has('bob')) // output: false\n```\n\n## Seeding and Hashing\n\nBy default every hash function is seeded with an internal seed which is equal to `0x1234567890`. If you want to change it:\n\n```javascript\nconst { BloomFilter } = require('bloom-filter')\nconst bl = new BloomFilter(...)\nconsole.log(bl.seed) // 78187493520\nbl.seed = 0xABCD\nconsole.log(bl.seed) // 43981\n```\n\nBy default we hash elements using `XXH.h64` function from [`xxhashjs`](https://github.com/pierrec/js-xxhash).\nIn the case you want to use your own hash functions, you can use your own Hashing class by extending the default one. Example:\n\n```js\nconst {BloomFilter, Hashing} = require('bloom-filters')\n\nclass CustomHashing extends Hashing {\n  serialize(_element, _seed) {\n    return Number(1)\n  }\n}\n\nconst bl = BloomFilter.create(2, 0.01)\n// override just your structure locally\nbl._hashing = new CustomHashing()\nbl.add('a')\n```\n\nSee `test/utils-test.js` \"_Use different hash functions_\" describe close.\n\n## Documentation\n\nSee [documentation online](https://callidon.github.io/bloom-filters/) or generate it in directory `doc/` with: `npm run doc`\n\n## Tests and Development\n\n- Tests are performed using [mocha](https://github.com/mochajs/mocha) and [nyc](https://github.com/istanbuljs/nyc) (code coverage) on node 12.x, 14.x, 15.x and 16.x for the moment.\n- Linting and formatting are made using `prettier` and `eslint`\n\nWhen submitting pull requests please follow the following guidance:\n\n- Please open pull requests on the develop branch. **Direct contributions to the master branch will be refused without comments**\n- Add tests when possible in the `test` folder.\n- Functions, methods, variables and types must be documented using typedoc annotations\n- Run `yarn test` (build, lint and run the mocha tests suite)\n\n## References\n\n- [Classic Bloom Filter](http://crystal.uta.edu/~mcguigan/cse6350/papers/Bloom.pdf): Bloom, B. H. (1970). _Space/time trade-offs in hash coding with allowable errors._ Communications of the ACM, 13(7), 422-426.\n- [Partitioned Bloom Filter](http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.153.6902\u0026rep=rep1\u0026type=pdf): Chang, F., Feng, W. C., \u0026 Li, K. (2004, March). _Approximate caches for packet classification._ In INFOCOM 2004. Twenty-third AnnualJoint Conference of the IEEE Computer and Communications Societies (Vol. 4, pp. 2196-2207). IEEE.\n- [Cuckoo Filter](https://www.cs.cmu.edu/~dga/papers/cuckoo-conext2014.pdf): Fan, B., Andersen, D. G., Kaminsky, M., \u0026 Mitzenmacher, M. D. (2014, December). _Cuckoo filter: Practically better than bloom._ In Proceedings of the 10th ACM International on Conference on emerging Networking Experiments and Technologies (pp. 75-88). ACM.\n- [Counting Bloom Filter](http://www.eecs.harvard.edu/~michaelm/postscripts/esa2006b.pdf): F. Bonomi, M. Mitzenmacher, R. Panigrahy, S. Singh, and G. Varghese, _An Improved Construction for Counting Bloom Filters_, in 14th Annual European Symposium on Algorithms, LNCS 4168, 2006, pp.\n- [Count Min Sketch](http://vaffanculo.twiki.di.uniroma1.it/pub/Ing_algo/WebHome/p14_Cormode_JAl_05.pdf): Cormode, G., \u0026 Muthukrishnan, S. (2005). _An improved data stream summary: the count-min sketch and its applications._ Journal of Algorithms, 55(1), 58-75.\n- [HyperLogLog](http://algo.inria.fr/flajolet/Publications/FlFuGaMe07.pdf): Philippe Flajolet, Éric Fusy, Olivier Gandouet and Frédéric Meunier (2007). _\"Hyperloglog: The analysis of a near-optimal cardinality estimation algorithm\"_. Discrete Mathematics and Theoretical Computer Science Proceedings.\n- [MinHash](https://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.24.779\u0026rep=rep1\u0026type=pdf): Andrei Z. Broder, _\"On the resemblance and containment of documents\"_, in Compression and Complexity of Sequences: Proceedings (1997).\n- [Invertible Bloom Filters](http://www.sysnet.ucsd.edu/sysnet/miscpapers/EppGooUye-SIGCOMM-11.pdf): Eppstein, D., Goodrich, M. T., Uyeda, F., \u0026 Varghese, G. (2011). _What's the difference?: efficient set reconciliation without prior context._ ACM SIGCOMM Computer Communication Review, 41(4), 218-229.\n- [Xor Filters: Faster and Smaller Than Bloom and Cuckoo Filters](https://arxiv.org/abs/1912.08258) Thomas Mueller Graf, Daniel Lemire, Journal of Experimental Algorithmics 25 (1), 2020. DOI: 10.1145/3376122\n- [Scalable Bloom Filters](https://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.725.390\u0026rep=rep1\u0026type=pdf) ALMEIDA, Paulo Sérgio, BAQUERO, Carlos, PREGUIÇA, Nuno, et al. _Scalable bloom filters_. Information Processing Letters, 2007, vol. 101, no 6, p. 255-261.\n\n## Changelog\n\n| **Version** | **Release date** | **Major changes**                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                      |\n| ----------- | ---------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |\n| `v2.1.0`    | 03/2022          | - Add Scalable Bloom filters \u003cbr/\u003e - Use array of BitSet for Partitionned Bloom Filter \u003cbr/\u003e - Fix wrong MinHash comparison                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                            |\n| `v2.0.0`    | 02/2022          | - Use correctly double hashing [#issue43](https://github.com/Callidon/bloom-filters/issues/43). \u003cbr/\u003e - Move all hashing related functions to its specific Hash class in a component of the BaseFilter class. It also allows for overriding the serizalize function for using custom hash functions \u003cbr/\u003e - Add [#PR44](https://github.com/Callidon/bloom-filters/pull/44) optimizing the BloomFilter internal storage with Uint arrays. \u003cbr/\u003e - Disable 10.x, 15.x node tests. \u003cbr/\u003e - Add XorFilter [#29](https://github.com/Callidon/bloom-filters/issues/29) \u003cbr/\u003e - Add `.nextInt32()` function to get a new random seeded int 32-bits from the current seed. \u003cbr/\u003e - Make all properties public for allowing developpers to override everything. |\n| `v1.3.0`    | 10/04/2020       | Added the MinHash set                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                  |\n| `v1.2.0`    | 08/04/2020       | Add the TopK class                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                     |\n| `v1.1.0`    | 03/04/2020       | Add the HyperLogLog sketch                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                             |\n| `v1.0.0`    | 23/03/2020       | Rework the whole library using TypeScript, unify the API and fix the documentation                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                     |\n| `v0.8.0`    | 11/11/2019       | Fix some issues with the cuckoo filter (performances). Fix the global API. It allows now to customize each Filter. If you want to use the old API, use the `.create()` or `.from()` functions to match the old api.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    |\n| `v0.7.1`    | 11/09/2019       | Add the Counting Bloom Filter                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                          |\n| `v0.7.0`    | 01/07/2019       | Move to [XXHASH](https://cyan4973.github.io/xxHash/) for hashing elements in the library. One property has been added into the exported json `_seed` which is used to seed every hash of every elements. Update Invertible Bloom Filters with #add, #has, #delete, #listEntries, #substract, #Static.decode methods. Updated the way to get distinct indices which could have collisions in many cases.                                                                                                                                                                                                                                                                                                                                                |\n| `v0.6.1`    | 18/06/2019       | Add Invertible Bloom Filters (only #encode/#substract/#Static.decode methods)                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                          |\n\n## License\n\n[MIT License](https://github.com/Callidon/bloom-filters/blob/master/LICENSE)\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FCallidon%2Fbloom-filters","html_url":"https://awesome.ecosyste.ms/projects/github.com%2FCallidon%2Fbloom-filters","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FCallidon%2Fbloom-filters/lists"}