https://github.com/adroll/python-hll

python-hll
https://github.com/adroll/python-hll

Last synced: 14 days ago
JSON representation

python-hll

• Host: GitHub
• URL: https://github.com/adroll/python-hll
• Owner: AdRoll
• License: mit
• Created: 2019-09-11T20:19:42.000Z (about 5 years ago)
• Default Branch: master
• Last Pushed: 2022-12-26T20:47:53.000Z (almost 2 years ago)
• Last Synced: 2024-10-03T14:48:31.640Z (about 2 months ago)
• Language: Python
• Size: 1.98 MB
• Stars: 18
• Watchers: 8
• Forks: 6
• Open Issues: 4
• Metadata Files:
  • Readme: README.rst
  • Changelog: HISTORY.rst
  • Contributing: CONTRIBUTING.rst
  • License: LICENSE

Awesome Lists containing this project

README

        
==========

python-hll

==========

.. image:: https://img.shields.io/pypi/v/python_hll.svg

        :target: https://pypi.python.org/pypi/python_hll

.. image:: https://readthedocs.org/projects/python-hll/badge/?version=latest

        :target: https://python-hll.readthedocs.io/en/latest/?badge=latest

        :alt: Documentation Status

.. image:: https://img.shields.io/badge/github-python--hll-yellow

        :target: https://github.com/AdRoll/python-hll

A Python implementation of `HyperLogLog `_

whose goal is to be `storage compatible `_

with `java-hll `_, `js-hll `_

and `postgresql-hll `_.

**NOTE:** This is a fairly literal translation/port of `java-hll `_

to Python. Internally, bytes are represented as Java-style bytes (-128 to 127) rather than Python-style bytes (0 to 255).

Also this implementation is quite slow: for example, in Java ``HLLSerializationTest`` takes 12 seconds to run

while in Python ``test_hll_serialization`` takes 1.5 hours to run (about 400x slower).

* Runs on: Python 2.7 and 3

* Free software: MIT license

* Documentation: https://python-hll.readthedocs.io

* GitHub: https://github.com/AdRoll/python-hll

Overview

---------------

See `java-hll `_ for an overview of what HLLs are and how they work.

Usage

---------------

Hashing and adding a value to a new HLL::

    from python_hll.hll import HLL

    import mmh3

    value_to_hash = 'foo'

    hashed_value = mmh3.hash(value_to_hash)

    hll = HLL(13, 5) # log2m=13, regwidth=5

    hll.add_raw(hashed_value)

Retrieving the cardinality of an HLL::

    cardinality = hll.cardinality()

Unioning two HLLs together (and retrieving the resulting cardinality)::

    hll1 = HLL(13, 5) # log2m=13, regwidth=5

    hll2 = HLL(13, 5) # log2m=13, regwidth=5

    # ... (add values to both sets) ...

    hll1.union(hll2) # modifies hll1 to contain the union

    cardinalityUnion = hll1.cardinality()

Reading an HLL from a hex representation of

`storage specification, v1.0.0 `_

(for example, retrieved from a `PostgreSQL database `_)::

    from python_hll.util import NumberUtil

    input = '\\x128D7FFFFFFFFFF6A5C420'

    hex_string = input[2:]

    hll = HLL.from_bytes(NumberUtil.from_hex(hex_string, 0, len(hex_string)))

Writing an HLL to its hex representation of

`storage specification, v1.0.0 `_

(for example, to be inserted into a `PostgreSQL database `_)::

    bytes = hll.to_bytes()

    output = "\\x" + NumberUtil.to_hex(bytes, 0, len(bytes))

Also see the `API documentation `_.

Development

---------------

See `Contributing `_ for how to get started building, testing, and deploying the code.