https://github.com/kislyuk/rehash

Resumable hashlib: a picklable interface to CPython's OpenSSL-based hashlib standard library
https://github.com/kislyuk/rehash

hashlib persistence python serialization

Last synced: about 1 month ago
JSON representation

Resumable hashlib: a picklable interface to CPython's OpenSSL-based hashlib standard library

• Host: GitHub
• URL: https://github.com/kislyuk/rehash
• Owner: kislyuk
• License: apache-2.0
• Created: 2017-07-12T02:59:49.000Z (almost 8 years ago)
• Default Branch: main
• Last Pushed: 2022-08-28T22:58:03.000Z (over 2 years ago)
• Last Synced: 2025-02-27T14:46:46.337Z (about 2 months ago)
• Topics: hashlib, persistence, python, serialization
• Language: Python
• Homepage:
• Size: 48.8 KB
• Stars: 37
• Watchers: 4
• Forks: 3
• Open Issues: 3
• Metadata Files:
  • Readme: README.rst
  • Changelog: Changes.rst
  • License: LICENSE

Awesome Lists containing this project

README

        
Rehash: Resumable Hashlib

=========================

Rehash is a resumable interface to the OpenSSL-based hashers (message digest objects) in the

CPython ``hashlib`` standard library. Rehash provides hashers that

can be pickled, persisted and reconstituted from their ``repr()``,

and otherwise serialized. The rest of the Rehash API is identical to

``hashlib``.

Rehash hashers can be used to checkpoint and restore progress

when hashing large byte streams:

.. code-block:: python

  import pickle, rehash

  hasher = rehash.sha256(b"foo")

  state = pickle.dumps(hasher)

  hasher2 = pickle.loads(state)

  hasher2.update(b"bar")

  assert hasher2.hexdigest() == rehash.sha256(b"foobar").hexdigest()

Installation

------------

::

    pip install rehash

Applications

~~~~~~~~~~~~

Rehash is useful in any situation when your VM is short-lived or preemptible, and the object you're hashing is huge. For

example, Rehash can be used to hand off the hashing state of large objects between AWS Lambda functions or Google Cloud

Functions, which have runtime limits of 15 and 9 minutes, respectively.

.. admonition:: Non-openssl hashers

  ``sha3`` and ``blake2`` hash algorithms in Python 3.6 are not OpenSSL-based and not supported by rehash.

.. admonition:: PyPy

  PyPy uses its own hasher implementations. Those are not serializable using rehash.

.. admonition:: Security note

  By default, rehash objects present themselves with a ``repr()`` that exposes their internal state. This allows one to

  resume the hashing from the point where it stopped. If exposed through an untrusted channel under specific conditions,

  this could potentially allow an attacker to use an extension attack. If you are unsure about the implications of this,

  set ``rehash.opaque_repr = True`` after importing rehash.

Links

-----

* `Project home page (GitHub) `_

* `Documentation (Read the Docs) `_

* `Package distribution (PyPI) `_

* `Change log `_

Bugs

~~~~

Please report bugs, issues, feature requests, etc. on `GitHub `_.

License

-------

Licensed under the terms of the `Apache License, Version 2.0 `_.

.. image:: https://github.com/kislyuk/rehash/workflows/Python%20package/badge.svg

        :target: https://github.com/kislyuk/rehash/actions

.. image:: https://codecov.io/github/kislyuk/rehash/coverage.svg?branch=master

        :target: https://codecov.io/github/kislyuk/rehash?branch=master

.. image:: https://img.shields.io/pypi/v/rehash.svg

        :target: https://pypi.python.org/pypi/rehash

.. image:: https://img.shields.io/pypi/l/rehash.svg

        :target: https://pypi.python.org/pypi/rehash

.. image:: https://readthedocs.org/projects/rehash/badge/?version=latest

        :target: https://rehash.readthedocs.org/