{"id":16714185,"url":"https://github.com/flier/pyfasthash","last_synced_at":"2025-04-04T08:07:23.115Z","repository":{"id":19136359,"uuid":"22366239","full_name":"flier/pyfasthash","owner":"flier","description":"Python Non-cryptographic Hash Library","archived":false,"fork":false,"pushed_at":"2023-08-19T17:37:36.000Z","size":423,"stargazers_count":281,"open_issues_count":20,"forks_count":50,"subscribers_count":8,"default_branch":"master","last_synced_at":"2024-10-13T20:49:40.688Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":"C","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/flier.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE.txt","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2014-07-29T03:21:14.000Z","updated_at":"2024-09-22T16:27:03.000Z","dependencies_parsed_at":"2024-06-18T15:33:24.142Z","dependency_job_id":"d3e51577-43df-4938-a265-f5f0d25111d6","html_url":"https://github.com/flier/pyfasthash","commit_stats":{"total_commits":238,"total_committers":7,"mean_commits":34.0,"dds":0.0672268907563025,"last_synced_commit":"20a53f9bb7bf15f98e3e549f523b49e1e0f62e15"},"previous_names":[],"tags_count":14,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/flier%2Fpyfasthash","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/flier%2Fpyfasthash/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/flier%2Fpyfasthash/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/flier%2Fpyfasthash/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/flier","download_url":"https://codeload.github.com/flier/pyfasthash/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":247142066,"owners_count":20890652,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-10-12T20:49:48.774Z","updated_at":"2025-04-04T08:07:23.083Z","avatar_url":"https://github.com/flier.png","language":"C","funding_links":[],"categories":["C"],"sub_categories":[],"readme":"# Introduction [![pypi](https://img.shields.io/pypi/v/pyhash.svg)](https://pypi.org/project/pyhash/) [![Travis CI Status](https://travis-ci.org/flier/pyfasthash.svg?branch=master)](https://travis-ci.org/flier/pyfasthash) [![codecov](https://codecov.io/gh/flier/pyfasthash/branch/master/graph/badge.svg)](https://codecov.io/gh/flier/pyfasthash)\r\n\r\n`pyhash` is a python non-cryptographic hash library.\r\n\r\nIt provides several common hash algorithms with C/C++ implementation for performance and compatibility.\r\n\r\n```python\r\n\u003e\u003e\u003e import pyhash\r\n\u003e\u003e\u003e hasher = pyhash.fnv1_32()\r\n\r\n\u003e\u003e\u003e hasher('hello world')\r\n2805756500L\r\n\r\n\u003e\u003e\u003e hasher('hello', ' ', 'world')\r\n2805756500L\r\n\r\n\u003e\u003e\u003e hasher('world', seed=hasher('hello '))\r\n2805756500L\r\n```\r\n\r\nIt also can be used to generate fingerprints without seed.\r\n\r\n```python\r\n\u003e\u003e\u003e import pyhash\r\n\u003e\u003e\u003e fp = pyhash.farm_fingerprint_64()\r\n\r\n\u003e\u003e\u003e fp('hello')\r\n\u003e\u003e\u003e 13009744463427800296L\r\n\r\n\u003e\u003e\u003e fp('hello', 'world')\r\n\u003e\u003e\u003e [13009744463427800296L, 16436542438370751598L]\r\n```\r\n\r\n**Notes**\r\n\r\n`hasher('hello', ' ', 'world')` is a syntax sugar for `hasher('world', seed=hasher(' ', seed=hasher('hello')))`, and may not equals to `hasher('hello world')`, because some hash algorithms use different `hash` and `seed` size.\r\n\r\nFor example, `metro` hash always use 32bit seed for 64/128 bit hash value.\r\n\r\n```python\r\n\u003e\u003e\u003e import pyhash\r\n\u003e\u003e\u003e hasher = pyhash.metro_64()\r\n\r\n\u003e\u003e\u003e hasher('hello world')\r\n\u003e\u003e\u003e 5622782129197849471L\r\n\r\n\u003e\u003e\u003e hasher('hello', ' ', 'world')\r\n\u003e\u003e\u003e 16402988188088019159L\r\n\r\n\u003e\u003e\u003e hasher('world', seed=hasher(' ', seed=hasher('hello')))\r\n\u003e\u003e\u003e 16402988188088019159L\r\n```\r\n\r\n# Installation\r\n\r\n```bash\r\n$ pip install pyhash\r\n```\r\n\r\n**Notes**\r\n\r\nIf `pip` install failed with similar errors, [#27](https://github.com/flier/pyfasthash/issues/27)\r\n\r\n```\r\n/usr/lib/gcc/x86_64-linux-gnu/6/include/smmintrin.h:846:1: error: inlining failed in call to always_inline 'long long unsigned int _mm_crc32_u64(long long unsigned int, long long unsigned int)': target specific option mismatch\r\n _mm_crc32_u64 (unsigned long long __C, unsigned long long __V)\r\n ^~~~~~~~~~~~~\r\nsrc/smhasher/metrohash64crc.cpp:52:34: note: called from here\r\n             v[0] ^= _mm_crc32_u64(v[0], read_u64(ptr)); ptr += 8;\r\n                     ~~~~~~~~~~~~~^~~~~~~~~~~~~~~~~~~~~\r\n```\r\n\r\nPlease upgrade `pip` and `setuptools` to latest version and try again\r\n\r\n```bash\r\n$ pip install --upgrade pip setuptools\r\n```\r\n\r\n**Notes**\r\n\r\nIf `pip` install failed on MacOS with similar errors [#28](https://github.com/flier/pyfasthash/issues/28)\r\n\r\n```\r\n   creating build/temp.macosx-10.6-intel-3.6\r\n   ...\r\n   /usr/bin/clang -fno-strict-aliasing -Wsign-compare -fno-common -dynamic -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -arch i386 -arch x86_64 -g -c src/smhasher/metrohash64crc.cpp -o build/temp.macosx-10.6-intel-3.6/src/smhasher/metrohash64crc.o -msse4.2 -maes -mavx -mavx2\r\n    src/smhasher/metrohash64crc.cpp:52:21: error: use of undeclared identifier '_mm_crc32_u64'\r\n                v[0] ^= _mm_crc32_u64(v[0], read_u64(ptr)); ptr += 8;\r\n                        ^\r\n```\r\n\r\nYou may try to\r\n\r\n```bash\r\n$ CFLAGS=\"-mmacosx-version-min=10.13\" pip install pyhash\r\n```\r\n\r\n**Notes**\r\n\r\n`pyhash` only support `pypy` v6.0 or newer, please [download and install](https://pypy.org/download.html) the latest `pypy`.\r\n\r\n# Algorithms\r\n\r\npyhash supports the following hash algorithms\r\n\r\n- [FNV](http://isthe.com/chongo/tech/comp/fnv/) (Fowler-Noll-Vo) hash\r\n  - fnv1_32\r\n  - fnv1a_32\r\n  - fnv1_64\r\n  - fnv1a_64\r\n- [MurmurHash](http://code.google.com/p/smhasher/)\r\n  - murmur1_32\r\n  - murmur1_aligned_32\r\n  - murmur2_32\r\n  - murmur2a_32\r\n  - murmur2_aligned_32\r\n  - murmur2_neutral_32\r\n  - murmur2_x64_64a\r\n  - murmur2_x86_64b\r\n  - murmur3_32\r\n  - murmur3_x86_128\r\n  - murmur3_x64_128\r\n- [lookup3](http://burtleburtle.net/bob/hash/doobs.html)\r\n  - lookup3\r\n  - lookup3_little\r\n  - lookup3_big\r\n- [SuperFastHash](http://www.azillionmonkeys.com/qed/hash.html)\r\n  - super_fast_hash\r\n- [City Hash](https://code.google.com/p/cityhash/)\r\n  _ city_32\r\n  - city_64\r\n  - city_128\r\n  - city_crc_128\r\n  - city_fingerprint_256\r\n- [Spooky Hash](http://burtleburtle.net/bob/hash/spooky.html)\r\n  - spooky_32\r\n  - spooky_64\r\n  - spooky_128\r\n- [FarmHash](https://github.com/google/farmhash)\r\n  - farm_32\r\n  - farm_64\r\n  - farm_128\r\n  - farm_fingerprint_32\r\n  - farm_fingerprint_64\r\n  - farm_fingerprint_128\r\n- [MetroHash](https://github.com/jandrewrogers/MetroHash)\r\n  - metro_64\r\n  - metro_128\r\n  - metro_crc_64\r\n  - metro_crc_128\r\n- [MumHash](https://github.com/vnmakarov/mum-hash)\r\n  - mum_64\r\n- [T1Ha](https://github.com/leo-yuriev/t1ha)\r\n  - t1ha2 _(64-bit little-endian)_\r\n  - t1ha2_128 _(128-bit little-endian)_\r\n  - t1ha1 _(64-bit native-endian)_\r\n  - t1ha1_le _(64-bit little-endian)_\r\n  - t1ha1_be _(64-bit big-endian)_\r\n  - t1ha0 _(64-bit, choice fastest function in runtime.)_\r\n  - ~~t1_32~~\r\n  - ~~t1_32_be~~\r\n  - ~~t1_64~~\r\n  - ~~t1_64_be~~\r\n- [XXHash](https://github.com/Cyan4973/xxHash)\r\n  - xx_32\r\n  - xx_64\r\n  - xxh3_64 **NEW**\r\n  - xxh3_128 **NEW**\r\n- [Highway Hash](https://github.com/google/highwayhash)\r\n  - highway_64 **NEW**\r\n  - highway_128 **NEW**\r\n  - highway_256 **NEW**\r\n\r\n## String and Bytes literals\r\n\r\nPython has two types can be used to present string literals, the hash values of the two types are definitely different.\r\n\r\n- For Python 2.x [String literals](https://docs.python.org/2/reference/lexical_analysis.html#string-literals), `str` will be used by default, `unicode` can be used with the `u` prefix.\r\n- For Python 3.x [String and Bytes literals](https://docs.python.org/3/reference/lexical_analysis.html#string-and-bytes-literals), `unicode` will be used by default, `bytes` can be used with the `b` prefix.\r\n\r\nFor example,\r\n\r\n```\r\n$ python2\r\nPython 2.7.15 (default, Jun 17 2018, 12:46:58)\r\n[GCC 4.2.1 Compatible Apple LLVM 9.1.0 (clang-902.0.39.2)] on darwin\r\nType \"help\", \"copyright\", \"credits\" or \"license\" for more information.\r\n\u003e\u003e\u003e import pyhash\r\n\u003e\u003e\u003e hasher = pyhash.murmur3_32()\r\n\u003e\u003e\u003e hasher('foo')\r\n4138058784L\r\n\u003e\u003e\u003e hasher(u'foo')\r\n2085578581L\r\n\u003e\u003e\u003e hasher(b'foo')\r\n4138058784L\r\n```\r\n\r\n```\r\n$ python3\r\nPython 3.7.0 (default, Jun 29 2018, 20:13:13)\r\n[Clang 9.1.0 (clang-902.0.39.2)] on darwin\r\nType \"help\", \"copyright\", \"credits\" or \"license\" for more information.\r\n\u003e\u003e\u003e import pyhash\r\n\u003e\u003e\u003e hasher = pyhash.murmur3_32()\r\n\u003e\u003e\u003e hasher('foo')\r\n2085578581\r\n\u003e\u003e\u003e hasher(u'foo')\r\n2085578581\r\n\u003e\u003e\u003e hasher(b'foo')\r\n4138058784\r\n```\r\n\r\nYou can also import [unicode_literals](http://python-future.org/unicode_literals.html) to use unicode literals in Python 2.x\r\n\r\n```python\r\nfrom __future__ import unicode_literals\r\n```\r\n\r\n\u003e In general, it is more compelling to use unicode_literals when back-porting new or existing Python 3 code to Python 2/3 than when porting existing Python 2 code to 2/3. In the latter case, explicitly marking up all unicode string literals with u'' prefixes would help to avoid unintentionally changing the existing Python 2 API. However, if changing the existing Python 2 API is not a concern, using unicode_literals may speed up the porting process.\r\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fflier%2Fpyfasthash","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fflier%2Fpyfasthash","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fflier%2Fpyfasthash/lists"}