{"id":17325668,"url":"https://github.com/tessil/array-hash","last_synced_at":"2025-03-21T18:07:06.695Z","repository":{"id":45975390,"uuid":"88193422","full_name":"Tessil/array-hash","owner":"Tessil","description":"C++ implementation of a fast and memory efficient hash map and hash set specialized for strings","archived":false,"fork":false,"pushed_at":"2024-09-22T10:44:51.000Z","size":1037,"stargazers_count":178,"open_issues_count":3,"forks_count":29,"subscribers_count":18,"default_branch":"master","last_synced_at":"2025-03-01T23:26:50.734Z","etag":null,"topics":["c-plus-plus","cpp","data-structures","hash-map","hash-table","header-only"],"latest_commit_sha":null,"homepage":"","language":"C++","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/Tessil.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2017-04-13T18:13:03.000Z","updated_at":"2025-02-26T05:43:28.000Z","dependencies_parsed_at":"2024-10-26T21:34:47.920Z","dependency_job_id":null,"html_url":"https://github.com/Tessil/array-hash","commit_stats":null,"previous_names":[],"tags_count":8,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Tessil%2Farray-hash","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Tessil%2Farray-hash/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Tessil%2Farray-hash/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Tessil%2Farray-hash/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/Tessil","download_url":"https://codeload.github.com/Tessil/array-hash/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":243456529,"owners_count":20293899,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["c-plus-plus","cpp","data-structures","hash-map","hash-table","header-only"],"created_at":"2024-10-15T14:14:01.405Z","updated_at":"2025-03-13T18:08:49.452Z","avatar_url":"https://github.com/Tessil.png","language":"C++","readme":"[![CI](https://github.com/Tessil/array-hash/actions/workflows/ci.yml/badge.svg?branch=master)](https://github.com/Tessil/array-hash/actions/workflows/ci.yml)\n\n## A C++ implementation of a fast and memory efficient hash map/set for strings \n\nCache conscious hash map and hash set for strings based on the \"Cache-conscious collision resolution in string hash tables.\" (Askitis Nikolas and Justin Zobel, 2005) paper. You can find some details regarding the structure [here](https://tessil.github.io/2017/06/22/hat-trie.html#array-hash-table).\n\nThanks to its cache friendliness, the structure provides fast lookups while keeping a low memory usage. The main drawback is the rehash process which is a bit slow and need some spare memory to copy the strings from the old hash table to the new hash table (it can’t use `std::move` as the other hash tables using `std::string` as key).\n\n\u003cp align=\"center\"\u003e\n  \u003cimg src=\"https://tessil.github.io/images/array_hash.png\" width=\"500px\" /\u003e\n\u003c/p\u003e\n\nFour classes are provided: `tsl::array_map`, `tsl::array_set`, `tsl::array_pg_map` and `tsl::array_pg_set`. The first two are faster and use a power of two growth policy, the last two use a prime growth policy instead and are able to cope better with a poor hash function. Use the prime version if there is a chance of repeating patterns in the lower bits of your hash (e.g. you are storing pointers with an identity hash function). See [GrowthPolicy](#growth-policy) for details.\n\nA **benchmark** of `tsl::array_map` against other hash maps can be found [here](https://tessil.github.io/2016/08/29/benchmark-hopscotch-map.html). This page also gives some advices on which hash table structure you should try for your use case (useful if you are a bit lost with the multiple hash tables implementations in the `tsl` namespace). You can also find another benchmark on the [`tsl::hat-trie`](https://github.com/Tessil/hat-trie#benchmark) page.\n\n### Overview\n\n- Header-only library, just add the [include](include/) directory to your include path and you are ready to go. If you use CMake, you can also use the `tsl::array_hash` exported target from the [CMakeLists.txt](CMakeLists.txt).\n- Low memory usage with good performances, see the [benchmark](https://tessil.github.io/2016/08/29/benchmark-hopscotch-map.html) for some numbers.\n- Support for move-only and non-default constructible values.\n- Strings with null characters inside them are supported (you can thus store binary data as key).\n- If the hash is known before a lookup, it is possible to pass it as parameter to speed-up the lookup (see `precalculated_hash` parameter in [API](https://tessil.github.io/array-hash/doc/html/classtsl_1_1array__map.html)).\n- Support for efficient serialization and deserialization (see [example](#serialization) and the `serialize/deserialize` methods in the [API](https://tessil.github.io/array-hash/doc/html/classtsl_1_1array__map.html) for details).\n- By default the maximum allowed size for a key is set to 65 535. This can be raised through the `KeySizeT` template parameter (see [API](https://tessil.github.io/array-hash/doc/html/classtsl_1_1array__map.html#details) for details).\n- By default the maximum size of the map is limited to 4 294 967 296 elements. This can be raised through the `IndexSizeT` template parameter (see [API](https://tessil.github.io/array-hash/doc/html/classtsl_1_1array__map.html#details) for details).\n\n### Differences compared to `std::unordered_map`\n\n`tsl::array_map` tries to have an interface similar to `std::unordered_map`, but some differences exist:\n- Iterator invalidation doesn't behave in the same way, any operation modifying the hash table invalidate them (see [API](https://tessil.github.io/array-hash/doc/html/classtsl_1_1array__map.html#details) for details).\n- References and pointers to keys or values in the map are invalidated in the same way as iterators to these keys-values.\n- Erase operations have an amortized runtime complexity of O(1) for `tsl::array_map`. An erase operation will delete the key immediately but for the value part of the map, the deletion may be delayed. The destructor of the value is only called when the ratio between the size of the map and the size of the map + the number of deleted values still stored is low enough. The method `shrink_to_fit` may be called to force the deletion.\n- The key and the value are stored separately and not in a `std::pair\u003cconst Key, T\u003e`. Methods like `insert` or `emplace` take the key and the value separately instead of a `std::pair`. The insert method looks like `std::pair\u003citerator, bool\u003e insert(const CharT* key, const T\u0026 value)` instead of `std::pair\u003citerator, bool\u003e insert(const std::pair\u003cconst Key, T\u003e\u0026 value)` (see [API](https://tessil.github.io/array-hash/doc/html/classtsl_1_1array__map.html) for details).\n- For iterators, `operator*()` and `operator-\u003e()` return a reference and a pointer to the value `T` instead of `std::pair\u003cconst Key, T\u003e`. For an access to the key string, the `key()` (which returns a `const CharT*`) or `key_sv()` (which returns a `std::basic_string_view\u003cCharT\u003e`) method of the iterator must be called.\n- No support for some bucket related methods (like `bucket_size`, `bucket`, ...).\n\n\nThese differences also apply between `std::unordered_set` and `tsl::array_set`.\n\nThread-safety and exception guarantees are similar to the STL containers.\n\n### Hash function\n\nThe default hash function used by the structure depends on the presence of `std::string_view`. If it is available, `std::hash\u003cstd::string_view\u003e` is used, otherwise a simple [FNV-1a](https://en.wikipedia.org/wiki/Fowler%E2%80%93Noll%E2%80%93Vo_hash_function#FNV-1a_hash) hash function is used to avoid any dependency.\n\nIf you can't use C++17 or later, we recommend to replace the hash function with something like [CityHash](https://github.com/google/cityhash), MurmurHash, [FarmHash](https://github.com/google/farmhash), ... for better performances. On the tests we did, CityHash64 offers a ~40% improvement on reads compared to FNV-1a.\n\n\n```c++\n#include \u003ccity.h\u003e\n\nstruct str_hash {\n    std::size_t operator()(const char* key, std::size_t key_size) const {\n        return CityHash64(key, key_size);\n    }\n};\n\ntsl::array_map\u003cchar, int, str_hash\u003e map;\n```\n\nThe `std::hash\u003cstd::string\u003e` can't be used efficiently as the structure doesn't store any `std::string` object. Any time a hash would be needed, a temporary `std::string` would have to be created.\n\n\n### Growth policy\n\nThe library supports multiple growth policies through the `GrowthPolicy` template parameter. Three policies are provided by the library but you can easily implement your own if needed.\n\n* **[tsl::ah::power_of_two_growth_policy.](https://tessil.github.io/array-hash/doc/html/classtsl_1_1ah_1_1power__of__two__growth__policy.html)** Default policy used by `tsl::array_map/set`. This policy keeps the size of the bucket array of the hash table to a power of two. This constraint allows the policy to avoid the usage of the slow modulo operation to map a hash to a bucket, instead of \u003ccode\u003ehash % 2\u003csup\u003en\u003c/sup\u003e\u003c/code\u003e, it uses \u003ccode\u003ehash \u0026 (2\u003csup\u003en\u003c/sup\u003e - 1)\u003c/code\u003e (see [fast modulo](https://en.wikipedia.org/wiki/Modulo_operation#Performance_issues)). Fast but this may cause a lot of collisions with a poor hash function as the modulo with a power of two only masks the most significant bits in the end.\n* **[tsl::ah::prime_growth_policy.](https://tessil.github.io/array-hash/doc/html/classtsl_1_1ah_1_1prime__growth__policy.html)** Default policy used by `tsl::array_pg_map/set`. The policy keeps the size of the bucket array of the hash table to a prime number. When mapping a hash to a bucket, using a prime number as modulo will result in a better distribution of the hash across the buckets even with a poor hash function. To allow the compiler to optimize the modulo operation, the policy use a lookup table with constant primes modulos (see [API](https://tessil.github.io/array-hash/doc/html/classtsl_1_1ah_1_1prime__growth__policy.html#details) for details). Slower than `tsl::ah::power_of_two_growth_policy` but more secure.\n* **[tsl::ah::mod_growth_policy.](https://tessil.github.io/array-hash/doc/html/classtsl_1_1ah_1_1mod__growth__policy.html)** The policy grows the map by a customizable growth factor passed in parameter. It then just use the modulo operator to map a hash to a bucket. Slower but more flexible.\n\n\nTo implement your own policy, you have to implement the following interface.\n\n```c++\nstruct custom_policy {\n    // Called on hash table construction and rehash, min_bucket_count_in_out is the minimum buckets\n    // that the hash table needs. The policy can change it to a higher number of buckets if needed \n    // and the hash table will use this value as bucket count. If 0 bucket is asked, then the value\n    // must stay at 0.\n    explicit custom_policy(std::size_t\u0026 min_bucket_count_in_out);\n    \n    // Return the bucket [0, bucket_count()) to which the hash belongs. \n    // If bucket_count() is 0, it must always return 0.\n    std::size_t bucket_for_hash(std::size_t hash) const noexcept;\n    \n    // Return the number of buckets that should be used on next growth\n    std::size_t next_bucket_count() const;\n    \n    // Maximum number of buckets supported by the policy\n    std::size_t max_bucket_count() const;\n    \n    // Reset the growth policy as if the policy was created with a bucket count of 0.\n    // After a clear, the policy must always return 0 when bucket_for_hash() is called.\n    void clear() noexcept;\n}\n```\n\n### Installation\n\nTo use the library, just add the [include](include/) directory to your include path. It is a **header-only** library.\n\nIf you use CMake, you can also use the `tsl::array_hash` exported target from the [CMakeLists.txt](CMakeLists.txt) with `target_link_libraries`. \n```cmake\n# Example where the array-hash project is stored in a third-party directory\nadd_subdirectory(third-party/array-hash)\ntarget_link_libraries(your_target PRIVATE tsl::array_hash)  \n```\n\nIf the project has been installed through `make install`, you can also use `find_package(tsl-array-hash REQUIRED)` instead of `add_subdirectory`.\n\nThe code should work with any C++11 standard-compliant compiler and has been tested with GCC 4.8.4, Clang 3.5.0 and Visual Studio 2015.\n\nTo run the tests you will need the Boost Test library and CMake. \n\n```bash\ngit clone https://github.com/Tessil/array-hash.git\ncd array-hash/tests\nmkdir build\ncd build\ncmake ..\ncmake --build .\n./tsl_array_hash_tests\n```\n\n\n\n### Usage\n\nThe API can be found [here](https://tessil.github.io/array-hash/doc_without_string_view/html). If `std::string_view` is available, the API changes slightly and can be found [here](https://tessil.github.io/array-hash/doc/html/).\n\n\n### Example\n\n```c++\n#include \u003ciostream\u003e\n#include \u003ctsl/array_map.h\u003e\n#include \u003ctsl/array_set.h\u003e\n\n\nint main() {\n    // Map of const char* to int\n    tsl::array_map\u003cchar, int\u003e map = {{\"one\", 1}, {\"two\", 2}};\n    map[\"three\"] = 3;\n    map[\"four\"] = 4;\n    \n    map.insert(\"five\", 5);\n    map.insert_ks(\"six_with_extra_chars_we_ignore\", 3, 6);\n    \n    map.erase(\"two\");\n    \n    // When template parameter StoreNullTerminator is true (default) and there is no\n    // null character in the strings.\n    for(auto it = map.begin(); it != map.end(); ++it) {\n        std::cout \u003c\u003c \"{\" \u003c\u003c it.key() \u003c\u003c \", \" \u003c\u003c it.value() \u003c\u003c \"}\" \u003c\u003c std::endl;\n    }\n    \n    // If StoreNullTerminator is false for space efficiency or you are storing null characters, \n    // you can access to the size of the key.\n    for(auto it = map.begin(); it != map.end(); ++it) {\n        (std::cout \u003c\u003c \"{\").write(it.key(), it.key_size()) \u003c\u003c \", \" \u003c\u003c it.value() \u003c\u003c \"}\" \u003c\u003c std::endl;\n    }\n    \n    // Better, use key_sv() if you compiler provides access to std::string_view.\n    for(auto it = map.begin(); it != map.end(); ++it) {\n        std::cout \u003c\u003c \"{\" \u003c\u003c it.key_sv() \u003c\u003c \", \" \u003c\u003c it.value() \u003c\u003c \"}\" \u003c\u003c std::endl;\n    }\n    \n    // Or if you just want the values.\n    for(int value: map) {\n        std::cout \u003c\u003c \"{\" \u003c\u003c value \u003c\u003c \"}\" \u003c\u003c std::endl;\n    }\n\n\n    // Map of const char32_t* to int\n    tsl::array_map\u003cchar32_t, int\u003e map_char32 = {{U\"one\", 1}, {U\"two\", 2}};\n    map_char32[U\"three\"] = 3;\n    \n    \n    // Set of const char*\n    tsl::array_set\u003cchar\u003e set = {\"one\", \"two\", \"three\"};\n    set.insert({\"four\", \"five\"});\n    \n    for(auto it = set.begin(); it != set.end(); ++it) {\n        std::cout \u003c\u003c \"{\" \u003c\u003c it.key() \u003c\u003c \"}\" \u003c\u003c std::endl;\n    }\n}\n```\n\n#### Serialization\n\nThe library provides an efficient way to serialize and deserialize a map or a set so that it can be saved to a file or send through the network.\nTo do so, it requires the user to provide a function object for both serialization and deserialization.\n\n```c++\nstruct serializer {\n    // Must support the following types for U: std::uint64_t, float and T if a map is used.\n    template\u003ctypename U\u003e\n    void operator()(const U\u0026 value);\n    void operator()(const CharT* value, std::size_t value_size);\n};\n```\n\n```c++\nstruct deserializer {\n    // Must support the following types for U: std::uint64_t, float and T if a map is used.\n    template\u003ctypename U\u003e\n    U operator()();\n    void operator()(CharT* value_out, std::size_t value_size);\n};\n```\n\nNote that the implementation leaves binary compatibility (endianness, float binary representation, size of int, ...) of the types it serializes/deserializes in the hands of the provided function objects if compatibility is required.\n\nMore details regarding the `serialize` and `deserialize` methods can be found in the [API](https://tessil.github.io/array-hash/doc/html/classtsl_1_1array__map.html).\n\n```c++\n#include \u003ccassert\u003e\n#include \u003ccstdint\u003e\n#include \u003cfstream\u003e\n#include \u003ctype_traits\u003e\n#include \u003ctsl/array_map.h\u003e\n\n\nclass serializer {\npublic:\n    serializer(const char* file_name) {\n        m_ostream.exceptions(m_ostream.badbit | m_ostream.failbit);\n        m_ostream.open(file_name);\n    }\n    \n    template\u003cclass T,\n             typename std::enable_if\u003cstd::is_arithmetic\u003cT\u003e::value\u003e::type* = nullptr\u003e\n    void operator()(const T\u0026 value) {\n        m_ostream.write(reinterpret_cast\u003cconst char*\u003e(\u0026value), sizeof(T));\n    }\n    \n    void operator()(const char32_t* value, std::size_t value_size) {\n        m_ostream.write(reinterpret_cast\u003cconst char*\u003e(value), value_size*sizeof(char32_t));\n    }\n\nprivate:\n    std::ofstream m_ostream;\n};\n\nclass deserializer {\npublic:\n    deserializer(const char* file_name) {\n        m_istream.exceptions(m_istream.badbit | m_istream.failbit | m_istream.eofbit);\n        m_istream.open(file_name);\n    }\n    \n    template\u003cclass T,\n             typename std::enable_if\u003cstd::is_arithmetic\u003cT\u003e::value\u003e::type* = nullptr\u003e\n    T operator()() {\n        T value;\n        m_istream.read(reinterpret_cast\u003cchar*\u003e(\u0026value), sizeof(T));\n        \n        return value;\n    }\n    \n    void operator()(char32_t* value_out, std::size_t value_size) {\n        m_istream.read(reinterpret_cast\u003cchar*\u003e(value_out), value_size*sizeof(char32_t));\n    }\n\nprivate:\n    std::ifstream m_istream;\n};\n\n\nint main() {\n    const tsl::array_map\u003cchar32_t, std::int64_t\u003e map = {{U\"one\", 1}, {U\"two\", 2}, \n                                                        {U\"three\", 3}, {U\"four\", 4}};\n    \n    \n    const char* file_name = \"array_map.data\";\n    {\n        serializer serial(file_name);\n        map.serialize(serial);\n    }\n    \n    {\n        deserializer dserial(file_name);\n        auto map_deserialized = tsl::array_map\u003cchar32_t, std::int64_t\u003e::deserialize(dserial);\n        \n        assert(map == map_deserialized);\n    }\n    \n    {\n        deserializer dserial(file_name);\n        \n        /**\n         * If the serialized and deserialized map are hash compatibles (see conditions in API), \n         * setting the argument to true speed-up the deserialization process as we don't have \n         * to recalculate the hash of each key. We also know how much space each bucket needs.\n         */\n        const bool hash_compatible = true;\n        auto map_deserialized = \n            tsl::array_map\u003cchar32_t, std::int64_t\u003e::deserialize(dserial, hash_compatible);\n        \n        assert(map == map_deserialized);\n    }\n}\n```\n\n##### Serialization with Boost Serialization and compression with zlib\n\nIt's possible to use a serialization library to avoid some of the boilerplate if the types to serialize are more complex.\n\nThe following example uses Boost Serialization with the Boost zlib compression stream to reduce the size of the resulting serialized file.\n\n\n```c++\n#include \u003cboost/archive/binary_iarchive.hpp\u003e\n#include \u003cboost/archive/binary_oarchive.hpp\u003e\n#include \u003cboost/iostreams/filter/zlib.hpp\u003e\n#include \u003cboost/iostreams/filtering_stream.hpp\u003e\n#include \u003cboost/serialization/split_free.hpp\u003e\n#include \u003cboost/serialization/utility.hpp\u003e\n#include \u003ccassert\u003e\n#include \u003ccstdint\u003e\n#include \u003cfstream\u003e\n#include \u003ctsl/array_map.h\u003e\n\n\ntemplate\u003ctypename Archive\u003e\nstruct serializer {\n    Archive\u0026 ar;\n    \n    template\u003ctypename T\u003e\n    void operator()(const T\u0026 val) { ar \u0026 val; }\n    \n    template\u003ctypename CharT\u003e\n    void operator()(const CharT* val, std::size_t val_size) {\n        ar.save_binary(reinterpret_cast\u003cconst void*\u003e(val), val_size*sizeof(CharT));\n    }   \n};\n\ntemplate\u003ctypename Archive\u003e\nstruct deserializer {\n    Archive\u0026 ar;\n    \n    template\u003ctypename T\u003e\n    T operator()() { T val; ar \u0026 val; return val; }\n    \n    template\u003ctypename CharT\u003e\n    void operator()(CharT* val_out, std::size_t val_size) {\n        ar.load_binary(reinterpret_cast\u003cvoid*\u003e(val_out), val_size*sizeof(CharT));\n    }  \n};\n\nnamespace boost { namespace serialization {\ntemplate\u003cclass Archive, class CharT, class T\u003e\nvoid serialize(Archive \u0026 ar, tsl::array_map\u003cCharT, T\u003e\u0026 map, const unsigned int version) {\n    split_free(ar, map, version); \n}\n\ntemplate\u003cclass Archive, class CharT, class T\u003e\nvoid save(Archive \u0026 ar, const tsl::array_map\u003cCharT, T\u003e\u0026 map, const unsigned int version) {\n    serializer\u003cArchive\u003e serial{ar};\n    map.serialize(serial);\n}\n\n\ntemplate\u003cclass Archive, class CharT, class T\u003e\nvoid load(Archive \u0026 ar, tsl::array_map\u003cCharT, T\u003e\u0026 map, const unsigned int version) {\n    deserializer\u003cArchive\u003e deserial{ar};\n    map = tsl::array_map\u003cCharT, T\u003e::deserialize(deserial);\n}\n}}\n\n\nint main() {\n    const tsl::array_map\u003cchar32_t, std::int64_t\u003e map = {{U\"one\", 1}, {U\"two\", 2}, \n                                                        {U\"three\", 3}, {U\"four\", 4}};\n    \n    \n    const char* file_name = \"array_map.data\";\n    {\n        std::ofstream ofs;\n        ofs.exceptions(ofs.badbit | ofs.failbit);\n        ofs.open(file_name, std::ios::binary);\n        \n        boost::iostreams::filtering_ostream fo;\n        fo.push(boost::iostreams::zlib_compressor());\n        fo.push(ofs);\n        \n        boost::archive::binary_oarchive oa(fo);\n        \n        oa \u003c\u003c map;\n    }\n    \n    {\n        std::ifstream ifs;\n        ifs.exceptions(ifs.badbit | ifs.failbit | ifs.eofbit);\n        ifs.open(file_name, std::ios::binary);\n        \n        boost::iostreams::filtering_istream fi;\n        fi.push(boost::iostreams::zlib_decompressor());\n        fi.push(ifs);\n        \n        boost::archive::binary_iarchive ia(fi);\n     \n        tsl::array_map\u003cchar32_t, std::int64_t\u003e map_deserialized;   \n        ia \u003e\u003e map_deserialized;\n        \n        assert(map == map_deserialized);\n    }\n}\n```\n\n### License\n\nThe code is licensed under the MIT license, see the [LICENSE file](LICENSE) for details.\n","funding_links":[],"categories":[],"sub_categories":[],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ftessil%2Farray-hash","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Ftessil%2Farray-hash","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ftessil%2Farray-hash/lists"}