{"id":19028804,"url":"https://github.com/nvidia-merlin/hierarchicalkv","last_synced_at":"2025-04-10T12:54:41.201Z","repository":{"id":62880869,"uuid":"503946508","full_name":"NVIDIA-Merlin/HierarchicalKV","owner":"NVIDIA-Merlin","description":"HierarchicalKV is a part of NVIDIA Merlin and provides hierarchical key-value storage to meet RecSys requirements. The key capability of HierarchicalKV is to store key-value feature-embeddings on high-bandwidth memory (HBM) of GPUs and in host memory. It also can be used as a generic key-value storage.","archived":false,"fork":false,"pushed_at":"2024-05-22T19:31:29.000Z","size":6386,"stargazers_count":102,"open_issues_count":9,"forks_count":22,"subscribers_count":19,"default_branch":"master","last_synced_at":"2024-05-22T19:32:22.176Z","etag":null,"topics":["cuda","dynamic-embedding","embedding-storage","gpu","hashtable","key-value-store","recommender-system"],"latest_commit_sha":null,"homepage":"","language":"Cuda","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/NVIDIA-Merlin.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2022-06-15T23:02:40.000Z","updated_at":"2024-06-07T20:50:44.983Z","dependencies_parsed_at":"2023-12-10T02:21:00.718Z","dependency_job_id":"50fe8a15-e86b-4cb5-8d5a-5e7f07f6566f","html_url":"https://github.com/NVIDIA-Merlin/HierarchicalKV","commit_stats":null,"previous_names":[],"tags_count":11,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/NVIDIA-Merlin%2FHierarchicalKV","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/NVIDIA-Merlin%2FHierarchicalKV/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/NVIDIA-Merlin%2FHierarchicalKV/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/NVIDIA-Merlin%2FHierarchicalKV/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/NVIDIA-Merlin","download_url":"https://codeload.github.com/NVIDIA-Merlin/HierarchicalKV/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248221525,"owners_count":21067571,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["cuda","dynamic-embedding","embedding-storage","gpu","hashtable","key-value-store","recommender-system"],"created_at":"2024-11-08T21:12:20.928Z","updated_at":"2025-04-10T12:54:41.185Z","avatar_url":"https://github.com/NVIDIA-Merlin.png","language":"Cuda","funding_links":[],"categories":[],"sub_categories":[],"readme":"# [NVIDIA HierarchicalKV(Beta)](https://github.com/NVIDIA-Merlin/HierarchicalKV)\n\n[![Version](https://img.shields.io/github/v/release/NVIDIA-Merlin/HierarchicalKV?color=orange\u0026include_prereleases)](https://github.com/NVIDIA-Merlin/HierarchicalKV/releases)\n[![GitHub License](https://img.shields.io/github/license/NVIDIA-Merlin/HierarchicalKV)](https://github.com/NVIDIA-Merlin/HierarchicalKV/blob/master/LICENSE)\n[![Documentation](https://img.shields.io/badge/documentation-blue.svg)](https://nvidia-merlin.github.io/HierarchicalKV/master/README.html)\n\n## About HierarchicalKV\n\nHierarchicalKV is a part of NVIDIA Merlin and provides hierarchical key-value storage to meet RecSys requirements.\n\nThe key capability of HierarchicalKV is to store key-value (feature-embedding) on high-bandwidth memory (HBM) of GPUs and in host memory.\n\nYou can also use the library for generic key-value storage.\n\n## Benefits\n\nWhen building large recommender systems, machine learning (ML) engineers face the following challenges:\n\n- GPUs are needed, but HBM on a single GPU is too small for the large DLRMs that scale to several terabytes.\n- Improving communication performance is getting more difficult in larger and larger CPU clusters.\n- It is difficult to efficiently control consumption growth of limited HBM with customized strategies.\n- Most generic key-value libraries provide low HBM and host memory utilization.\n\nHierarchicalKV alleviates these challenges and helps the machine learning engineers in RecSys with the following benefits:\n\n- Supports training large RecSys models on **HBM and host memory** at the same time.\n- Provides better performance by **full bypassing CPUs** and reducing the communication workload.\n- Implements table-size restraint strategies that are based on **LRU or customized strategies**.\n The strategies are implemented by CUDA kernels.\n- Operates at a high working-status load factor that is close to 1.0.\n\n\n## Key ideas\n\n- Buckets are locally ordered\n- Store keys and values separately\n- Store all the keys in HBM\n- Build-in and customizable eviction strategy\n\nHierarchicalKV makes NVIDIA GPUs more suitable for training large and super-large models of ***search, recommendations, and advertising***.\nThe library simplifies the common challenges to building, evaluating, and serving sophisticated recommenders models.\n\n## API Documentation\n\nThe main classes and structs are below, but reading the comments in the source code is recommended:\n\n- [`class HashTable`](https://github.com/NVIDIA-Merlin/HierarchicalKV/blob/master/include/merlin_hashtable.cuh#L151)\n- [`class EvictStrategy`](https://github.com/NVIDIA-Merlin/HierarchicalKV/blob/master/include/merlin_hashtable.cuh#L52)\n- [`struct HashTableOptions`](https://github.com/NVIDIA-Merlin/HierarchicalKV/blob/master/include/merlin_hashtable.cuh#L60)\n\nFor regular API doc, please refer to [API Docs](https://nvidia-merlin.github.io/HierarchicalKV/master/api/index.html)\n\n### API Maturity Matrix\n\n`industry-validated` means the API has been well-tested and verified in at least one real-world scenario.\n\n| Name | Description | Function |\n|:---------------------|:-------------------------------------------------------------------------------------------------------------------------|:-------------------|\n| __insert_or_assign__ | Insert or assign for the specified keys. \u003cbr\u003eOverwrite one key with minimum score when bucket is full. | industry-validated |\n| __insert_and_evict__ | Insert new keys, and evict keys with minimum score when bucket is full. | industry-validated |\n| __find_or_insert__ | Search for the specified keys, and insert them when missed. | well-tested |\n| __assign__ | Update for each key and bypass when missed. | well-tested |\n| __accum_or_assign__ | Search and update for each key. If found, add value as a delta to the original value. \u003cbr\u003eIf missed, update it directly. | well-tested |\n| __find_or_insert\\*__ | Search for the specified keys and return the pointers of values. Insert them firstly when missing. | well-tested |\n| __find__ | Search for the specified keys. | industry-validated |\n| __find\\*__ | Search and return the pointers of values, thread-unsafe but with high performance. | well-tested |\n| __export_batch__ | Exports a certain number of the key-value-score tuples. | industry-validated |\n| __export_batch_if__ | Exports a certain number of the key-value-score tuples which match specific conditions. | industry-validated |\n| __warmup__ | Move the hot key-values from HMEM to HBM | June 15, 2023 |\n\n\n### Evict Strategy\n\nThe `score` is introduced to define the importance of each key, the larger, the more important, the less likely they will be evicted. Eviction only happens when a bucket is full.\nThe `score_type` must be `uint64_t`. For more detail, please refer to [`class EvictStrategy`](https://github.com/NVIDIA-Merlin/HierarchicalKV/blob/master/include/merlin_hashtable.cuh#L52).\n\n| Name | Definition of `Score` |\n|:---------------|:----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|\n| __Lru__ | Device clock in a nanosecond, which could differ slightly from host clock. |\n| __Lfu__ | Frequency increment provided by caller via the input parameter of `scores` of `insert-like` APIs as the increment of frequency. |\n| __EpochLru__ | The high 32bits is the global epoch provided via the input parameter of `global_epoch`, \u003cbr\u003ethe low 32bits is equal to `(device_clock \u003e\u003e 20) \u0026 0xffffffff` with granularity close to 1 ms. |\n| __EpochLfu__ | The high 32bits is the global epoch provided via the input parameter of `global_epoch`, \u003cbr\u003ethe low 32bits is the frequency, \u003cbr\u003ethe frequency will keep constant after reaching the max value of `0xffffffff`. |\n| __Customized__ | Fully provided by the caller via the input parameter of `scores` of `insert-like` APIs. |\n\n\n* __Note__:\n - The `insert-like` APIs mean the APIs of `insert_or_assign`, `insert_and_evict`, `find_or_insert`, `accum_or_assign`, and `find_or_insert`. \n - The `global_epoch` should be maintained by the caller and input as the input parameter of `insert-like` APIs.\n\n### Configuration Options\n\nIt's recommended to keep the default configuration for the options ending with `*`.\n\n| Name | Type | Default | Description |\n|:---------------------------|:-------|:--------|:------------------------------------------------------|\n| __init_capacity__ | size_t | 0 | The initial capacity of the hash table. |\n| __max_capacity__ | size_t | 0 | The maximum capacity of the hash table. |\n| __max_hbm_for_vectors__ | size_t | 0 | The maximum HBM for vectors, in bytes. |\n| __dim__ | size_t | 64 | The dimension of the value vectors. |\n| __max_bucket_size*__ | size_t | 128 | The length of each bucket. |\n| __max_load_factor*__ | float | 0.5f | The max load factor before rehashing. |\n| __block_size*__ | int | 128 | The default block size for CUDA kernels. |\n| __io_block_size*__ | int | 1024 | The block size for IO CUDA kernels. |\n| __device_id*__ | int | -1 | The ID of device. Managed internally when set to `-1` |\n| __io_by_cpu*__ | bool | false | The flag indicating if the CPU handles IO. |\n| __reserved_key_start_bit__ | int | 0 | The start bit offset of reserved key in the 64 bit |\n\n- Fore more details refer to [`struct HashTableOptions`](https://github.com/NVIDIA-Merlin/HierarchicalKV/blob/master/include/merlin_hashtable.cuh#L60).\n\n#### Reserved Keys\n- By default, the keys of `0xFFFFFFFFFFFFFFFD`, `0xFFFFFFFFFFFFFFFE`, and `0xFFFFFFFFFFFFFFFF` are reserved for internal using.\n change `options.reserved_key_start_bit` if you want to use the above keys.\n `reserved_key_start_bit` has a valid range from 0 to 62. The default value is 0, which is the above default reserved keys. When `reserved_key_start_bit` is set to any value other than 0, the least significant bit (bit 0) is always `0` for any reserved key.\n\n- Setting `reserved_key_start_bit = 1`:\n - This setting reserves the two least significant bits 1 and 2 for the reserved keys.\n - In binary, the last four bits range from `1000` to `1110`. Here, the least significant bit (bit 0) is always `0`, and bits from 3 to 63 are set to `1`.\n - The new reserved keys in hexadecimal representation are as follows:\n - `0xFFFFFFFFFFFFFFFE`\n - `0xFFFFFFFFFFFFFFFC`\n - `0xFFFFFFFFFFFFFFF8`\n - `0xFFFFFFFFFFFFFFFA`\n\n- Setting `reserved_key_start_bit = 2`:\n - This configuration reserves bits 2 and 3 as reserved keys.\n - The binary representation for the last five bits ranges from `10010` to `11110`, with the least significant bit (bit 0) always set to `0`, and bits from 4 to 63 are set to `1`.\n\n- if you change the reserved_key_start_bit, you should use same value for save/load\n For more detail, please refer to [`init_reserved_keys`](https://github.com/search?q=repo%3ANVIDIA-Merlin%2FHierarchicalKV%20init_reserved_keys\u0026type=code)\n\n### How to use:\n```cpp\n#include \"merlin_hashtable.cuh\"\n\n\nusing TableOptions = nv::merlin::HashTableOptions;\nusing EvictStrategy = nv::merlin::EvictStrategy;\n\nint main(int argc, char *argv[])\n{\n using K = uint64_t;\n using V = float;\n using S = uint64_t;\n \n // 1. Define the table and use LRU eviction strategy.\n using HKVTable = nv::merlin::HashTable\u003cK, V, S, EvictStrategy::kLru\u003e;\n std::unique_ptr\u003cHKVTable\u003e table = std::make_unique\u003cHKVTable\u003e();\n \n // 2. Define the configuration options.\n TableOptions options;\n options.init_capacity = 16 * 1024 * 1024;\n options.max_capacity = options.init_capacity;\n options.dim = 16;\n options.max_hbm_for_vectors = nv::merlin::GB(16);\n \n \n // 3. Initialize the table memory resource.\n table-\u003einit(options);\n \n // 4. Use table to do something.\n \n return 0;\n}\n\n```\n\n### Usage restrictions\n\n- The `key_type` must be `int64_t` or `uint64_t`.\n- The `score_type` must be `uint64_t`.\n## Contributors\n\nHierarchicalKV is co-maintianed by [NVIDIA Merlin Team](https://github.com/NVIDIA-Merlin) and NVIDIA product end-users,\nand also open for public contributions, bug fixes, and documentation. [[Contribute](CONTRIBUTING.md)]\n\n## How to build\n\nBasically, HierarchicalKV is a headers only library, the commands below only create binaries for benchmark and unit testing.\n\nYour environment must meet the following requirements:\n\n- CUDA version \u003e= 11.2\n- NVIDIA GPU with compute capability 8.0, 8.6, 8.7 or 9.0\n- GCC supports `C++17' standard or later.\n- Bazel version \u003e= 3.7.2 (Bazel compile only)\n\n### with cmake\n```shell\ngit clone --recursive https://github.com/NVIDIA-Merlin/HierarchicalKV.git\ncd HierarchicalKV \u0026\u0026 mkdir -p build \u0026\u0026 cd build\ncmake -DCMAKE_BUILD_TYPE=Release -Dsm=80 .. \u0026\u0026 make -j\n```\n\nFor Debug:\n```shell\ncmake -DCMAKE_BUILD_TYPE=Debug -Dsm=80 .. \u0026\u0026 make -j\n```\n\nFor Benchmark:\n```shell\n./merlin_hashtable_benchmark\n```\n\nFor Unit Test:\n```shell\n./merlin_hashtable_test\n```\n\n### with bazel\n\n- DON'T use the option of `--recursive` for `git clone`.\n- Please modify the environment variables in the `.bazelrc` file in advance if using the customized docker images.\n- The docker images maintained on `nvcr.io/nvidia/tensorflow` are highly recommended.\n\nPull the docker image:\n```shell\ndocker pull nvcr.io/nvidia/tensorflow:22.09-tf2-py3\ndocker run --gpus all -it --rm nvcr.io/nvidia/tensorflow:22.09-tf2-py3\n```\n\nCompile in docker container:\n```shell\ngit clone https://github.com/NVIDIA-Merlin/HierarchicalKV.git\ncd HierarchicalKV \u0026\u0026 bash bazel_build.sh\n```\n\nFor Benchmark:\n```shell\n./benchmark_util\n```\n\n\n## Benchmark \u0026 Performance(W.I.P)\n\n* GPU: 1 x NVIDIA A100 80GB PCIe: 8.0\n* Key Type = uint64_t\n* Value Type = float32 * {dim}\n* Key-Values per OP = 1048576\n* Evict strategy: LRU\n* `λ`: load factor\n* `find*` means the `find` API that directly returns the addresses of values.\n* `find_or_insert*` means the `find_or_insert` API that directly returns the addresses of values.\n* ***Throughput Unit: Billion-KV/second***\n\n### On pure HBM mode: \n\n* dim = 8, capacity = 128 Million-KV, HBM = 4 GB, HMEM = 0 GB\n\n| λ | insert_or_assign | find | find_or_insert | assign | find* | find_or_insert* | insert_and_evict |\n|-----:|-----------------:|-------:|---------------:|-------:|-------:|----------------:|-----------------:|\n| 0.50 | 1.093 | 2.470 | 1.478 | 1.770 | 3.726 | 1.447 | 1.075 |\n| 0.75 | 1.045 | 2.452 | 1.335 | 1.807 | 3.374 | 1.309 | 1.013 |\n| 1.00 | 0.655 | 2.481 | 0.612 | 1.815 | 1.865 | 0.619 | 0.511 |\n\n| λ | export_batch | export_batch_if | contains |\n|-----:|-------------:|----------------:|---------:|\n| 0.50 | 2.087 | 12.258 | 3.121 |\n| 0.75 | 2.045 | 12.447 | 3.094 |\n| 1.00 | 1.950 | 2.657 | 3.096 |\n\n* dim = 32, capacity = 128 Million-KV, HBM = 16 GB, HMEM = 0 GB\n\n| λ | insert_or_assign | find | find_or_insert | assign | find* | find_or_insert* | insert_and_evict |\n|-----:|-----------------:|-------:|---------------:|-------:|-------:|----------------:|-----------------:|\n| 0.50 | 0.961 | 2.272 | 1.278 | 1.706 | 3.718 | 1.435 | 0.931 |\n| 0.75 | 0.930 | 2.238 | 1.177 | 1.693 | 3.369 | 1.316 | 0.866 |\n| 1.00 | 0.646 | 2.321 | 0.572 | 1.783 | 1.873 | 0.618 | 0.469 |\n\n| λ | export_batch | export_batch_if | contains |\n|-----:|-------------:|----------------:|---------:|\n| 0.50 | 0.692 | 10.784 | 3.100 |\n| 0.75 | 0.569 | 10.240 | 3.075 |\n| 1.00 | 0.551 | 0.765 | 3.096 |\n\n* dim = 64, capacity = 64 Million-KV, HBM = 16 GB, HMEM = 0 GB\n\n| λ | insert_or_assign | find | find_or_insert | assign | find* | find_or_insert* | insert_and_evict |\n|-----:|-----------------:|-------:|---------------:|-------:|-------:|----------------:|-----------------:|\n| 0.50 | 0.834 | 1.982 | 1.113 | 1.499 | 3.950 | 1.502 | 0.805 |\n| 0.75 | 0.801 | 1.951 | 1.033 | 1.493 | 3.545 | 1.359 | 0.773 |\n| 1.00 | 0.621 | 2.021 | 0.608 | 1.541 | 1.965 | 0.613 | 0.481 |\n\n| λ | export_batch | export_batch_if | contains |\n|-----:|-------------:|----------------:|---------:|\n| 0.50 | 0.316 | 8.199 | 3.239 |\n| 0.75 | 0.296 | 8.549 | 3.198 |\n| 1.00 | 0.288 | 0.395 | 3.225 |\n\n### On HBM+HMEM hybrid mode: \n\n* dim = 64, capacity = 128 Million-KV, HBM = 16 GB, HMEM = 16 GB\n\n| λ | insert_or_assign | find | find_or_insert | assign | find* | find_or_insert* |\n|-----:|-----------------:|-------:|---------------:|-------:|-------:|----------------:|\n| 0.50 | 0.083 | 0.124 | 0.109 | 0.131 | 3.705 | 1.435 |\n| 0.75 | 0.083 | 0.122 | 0.111 | 0.129 | 3.221 | 1.274 |\n| 1.00 | 0.073 | 0.123 | 0.095 | 0.126 | 1.854 | 0.617 |\n\n| λ | export_batch | export_batch_if | contains |\n|-----:|-------------:|----------------:|---------:|\n| 0.50 | 0.318 | 8.086 | 3.122 |\n| 0.75 | 0.294 | 5.549 | 3.111 |\n| 1.00 | 0.287 | 0.393 | 3.075 |\n\n* dim = 64, capacity = 512 Million-KV, HBM = 32 GB, HMEM = 96 GB\n\n| λ | insert_or_assign | find | find_or_insert | assign | find* | find_or_insert* |\n|-----:|-----------------:|-------:|---------------:|-------:|-------:|----------------:|\n| 0.50 | 0.049 | 0.069 | 0.049 | 0.069 | 3.484 | 1.370 |\n| 0.75 | 0.049 | 0.069 | 0.049 | 0.069 | 3.116 | 1.242 |\n| 1.00 | 0.047 | 0.072 | 0.047 | 0.070 | 1.771 | 0.607 |\n\n| λ | export_batch | export_batch_if | contains |\n|-----:|-------------:|----------------:|---------:|\n| 0.50 | 0.316 | 8.181 | 3.073 |\n| 0.75 | 0.293 | 8.950 | 3.052 |\n| 1.00 | 0.292 | 0.394 | 3.026 |\n\n### Support and Feedback:\n\nIf you encounter any issues or have questions, go to [https://github.com/NVIDIA-Merlin/HierarchicalKV/issues](https://github.com/NVIDIA-Merlin/HierarchicalKV/issues) and submit an issue so that we can provide you with the necessary resolutions and answers.\n\n### Acknowledgment\nWe are very grateful to external initial contributors [@Zhangyafei](https://github.com/zhangyafeikimi) and [@Lifan](https://github.com/Lifann) for their design, coding, and review work.\n\n### License\nApache License 2.0\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fnvidia-merlin%2Fhierarchicalkv","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fnvidia-merlin%2Fhierarchicalkv","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fnvidia-merlin%2Fhierarchicalkv/lists"}