{"id":13693627,"url":"https://github.com/CPPAlliance/NuDB","last_synced_at":"2025-05-02T21:32:54.558Z","repository":{"id":40523126,"uuid":"50840327","full_name":"cppalliance/NuDB","owner":"cppalliance","description":"NuDB: A fast key/value insert-only database for SSD drives in C++11","archived":false,"fork":false,"pushed_at":"2025-04-25T18:51:23.000Z","size":2238,"stargazers_count":394,"open_issues_count":37,"forks_count":62,"subscribers_count":28,"default_branch":"master","last_synced_at":"2025-04-25T19:42:54.995Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":"C++","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"bsl-1.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/cppalliance.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":null,"funding":null,"license":"LICENSE_1_0.txt","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2016-02-01T13:14:26.000Z","updated_at":"2025-04-25T18:50:59.000Z","dependencies_parsed_at":"2024-11-12T19:41:49.106Z","dependency_job_id":null,"html_url":"https://github.com/cppalliance/NuDB","commit_stats":null,"previous_names":[],"tags_count":8,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/cppalliance%2FNuDB","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/cppalliance%2FNuDB/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/cppalliance%2FNuDB/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/cppalliance%2FNuDB/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/cppalliance","download_url":"https://codeload.github.com/cppalliance/NuDB/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":252109133,"owners_count":21696206,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-08-02T17:01:14.171Z","updated_at":"2025-05-02T21:32:53.329Z","avatar_url":"https://github.com/cppalliance.png","language":"C++","readme":"\u003cimg width=\"880\" height = \"80\" alt = \"NuDB\"\n    src=\"https://raw.githubusercontent.com/vinniefalco/NuDB/master/doc/images/readme2.png\"\u003e\n\nBranch          | [`master`](https://github.com/CPPAlliance/NuDB/tree/master) | [`develop`](https://github.com/CPPAlliance/NuDB/tree/develop) |\n--------------- | ----------------------------------------------------------- | ------------------------------------------------------------- |\nDocs            | [![Documentation](https://img.shields.io/badge/docs-master-brightgreen.svg)](http://vinniefalco.github.io/nudb/) | [![Documentation](https://img.shields.io/badge/docs-develop-brightgreen.svg)](http://vinniefalco.github.io/nudb/)\n[Drone](https://drone.io/) | [![Build Status](https://drone.cpp.al/api/badges/CPPAlliance/NuDB/status.svg)](https://drone.cpp.al/CPPAlliance/NuDB) | [![Build Status](https://drone.cpp.al/api/badges/CPPAlliance/NuDB/status.svg?ref=refs/heads/develop)](https://drone.cpp.al/CPPAlliance/NuDB)\n[codecov.io](https://codecov.io) | [![codecov](https://codecov.io/gh/cppalliance/NuDB/branch/master/graph/badge.svg)](https://codecov.io/gh/cppalliance/NuDB/branch/master) | [![codecov](https://codecov.io/gh/cppalliance/NuDB/branch/develop/graph/badge.svg)](https://codecov.io/gh/cppalliance/NuDB/branch/develop)\nLicense         | [![License](https://img.shields.io/badge/license-boost-brightgreen.svg)](LICENSE_1_0.txt)\n\n# A Key/Value Store For SSDs\n\n---\n\n## Contents\n\n- [Introduction](#introduction)\n- [Description](#description)\n- [Requirements](#requirements)\n- [Example](#example)\n- [Building](#building)\n- [Algorithm](#algorithm)\n- [License](#license)\n- [Contact](#contact)\n\n---\n\n## Introduction\n\nNuDB is an append-only, key/value store specifically optimized for random\nread performance on modern SSDs or equivalent high-IOPS devices. The most\ncommon application for NuDB is content addressable storage where a\ncryptographic digest of the data is used as the key. The read performance\nand memory usage are independent of the size of the database. These are\nsome other features:\n\n* Low memory footprint\n* Database size up to 281TB\n* All keys are the same size\n* Append-only, no update or delete\n* Value sizes from 1 to 2^32 bytes (4GB)\n* Performance independent of growth\n* Optimized for concurrent fetch\n* Key file can be rebuilt if needed\n* Inserts are atomic and consistent\n* Data file may be efficiently iterated\n* Key and data files may be on different devices\n* Hardened against algorithmic complexity attacks\n* Header-only, no separate library to build\n\n## Description\n\nThis software is close to final. Interfaces are stable.\nFor recent changes see the [CHANGELOG](CHANGELOG.md).\n\nNuDB has been in use for over a year on production servers\nrunning [rippled](https://github.com/ripple/rippled), with\ndatabase sizes over 3 terabytes.\n\n* [Repository](https://github.com/vinniefalco/Beast)\n* [Documentation](http://vinniefalco.github.io/nudb/)\n\n## Requirements\n\n* Boost 1.69 or higher\n* C++11 or greater\n* SSD drive, or equivalent device with high IOPS\n\nThese components are optionally required in order to build the\ntests and examples:\n\n* CMake 3.7.2 or later (optional)\n* Properly configured bjam/b2 (optional)\n\n## Example\n\nThis complete program creates a database, opens the database,\ninserts several key/value pairs, fetches the key/value pairs,\ncloses the database, then erases the database files. Source\ncode for this program is located in the examples directory.\n\n```C++\n#include \u003cnudb/nudb.hpp\u003e\n#include \u003ccstddef\u003e\n#include \u003ccstdint\u003e\n\nint main()\n{\n    using namespace nudb;\n    std::size_t constexpr N = 1000;\n    using key_type = std::uint32_t;\n    error_code ec;\n    auto const dat_path = \"db.dat\";\n    auto const key_path = \"db.key\";\n    auto const log_path = \"db.log\";\n    create\u003cxxhasher\u003e(\n        dat_path, key_path, log_path,\n        1,\n        make_salt(),\n        sizeof(key_type),\n        block_size(\".\"),\n        0.5f,\n        ec);\n    store db;\n    db.open(dat_path, key_path, log_path, ec);\n    char data = 0;\n    // Insert\n    for(key_type i = 0; i \u003c N; ++i)\n        db.insert(\u0026i, \u0026data, sizeof(data), ec);\n    // Fetch\n    for(key_type i = 0; i \u003c N; ++i)\n        db.fetch(\u0026i,\n            [\u0026](void const* buffer, std::size_t size)\n        {\n            // do something with buffer, size\n        }, ec);\n    db.close(ec);\n    erase_file(dat_path);\n    erase_file(key_path);\n    erase_file(log_path);\n}\n```\n\n## Building\n\nNuDB is header-only so there are no libraries to build. To use it in your\nproject, simply copy the NuDB sources to your project's source tree\n(alternatively, bring NuDB into your Git repository using the\n`git subtree` or `git submodule` commands). Then, edit your build scripts\nto add the `include/` directory to the list of paths checked by the C++\ncompiler when searching for includes. NuDB `#include` lines will look\nlike this:\n\n```\n#include \u003cnudb/nudb.hpp\u003e\n```\n\nTo link your program successfully, you'll need to add the Boost.Thread and\nBoost.System libraries to link with. Please visit the Boost documentation\nfor instructions on how to do this for your particular build system.\n\nNuDB tests require Beast, and the benchmarks require RocksDB. These projects\nare linked to the repository using git submodules. Before building the tests\nor benchmarks, these commands should be issued at the root of the repository:\n\n```\ngit submodule init\ngit submodule update\n```\n\nFor the examples and tests, NuDB provides build scripts for Boost.Build (b2)\nand CMake. To generate build scripts using CMake, execute these commands at\nthe root of the repository (project and solution files will be generated\nfor Visual Studio users):\n\n```\ncd bin\ncmake ..                                    # for 32-bit Windows build\n\ncd ../bin64\ncmake ..                                    # for Linux/Mac builds, OR\ncmake -G\"Visual Studio 14 2015 Win64\" ..    # for 64-bit Windows builds\n```\n\nTo build with Boost.Build, it is necessary to have the b2 executable\nin your path. And b2 needs to know how to find the Boost sources. The\neasiest way to do this is make sure that the version of b2 in your path\nis the one at the root of the Boost source tree, which is built when\nrunning `bootstrap.sh` (or `bootstrap.bat` on Windows).\n\nOnce b2 is in your path, simply run b2 in the root of the Beast\nrepository to automatically build the required Boost libraries if they\nare not already built, build the examples, then build and run the unit\ntests.\n\nOn OSX it may be necessary to pass \"toolset=clang\" on the b2 command line.\nAlternatively, this may be site in site-config.jam or user-config.jam.\n\nThe files in the repository are laid out thusly:\n\n```\n./\n    bench/          Holds the benchmark sources and scripts\n    bin/            Holds executables and project files\n    bin64/          Holds 64-bit Windows executables and project files\n    examples/       Holds example program source code\n    extras/         Additional APIs, may change\n    include/        Add this to your compiler includes\n        nudb/\n    test/           Unit tests and benchmarks\n    tools/          Holds the command line tool sources\n```\n\n## Algorithm\n\nThree files are used.\n\n* The data file holds keys and values stored sequentially and size-prefixed.\n* The key file holds a series of fixed-size bucket records forming an on-disk\n  hash table.\n* The log file stores bookkeeping information used to restore consistency when\nan external failure occurs.\n\nIn typical cases a fetch costs one I/O cycle to consult the key file, and if the\nkey is present, one I/O cycle to read the value.\n\n### Usage\n\nCallers must define these parameters when _creating_ a database:\n\n* `KeySize`: The size of a key in bytes.\n* `BlockSize`: The physical size of a key file record.\n\nThe ideal block size matches the sector size or block size of the\nunderlying physical media that holds the key file. Functions are\nprovided to return a best estimate of this value for a particular\ndevice, but a default of 4096 should work for typical installations.\nThe implementation tries to fit as many entries as possible in a key\nfile record, to maximize the amount of useful work performed per I/O.\n\n* `LoadFactor`: The desired fraction of bucket occupancy\n\n`LoadFactor` is chosen to make bucket overflows unlikely without\nsacrificing bucket occupancy. A value of 0.50 seems to work well with\na good hash function.\n\nCallers must also provide these parameters when a database is _opened:_\n\n* `Appnum`: An application-defined integer constant which can be retrieved\nlater from the database [TODO].\n* `AllocSize`: A significant multiple of the average data size.\n\nMemory is recycled to improve performance, so NuDB needs `AllocSize` as a\nhint about the average size of the data being inserted. For an average data size\nof 1KB (one kilobyte), `AllocSize` of sixteen megabytes (16MB) is sufficient. If\nthe `AllocSize` is too low, the memory recycler will not make efficient use of\nallocated blocks.\n\nTwo operations are defined: `fetch`, and `insert`.\n\n#### `fetch`\n\nThe `fetch` operation retrieves a variable length value given the\nkey. The caller supplies a factory used to provide a buffer for storing\nthe value. This interface allows custom memory allocation strategies.\n\n#### `insert`\n\n`insert` adds a key/value pair to the store. Value data must contain at least\none byte. Duplicate keys are disallowed. Insertions are serialized, which means\n[TODO].\n\n### Implementation\n\nAll insertions are buffered in memory, with inserted values becoming\nimmediately discoverable in subsequent or concurrent calls to fetch.\nPeriodically, buffered data is safely committed to disk files using\na separate dedicated thread associated with the database. This commit\nprocess takes place at least once per second, or more often during\na detected surge in insertion activity. In the commit process the\nkey/value pairs receive the following treatment:\n\nAn insertion is performed by appending a value record to the data file.\nThe value record has some header information including the size of the\ndata and a copy of the key; the data file is iteratable without the key\nfile. The value data follows the header. The data file is append-only\nand immutable: once written, bytes are never changed.\n\nInitially the hash table in the key file consists of a single bucket.\nAfter the load factor is exceeded from insertions, the hash table grows\nin size by one bucket by doing a \"split\". The split operation is the\n[linear hashing algorithm](http://en.wikipedia.org/wiki/Linear_hashing)\nas described by Litwin and Larson.\n\nWhen a bucket is split, each key is rehashed, and either remains in the\noriginal bucket or gets moved to the a bucket appended to the end of\nthe key file.\n\nAn insertion on a full bucket first triggers the \"spill\" algorithm.\n\nFirst, a spill record is appended to the data file, containing header\ninformation followed by the entire bucket record. Then the bucket's size is set\nto zero and the offset of the spill record is stored in the bucket. At this\npoint the insertion may proceed normally, since the bucket is empty. Spilled\nbuckets in the data file are always full.\n\nBecause every bucket holds the offset of the next spill record in the\ndata file, the buckets form a linked list. In practice, careful\nselection of capacity and load factor will keep the percentage of\nbuckets with one spill record to a minimum, with no bucket requiring\ntwo spill records.\n\nThe implementation of fetch is straightforward: first the bucket in the\nkey file is checked, then each spill record in the linked list of\nspill records is checked, until the key is found or there are no more\nrecords. As almost all buckets have no spill records, the average\nfetch requires one I/O (not including reading the value).\n\nOne complication in the scheme is when a split occurs on a bucket that\nhas one or more spill records. In this case, both the bucket being split\nand the new bucket may overflow. This is handled by performing the\nspill algorithm for each overflow that occurs. The new buckets may have\none or more spill records each, depending on the number of keys that\nwere originally present.\n\nBecause the data file is immutable, a bucket's original spill records\nare no longer referenced after the bucket is split. These blocks of data\nin the data file are unrecoverable wasted space. Correctly configured\ndatabases can have a typical waste factor of 1%, which is acceptable.\nThese unused bytes can be removed by visiting each value in the value\nfile using an off-line process and inserting it into a new database,\nthen delete the old database and use the new one instead.\n\n### Recovery\n\nTo provide atomicity and consistency, a log file associated with the\ndatabase stores information used to roll back partial commits.\n\n### Iteration\n\nEach record in the data file is prefixed with a header identifying\nwhether it is a value record or a spill record, along with the size of\nthe record in bytes and a copy of the key if it's a value record, so values can\nbe iterated by incrementing a byte counter. A key file can be regenerated from\njust the data file by iterating the values and performing the key\ninsertion algorithm.\n\n### Concurrency\n\nLocks are never held during disk reads and writes. Fetches are fully\nconcurrent, while inserts are serialized. Inserts fail on duplicate\nkeys, and are atomic: they either succeed immediately or fail.\nAfter an insert, the key is immediately visible to subsequent fetches.\n\n### Formats\n\nAll integer values are stored as big endian. The uint48_t format\nconsists of 6 bytes.\n\n#### Key File\n\nThe Key File contains the Header followed by one or more\nfixed-length Bucket Records.\n\n#### Header (104 bytes)\n\n    char[8]         Type            The characters \"nudb.key\"\n    uint16          Version         Holds the version number\n    uint64          UID             Unique ID generated on creation\n    uint64          Appnum          Application defined constant\n    uint16          KeySize         Key size in bytes\n\n    uint64          Salt            A random seed\n    uint64          Pepper          The salt hashed\n    uint16          BlockSize       Size of a file block in bytes\n\n    uint16          LoadFactor      Target fraction in 65536ths\n\n    uint8[56]       Reserved        Zeroes\n    uint8[]         Reserved        Zero-pad to block size\n\n`Type` identifies the file as belonging to nudb. `UID` is\ngenerated randomly when the database is created, and this value\nis stored in the data and log files as well - it's used\nto determine if files belong to the same database. `Salt` is\ngenerated when the database is created and helps prevent\ncomplexity attacks; it is prepended to the key material\nwhen computing a hash, or used to initialize the state of\nthe hash function. `Appnum` is an application defined constant\nset when the database is created. It can be used for anything,\nfor example to distinguish between different data formats.\n\n`Pepper` is computed by hashing `Salt` using a hash function\nseeded with the salt. This is used to fingerprint the hash\nfunction used. If a database is opened and the fingerprint\ndoes not match the hash calculation performed using the template\nargument provided when constructing the store, an exception\nis thrown.\n\nThe header for the key file contains the File Header followed by\nthe information above. The Capacity is the number of keys per\nbucket, and defines the size of a bucket record. The load factor\nis the target fraction of bucket occupancy.\n\nNone of the information in the key file header or the data file\nheader may be changed after the database is created, including\nthe Appnum.\n\n#### Bucket Record (fixed-length)\n\n    uint16              Count           Number of keys in this bucket\n    uint48              Spill           Offset of the next spill record or 0\n    BucketEntry[]       Entries         The bucket entries\n\n#### Bucket Entry\n\n    uint48              Offset          Offset in data file of the data\n    uint48              Size            The size of the value in bytes\n    uint48              Hash            The hash of the key\n\n### Data File\n\nThe Data File contains the Header followed by zero or more\nvariable-length Value Records and Spill Records.\n\n#### Header (92 bytes)\n\n    char[8]             Type            The characters \"nudb.dat\"\n    uint16              Version         Holds the version number\n    uint64              UID             Unique ID generated on creation\n    uint64              Appnum          Application defined constant\n    uint16              KeySize         Key size in bytes\n    uint8[64]           (reserved)      Zeroes\n\nUID contains the same value as the salt in the corresponding key\nfile. This is placed in the data file so that key and value files\nbelonging to the same database can be identified.\n\n#### Data Record (variable-length)\n\n    uint48              Size            Size of the value in bytes\n    uint8[KeySize]      Key             The key.\n    uint8[Size]         Data            The value data.\n\n#### Spill Record (fixed-length)\n\n    uint48              Zero            All zero, identifies a spill record\n    uint16              Size            Bytes in spill bucket (for skipping)\n    Bucket              SpillBucket     Bucket Record\n\n#### Log File\n\nThe Log file contains the Header followed by zero or more fixed size\nlog records. Each log record contains a snapshot of a bucket. When a\ndatabase is not closed cleanly, the recovery process applies the log\nrecords to the key file, overwriting data that may be only partially\nupdated with known good information. After the log records are applied,\nthe data and key files are truncated to the last known good size.\n\n#### Header (62 bytes)\n\n    char[8]             Type            The characters \"nudb.log\"\n    uint16              Version         Holds the version number\n    uint64              UID             Unique ID generated on creation\n    uint64              Appnum          Application defined constant\n    uint16              KeySize         Key size in bytes\n\n    uint64              Salt            A random seed.\n    uint64              Pepper          The salt hashed\n    uint16              BlockSize       Size of a file block in bytes\n\n    uint64              KeyFileSize     Size of key file.\n    uint64              DataFileSize    Size of data file.\n\n#### Log Record\n\n    uint64_t            Index           Bucket index (0-based)\n    Bucket              Bucket          Compact Bucket record\n\nCompact buckets include only Size entries. These are primarily\nused to minimize the volume of writes to the log file.\n\n## License\n\nDistributed under the Boost Software License, Version 1.0.\n(See accompanying file [LICENSE_1_0.txt](LICENSE_1_0.txt) or copy at\nhttp://www.boost.org/LICENSE_1_0.txt)\n\n## Contact\n\nPlease report issues or questions here:\nhttps://github.com/vinniefalco/NuDB/issues\n","funding_links":[],"categories":["C++"],"sub_categories":[],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FCPPAlliance%2FNuDB","html_url":"https://awesome.ecosyste.ms/projects/github.com%2FCPPAlliance%2FNuDB","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FCPPAlliance%2FNuDB/lists"}