{"id":17278284,"url":"https://github.com/kulpreet/libbitcoin-mvcc-database","last_synced_at":"2026-01-18T09:01:40.758Z","repository":{"id":146434978,"uuid":"232906350","full_name":"kulpreet/libbitcoin-mvcc-database","owner":"kulpreet","description":"Experimental in-memory MVCC bitcoin database","archived":false,"fork":false,"pushed_at":"2020-11-21T11:17:58.000Z","size":296,"stargazers_count":0,"open_issues_count":13,"forks_count":0,"subscribers_count":1,"default_branch":"master","last_synced_at":"2025-04-06T09:27:44.164Z","etag":null,"topics":["bitcoin","in-memory-database","libbitcoin"],"latest_commit_sha":null,"homepage":"","language":"C++","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/kulpreet.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2020-01-09T21:08:35.000Z","updated_at":"2020-04-12T09:48:35.000Z","dependencies_parsed_at":null,"dependency_job_id":"0248b42b-c9b5-476c-a154-fb75b25fb8f4","html_url":"https://github.com/kulpreet/libbitcoin-mvcc-database","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/kulpreet/libbitcoin-mvcc-database","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/kulpreet%2Flibbitcoin-mvcc-database","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/kulpreet%2Flibbitcoin-mvcc-database/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/kulpreet%2Flibbitcoin-mvcc-database/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/kulpreet%2Flibbitcoin-mvcc-database/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/kulpreet","download_url":"https://codeload.github.com/kulpreet/libbitcoin-mvcc-database/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/kulpreet%2Flibbitcoin-mvcc-database/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":28534154,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-01-18T00:39:45.795Z","status":"online","status_checked_at":"2026-01-18T02:00:07.578Z","response_time":98,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["bitcoin","in-memory-database","libbitcoin"],"created_at":"2024-10-15T09:11:17.266Z","updated_at":"2026-01-18T09:01:40.721Z","avatar_url":"https://github.com/kulpreet.png","language":"C++","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Build\n\nUses cmake. Requires libbitcoin-system.\n\n`mkdir build \u0026\u0026 cd build`\n`PKG_CONFIG_PATH=path/to/libbitcoin-system/lib/pkgconfig/ cmake -DCMAKE_BUILD_TYPE=Release|Debug ../`\n`make \u0026\u0026 make test`\n\n## Requirements\n\nWe need c++17 to enable `alignas` for use in raw memory block pool. I\ncouldn't make `class alignas(1 \u003c\u003c 20) raw_block` work on c++11 or\nc++14. Using jemalloc didn't help either.\n\n# Goals\n\n1. In memory multi version concurrency control.\n2. Handle databases larger than RAM available.\n3. Bias decisions in favour of faster block validation.\n4. API Compatibility with libbitcoin-database.\n\n## Background\n\nlibbitcoin-database uses mmap and that has lead to some problems in\ncase of a hard shutdown. Also, depending on the VM subsystem to keep\nUTXOs in memory has run into problems. This is highlighted by our need\nto build a utxo cache.\n\nThe design of this database is motivated by in-memory MVCC databases\nlike Microsoft Hackathon, MemSQL, HyPer and NuoDB.\n\n## Larger than memory\n\nMost users of libbitcoin first try it on their smaller machines with\nless RAM than required to hold all the blockchain data and indexes in\nmemory.\n\nThis database periodically runs garbage collection to move stale data\nlike spent transaction outputs and blocks with no UTXOs left to spend,\nout to the disk.\n\nThe RAM then holds the most relevant data possible, which further\ntakes away the need for a cache in front of this database. After all,\nthis is an in memory database.\n\n## Architecture Key Decisions\n\nThe database management system will provide access to records using\nMulti Version Concurrency Control. The ideal is to avoid depending on\nmutexes for thread synchronization.\n\nAt a high level, the following design decisions motivate the rest of\nthe architecture. These key decisions are informed by the \"the best\npaper ever on in-memory multi version concurrency control\", [Y. Wu, et\nal., An Empirical Evaluation of In-Memory Multi-Version Concurrency\nControl, in VLDB,\n2017](https://15721.courses.cs.cmu.edu/spring2020/papers/03-mvcc1/wu-vldb2017.pdf)\n\n1. Concurrency Control Protocol - Multi Version Timestamp Ordering to\n   simplify first implementation.\n1. Version Storage - Delta storage so we don't replicate bulky\n   transaction data into older versions.\n2. Garbage Collection - Background Vacuuming to move data from RAM to\n   disk. Pointer swizzling to track the location of the records.\n3. Index Management - Indexes records directly by primary key pointing\n   to the master record on the main table, from where we can follow\n   the delta storage, as required. For secondary index, maintain an\n   indirection layer from secondary key to primary key, which can then\n   be used to fetch the master record of the version chain.\n\n### MVTO\n\nFirst writer wins. If a new writer can't update a record, then the\nupdate operation returns an error. The blockchain layer will need to\nretry the transaction. This might already be handled but needs to be\nchecked. The hunch is that the blockchain layer handles this\nindirectly by depending on the sync protocol fetching the entire block\nagain.\n\nThis cost of retrying a block accept protocol seems expensive, but a\ntransaction rollback will happen very rarely and thus should be\namortised. Need to measure this once we have a working implementation.\n\n### Delta Storage\n\nIndex points to master record which in turn points to a delta\nrecord. For example, for block database, the delta record only has to\nstore the status field.\n\nThe master record always has the latest record, so index doesn't need\nto be updated on each record update.\n\nGarbage collector will delete the stale delta versions that no running or\nfuture transactions will need.\n\n### Garbage Collection\n\nPeriodically run garbage collector, or when a preset fraction of the\nUTXO store is full.\n\nGarbage collector will move spent UTXOs and blocks older than a preset\nnumber of blocks to disk.\n\nDepending on the memory available and configured to be used by the\nvarious in memory tables, the database layer can be tweaked to work\nfor small laptops to big multi core servers. That is one of the goals.\n\n### Index Management\n\nUse [libcuckoo](https://github.com/efficient/libcuckoo) for providing\na fast, concurrent hash table as the index manager.\n\nAll indexes are limited to hash tables, so there is no support for\nsequential scans over keys. This can be changed later to Bw-Tree to\nsupport sequential scans.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fkulpreet%2Flibbitcoin-mvcc-database","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fkulpreet%2Flibbitcoin-mvcc-database","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fkulpreet%2Flibbitcoin-mvcc-database/lists"}