{"id":19157774,"url":"https://github.com/coyorkdow/wisckey","last_synced_at":"2025-05-07T08:35:16.382Z","repository":{"id":51514059,"uuid":"358936799","full_name":"coyorkdow/wisckey","owner":"coyorkdow","description":"WiscKey is a highly SSD optimized key-value storage based on LevelDB.","archived":false,"fork":false,"pushed_at":"2021-12-25T15:37:12.000Z","size":1794,"stargazers_count":31,"open_issues_count":2,"forks_count":3,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-04-19T20:53:29.400Z","etag":null,"topics":["leveldb","paper-implementations","storage","wisckey"],"latest_commit_sha":null,"homepage":"","language":"C++","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"bsd-3-clause","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/coyorkdow.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2021-04-17T17:07:34.000Z","updated_at":"2025-03-23T18:51:47.000Z","dependencies_parsed_at":"2022-09-13T19:10:32.618Z","dependency_job_id":null,"html_url":"https://github.com/coyorkdow/wisckey","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/coyorkdow%2Fwisckey","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/coyorkdow%2Fwisckey/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/coyorkdow%2Fwisckey/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/coyorkdow%2Fwisckey/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/coyorkdow","download_url":"https://codeload.github.com/coyorkdow/wisckey/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":252843544,"owners_count":21812896,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["leveldb","paper-implementations","storage","wisckey"],"created_at":"2024-11-09T08:42:03.145Z","updated_at":"2025-05-07T08:35:16.361Z","avatar_url":"https://github.com/coyorkdow.png","language":"C++","funding_links":[],"categories":[],"sub_categories":[],"readme":"**[WiscKey](https://www.usenix.org/conference/fast16/technical-sessions/presentation/lu) is a highly SSD optimized\nkey-value storage based on LevelDB, presented in a paper published in 14th USENIX Conference on File and Storage\nTechnologies in Feb 2016. This is my implementation derived from LevelDB v1.23.**\n\n**[WiscKey](https://www.usenix.org/conference/fast16/technical-sessions/presentation/lu)\n是一个对SSD高度优化的基于LevelDB的KV存储，发表在2016年2月第14届USENIX文件和存储技术会议上。这是我从LevelDB v1.23派生出的对WiscKey的实现。**\n\n| [LevelDB Readme file](./LevelDB_README.md)| [LevelDB描述文档](./LevelDB_README.md)|\n| :----| ----:|\n\n---\n\n# Feature\n\n* Separate keys from values, only keys are stored in LSM-tree, while values are stored in value-log file (vlog).\n\n* Highly write performance optimized on SSD. When benchmarked with default configuration, Wisckey is double faster than\n  leveldb on sequential writing and treble faster on random writing.\n\n#### LevelDB vs Wisckey\n\n* Environment.\n\n```\n  LevelDB:    version 1.23\n  CPU:        8 * AMD EPYC 7K62 48-Core Processor\n  CPUCache:   512 KB\n  Keys:       16 bytes each\n  Values:     100 bytes each (50 bytes after compression)\n  Entries:    1000000\n  RawSize:    110.6 MB (estimated)\n  FileSize:   62.9 MB (estimated)\n```\n\n* LevelDB\n\n```\nfillseq      :       1.876 micros/op;   59.0 MB/s\nfillsync     :    4339.058 micros/op;    0.0 MB/s (1000 ops)\nfillrandom   :       4.270 micros/op;   25.9 MB/s\noverwrite    :       6.613 micros/op;   16.7 MB/s\nreadrandom   :       4.233 micros/op; (864322 of 1000000 found)\nreadrandom   :       2.791 micros/op; (864083 of 1000000 found)\nreadseq      :       0.159 micros/op;  696.6 MB/s\nreadreverse  :       0.371 micros/op;  298.3 MB/s\ncompact      : 1360175.000 micros/op;\nreadrandom   :       2.110 micros/op; (864105 of 1000000 found)\nreadseq      :       0.136 micros/op;  816.2 MB/s\nreadreverse  :       0.333 micros/op;  332.2 MB/s\nfill100K     :    2191.526 micros/op;   43.5 MB/s (1000 ops)\ncrc32c       :       1.543 micros/op; 2530.9 MB/s (4K per op)\n```\n\n* WiscKey\n\n```\nfillseq      :       0.999 micros/op;  110.7 MB/s\nfillsync     :     515.485 micros/op;    0.2 MB/s (1000 ops)\nfillrandom   :       1.323 micros/op;   83.6 MB/s\noverwrite    :       1.845 micros/op;   60.0 MB/s\nreadrandom   :       3.420 micros/op; (864322 of 1000000 found)\nreadrandom   :       3.077 micros/op; (864083 of 1000000 found)\nreadseq      :       0.671 micros/op;  164.8 MB/s\nreadreverse  :       0.853 micros/op;  129.6 MB/s\ncompact      :  447341.000 micros/op;\nreadrandom   :       2.406 micros/op; (864105 of 1000000 found)\nreadseq      :       0.638 micros/op;  173.3 MB/s\nreadreverse  :       0.809 micros/op;  136.8 MB/s\nfill100K     :      92.249 micros/op; 1034.0 MB/s (1000 ops)\ncrc32c       :       1.527 micros/op; 2557.8 MB/s (4K per op)\n```\n\n# 简介\n\nWisckey的主要思路是键-值分离。在LSM树中，键是按照顺序存储的，但是值却可以被分离且乱序地管理，只要访问键可以获得键到值的映射即可。\nWisckey将数据的值分离地存储在日志中（称为value-log，简称vlog），LSM树中仅保存对应值在vlog文件中的地址，在查询时先获取地址，然后再到文件中读取数据。\n下图示意了键-值分离存储的设计。关系型数据库引擎在实现索引时有着相同的设计：为索引建立的B+树仅存储对应行的指针或主键，当索引覆盖无效时，通过索引的查询需要在表中进行二次查找。\n\n![键-值分离的设计](./pics/wisckey_figure_1.png)\n\nLSM树的操作开销主要来源于Compaction，这是当LSM树某一层的SSTable过大时将数据压向下一层的操作。因为通常来说键的大小要小于值，\n因此把值从LSM树中分离出来能显著减小LSM树的大小，降低Compaction的开销。这能大大提高写入数据的性能，同时更少的额外写操作（称之为写放大）有利于提升存储设备寿命，\n减少性能损耗，这对于SSD尤为重要。\n\n在查询时Wisckey需要访问vlog文件，这次额外的随机读操作会降低查询的性能。但是较之传统的存储设备，SSD有着更好的随机读性能；同时Wisckey具有更小的LSM树和强度更小的Compaction，\n在更小的LSM树上进行查询耗时更少，Compaction的影响也被降低。因此Wisckey的随机查询性能并不会弱于LevelDB。然而在顺序查询的场景下，因为LevelDB中值和键在一起有序存放，\n通过LevelDB的迭代器进行范围查询可以顺序地读到数据，但是Wisckey需要随机读取vlog文件，这降低了Wisckey的顺序查询速度。","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcoyorkdow%2Fwisckey","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fcoyorkdow%2Fwisckey","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcoyorkdow%2Fwisckey/lists"}