{"id":16369713,"url":"https://github.com/dcsunset/kv-lsmt","last_synced_at":"2026-03-03T17:30:16.649Z","repository":{"id":180009440,"uuid":"661791827","full_name":"DCsunset/kv-lsmt","owner":"DCsunset","description":"A fast KV store inspired by LSM Tree","archived":false,"fork":false,"pushed_at":"2023-07-10T01:46:20.000Z","size":1476,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":2,"default_branch":"main","last_synced_at":"2024-12-31T09:44:46.180Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":"C++","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/DCsunset.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2023-07-03T16:52:12.000Z","updated_at":"2023-10-11T18:55:40.000Z","dependencies_parsed_at":null,"dependency_job_id":"7003f8bb-a35b-4628-985d-a758efc3386a","html_url":"https://github.com/DCsunset/kv-lsmt","commit_stats":null,"previous_names":["dcsunset/kv-lsmt"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/DCsunset%2Fkv-lsmt","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/DCsunset%2Fkv-lsmt/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/DCsunset%2Fkv-lsmt/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/DCsunset%2Fkv-lsmt/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/DCsunset","download_url":"https://codeload.github.com/DCsunset/kv-lsmt/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":239885449,"owners_count":19713309,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-10-11T02:56:06.418Z","updated_at":"2025-02-20T17:40:59.422Z","avatar_url":"https://github.com/DCsunset.png","language":"C++","funding_links":[],"categories":[],"sub_categories":[],"readme":"# KV store based on LSM Tree\n\nThis project implements a KV store inspired by LSM Tree.\n\n## Design\n\nThe basic storage unit is a table (`SSTable`).\nFor all the tables, once a key-value pair is written,\nit can't be modified anymore.\nThe way to modify existing keys is to overwrite previous values.\nWhen the table's size grows to a certain threshold,\na new table is created.\n\nFor read operations, we need to go through tables from latest to oldest to search for the key.\nOnce it's found, we can immediately return as it's the current value.\nThe key doesn't exist if and only if no table contains it.\n\nOne benefit of such design is that writes can be persisted to disk by simply appending an entry in the log,\nwhich greatly increase the efficiency of writes.\nReads that touch recently modified data will also be efficient.\n\nDefinition and implementation of the table is located in file `sstable.h` and `sshtable.cc`.\nTo improve the lookup process, each table is divided into index part and data part,\nwhich are stored in different files.\nThe index part includes the keys and the offsets of corresponding value in the data part.\nSo when looking up a key, it's only necessary to read the index file until we find it.\nIn this way, we can preload many keys (without its data) into the memory and greatly improve the efficiency of reads.\n\n## Crash Consistency\n\nThis data store guarantees crash consistency.\nThe way to implement it is that for each write, it must be persisted to disk before sending response to the API caller.\nFor this data store, writes can be simply appended to the index file and data file.\nOne detail is that length of the key or the value is written before the key or the value itself.\nThis makes sure that incomplete writes can be detected by checking the length and the actual size of the key or data,\nwhich in turn guarantees crash consistency.\n\n## Table Management\n\nTo manage a number of existing `SSTable`, a `DataStore` class is implemented.\nIt only stores the complete information (index and data) of current table in memory for fast read and write.\nFor previous tables, only indexes are kept in memory to reduce the memory usage.\nWhen the size of current table reachs a certain threshold, it will create a new table as the current table,\nand following write operations will be stored in the new table.\n\n## Future Work\n\nKeeping adding new tables is not a good idea when old keys are modified frequenetly.\nThis results in a lot of old entries that will not be read anymore.\nTo solve this issue, periodic consolidation of the old tables is necessary for garbage collection.\n\n## Performance\n\nIn the experiment,\nthe keys each operation accesses follows Zipf distribution.\nWhen `isSkewed` is 0, the $\\theta$ parameter in Zipf is 0.\nOtherwise, $\\theta = 0$.\n\nThe performance is shown in the following table.\n(when isSkewed is 1, it uses Zipf distribution:\n\n| # of Threads | Read Ratio (%) | isSkewed | Throughput (op/s) |\n|--------------|----------------|----------|-------------------|\n| 1            | 0              | 0        | 102k              |\n| 1            | 50             | 0        | 201k              |\n| 1            | 100            | 0        | 2369k             |\n| 1            | 0              | 1        | 136k              |\n| 1            | 50             | 1        | 241k              |\n| 1            | 100            | 1        | 1429k             |\n| 4            | 0              | 0        | 25k               |\n| 4            | 50             | 0        | 79k               |\n| 4            | 100            | 0        | 5197k             |\n| 4            | 0              | 1        | 29k               |\n| 4            | 50             | 1        | 94k               |\n| 4            | 100            | 1        | 3410k             |\n\nAs we can conclude from the table:\n\n- When read ratio is low, using single thread outperforms multiple threads.\n  Otherwise, using multiple threads has higher throughput.\n- When read ration is low, skewed distribution has better performance.\n  Otherwise, non-skewed distribution performs better.\n\nFor the first observation, the reason is that using multiple threads will need to acquire the write lock frequenetly.\nThe exclusive lock prevents concurrent writes and introduces unnecessary overhead.\n\nFor the second observation, the reason is under skewed distribution\noperations tend to access the same set of keys frequenetly,\nIf there are more writes, then reads are very likely to access the data in the current table,\nwithout fetching data from disk, thus outperforms the non-skewed workload.\nHowever, if there are more reads, it's more likely that more reads will access the key in previous tables,\nso the performance is worse compared to a non-skewed workload.\n\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdcsunset%2Fkv-lsmt","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fdcsunset%2Fkv-lsmt","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdcsunset%2Fkv-lsmt/lists"}