{"id":26546861,"url":"https://github.com/changsongl/lsm","last_synced_at":"2026-05-18T11:04:54.614Z","repository":{"id":112080576,"uuid":"381315572","full_name":"changsongl/lsm","owner":"changsongl","description":"A key/value db implemented by LSM.  借鉴RocksDB的LSM 实现Key/Value数据库。","archived":false,"fork":false,"pushed_at":"2021-07-23T10:42:08.000Z","size":52,"stargazers_count":1,"open_issues_count":0,"forks_count":0,"subscribers_count":2,"default_branch":"main","last_synced_at":"2024-12-18T08:39:37.143Z","etag":null,"topics":["database","lsm","lsm-tree","rocksdb"],"latest_commit_sha":null,"homepage":"","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/changsongl.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2021-06-29T09:50:17.000Z","updated_at":"2023-07-01T11:55:51.000Z","dependencies_parsed_at":null,"dependency_job_id":"c5e351e3-5844-4dde-bdb9-5d5d1531b046","html_url":"https://github.com/changsongl/lsm","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/changsongl%2Flsm","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/changsongl%2Flsm/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/changsongl%2Flsm/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/changsongl%2Flsm/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/changsongl","download_url":"https://codeload.github.com/changsongl/lsm/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":244912802,"owners_count":20530764,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["database","lsm","lsm-tree","rocksdb"],"created_at":"2025-03-22T05:28:08.562Z","updated_at":"2026-05-18T11:04:49.575Z","avatar_url":"https://github.com/changsongl.png","language":"Go","funding_links":[],"categories":[],"sub_categories":[],"readme":"# lsm\n\n### Introduction\nA key/value db implemented by LSM. It is for studying LSM `data structure`.\nI will design the db by the theories, and then compare with other famous\nlsm databases. This README is writen before I write any code, because I \ncan think through every detail before I start.\n\n### What is LSM?\nLSM is `Log-Structured MergeTree`. It is a data structure, but it's not a common\ntree structure like B+ tree. Many popular databases are implemented by LSM.\n\nThe core concept of LSM is to use `sequential IO` to increase writing speed, \nso it has `horizontal and vertical layers` in its design. Because of layers,\nit decreases reading speed. It is popular for More-Write-And-Less-Read situation.\n\n### The LSM architecture (RocksDB)\n![alt LSM arch](./design/img/arch.jpeg)\n\nThere are different layers in the architecture of RocksDB.\n\n1. MemTable: MemTable is an `in-memory` data structure, and it stores the recent data.\n   All keys are in order by the data structure. It is not restricted to any data structure\n   in this layer. For RocksDB, it is using `skip list`. Because it is in the memory,\n   it will lose data for some situation. Therefore, it will use WAL(Write-ahead logging)\n   for data durability. After a failure, it will read the WAL for the backup.\n   \n2. Immutable MemTable: When size of MemTable is increasing, it will turn into an \n   Immutable MemTable. Immutable is also in the memory, and it is a `middle layer` \n   between MemTable and SSTable. After the changing from MemTable, Immutable MemTable \n   will be `read only` and wait to be written to SSTable. All writing requests will \n   be processed by the new MemTable.\n   \n3. SSTable(Sorted String Table): This layer is in the `disk`. For the quick searching,\n   it has key indexing and bloom filters(Check whether a key is in the SSTable). In the\n   SSTable layer, it has many `levels`(l0, l1, l2...), also each level has many files to \n   store the logs. When there are enough files in same level, it will `merge` the files\n   and next level files to decrease the file number in this level. It can cause the\n   cascade situation.\n   ![SSTable key indexing](./design/img/sstable.jpeg)\n   \nLSM keeps all logs in the memory, when the operations are getting large enough,\nthey will be flushed in the SSTable by sequential IO. LSM is log style data \nstructure, and it saves all operations. It doesn't need to change the previous data\nfor faster sequential IO. In the contrast of B+ tree, it will always update the old\ndata by the new data. Therefore, B+ tree is an in-place update structure, and LSM is \nan out-place update structure. \n\n### Drawback of LSM\n1. Read Amplification: Because LSM is log structured, it needs to read through,\n   all layers and all levels to get a key. It can be solved by `merging` many SSTable\n   files. In the same level of SSTable, it can ensure there is no duplicate log.\n   Also, it can use the bloom filter to speed reading.\n\n2. Size Amplification: LSM appends all logs to the disk, and it will not delete any \n   keys. It causes the garbage data in the disk. It also can be solved by the `merging`.\n   After the merging, it will only keep the newest log for the same key.\n   \n3. Write Amplification: For solving read and size Amplification, it will have a background\n   process to merge logs and compaction. For every merge, it repeated writing `useless key` by \n   multiple times. It causes write amplification. This problem can be solved by `leveled\n   compaction`. It makes sure there is only one log of a key among the files in the SSTable level.\n   There is `no duplicate` key log in the same level.\n\n### Design of this project\nContinue...\n\n### Reference\n[LSM Tree-Based engine](https://www.jianshu.com/p/e89cd503c9ae?utm_campaign=hugo)\n[RocksDB](https://zhuanlan.zhihu.com/p/181498475)\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fchangsongl%2Flsm","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fchangsongl%2Flsm","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fchangsongl%2Flsm/lists"}