{"id":20596820,"url":"https://github.com/alexander-akhmetov/mdb","last_synced_at":"2025-04-14T23:51:56.659Z","repository":{"id":41377469,"uuid":"145947957","full_name":"alexander-akhmetov/mdb","owner":"alexander-akhmetov","description":"LSM tree based key-value database","archived":false,"fork":false,"pushed_at":"2024-03-05T11:21:59.000Z","size":115,"stargazers_count":29,"open_issues_count":0,"forks_count":8,"subscribers_count":2,"default_branch":"master","last_synced_at":"2025-04-14T23:51:43.415Z","etag":null,"topics":["database","go","key-value","learning","lsm-storage","lsm-tree","nobodyreadstags"],"latest_commit_sha":null,"homepage":"","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"other","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/alexander-akhmetov.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2018-08-24T05:50:05.000Z","updated_at":"2025-01-31T11:25:10.000Z","dependencies_parsed_at":"2024-11-16T08:19:24.550Z","dependency_job_id":"9f087893-2def-4637-b121-d410f15757e5","html_url":"https://github.com/alexander-akhmetov/mdb","commit_stats":null,"previous_names":[],"tags_count":1,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/alexander-akhmetov%2Fmdb","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/alexander-akhmetov%2Fmdb/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/alexander-akhmetov%2Fmdb/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/alexander-akhmetov%2Fmdb/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/alexander-akhmetov","download_url":"https://codeload.github.com/alexander-akhmetov/mdb/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248981260,"owners_count":21193144,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["database","go","key-value","learning","lsm-storage","lsm-tree","nobodyreadstags"],"created_at":"2024-11-16T08:18:52.728Z","updated_at":"2025-04-14T23:51:56.618Z","avatar_url":"https://github.com/alexander-akhmetov.png","language":"Go","funding_links":[],"categories":[],"sub_categories":[],"readme":"# mdb\n\n[![Go Report Card](https://goreportcard.com/badge/github.com/alexander-akhmetov/mdb)](https://goreportcard.com/report/github.com/alexander-akhmetov/mdb)\n\nA simple key-value storage.\n\nIt was created for learning purposes: I wanted to learn a bit of Go and create my own small database.\n\nNot intended for production use. :)\n\nImplemented storage types:\n\n* In-memory\n* File\n* Indexed file  (simple hash map, no sparse indexes)\n* Log structured merge tree ([LSM tree](https://en.wikipedia.org/wiki/Log-structured_merge-tree))\n\n## Usage\n\nBy default it uses `lsmt.Storage`, but you can change this in the [main.go](db/main.go).\n\n### Command-line interface\n\n```bash\n~/ » make run\n\ngo run cmd/*.go -i\n\n######### Started #########\n\n\u003e\u003e get key\nvalue='', exists=false\n\n\u003e\u003e set key value\nSaved    key=value\n\n\u003e\u003e get key\nvalue='value', exists=true\n```\n\n### Go\n\n```go\npackage main\n\nimport (\n    \"github.com/alexander-akhmetov/mdb/pkg\"\n    \"github.com/alexander-akhmetov/mdb/pkg/lsmt\"\n)\n\n\nfunc main() {\n    db := storage.NewLSMTStorage(lsmt.StorageConfig{\n        WorkDir:               \"./lsmt_data/\",\n        CompactionEnabled:     true,\n        MinimumFilesToCompact: 2,\n        MaxMemtableSize:       65536,\n        SSTableReadBufferSize: 4096,\n    })\n    defer db.Stop()\n\n    db.Set(\"key_1\", \"value_1\")\n\n    value, found := db.Get(\"key_1\")\n\n    println(\"Found:\", found)\n    println(\"Value:\", value)\n}\n```\n\nMore information about all these configuration options can be found in the `lsmt.Storage` section below.\n\n## Internals\n\nThe database supports different storage types.\n\n### memory.Storage\n\nIt's a simple hash map that holds everything in memory.\n\n### file.Storage\n\nIt stores all information in a file. When you add a new entry, it simply appends the key and value to the file. So it's very fast to add new information. However, when you try to retrieve a key, it scans the entire file (starting from the beginning, not the end) to find the latest key. Therefore, reading is slow.\n\n### indexedfile.Storage\n\nThis is a FileStorage with a simple index (hash map). When you add a new key, it saves the offset in bytes to the map in memory. To process the get command, it checks the index, finds the offset in bytes, and reads only a piece of the file. Writing and reading are fast, but you need a lot of memory to keep all keys in it.\n\n### lsmt.Storage\n\nIt stores all data in sorted string tables (SSTables), which are essentially binary files. It supports sparse indexes, so you don't need a lot of memory to store all your keys like in indexedfile.Storage.\n\nHowever, it will be slower than indexedfile.Storage because it uses a red-black tree to store a sparse index and checks all SSTables when you retrieve a value. This is because it can't determine whether it has this key without checking the SSTables on disk. It could probably use a Bloom filter for that.\n\n```none\n                     +------------+\n                     |  Client    |\n                     +------------+\n                        |\n                        | GET|SET\n                        v\n               +------------------------+              +---------------------+\n               | +--------------------+ |              | +-----------------+ |\n               | |  Append only log   | |              | |   memtable 10   | |\n               | +--------------------+ | when memtable| +-----------------+ |\n               | +--------------------+ | is too big   |       ...           |\n               | |     memtable       | |------------\u003e | +-----------------+ |\n               | +--------------------+ |              | |   memtable 1    | |\n               |                        |              | +-----------------+ |\n               |                        |              |                     |\n               | writer                 |              | flush queue         |\n               +------------------------+              +---------------------+\n                                                             ^\n                                                             | flusher dumps memtables\n                                                             | to disk in the background (as sstables)\n                                                             v\n                +------------+                         +---------------------+\n                | Compaction |                         |       flusher       |\n                +------------+                         +---------------------+\n                    ^  periodical compaction process                 |\n                    |  merges different small sstable files          |\n                    v  into a big one and removes unnnecessary data  v\n               +-------------------------------------------------------------+\n               | +------------+  +-----------+                 +-----------+ |\n               | | sstable 10 |  | sstable 9 |     ...         | sstable 0 | |\n               | +------------+  +-----------+                 +-----------+ |\n               |                                                             |\n               |                                                             |\n               | SSTables storage                                            |\n               +-------------------------------------------------------------+\n\n```\n\nMain parts:\n\n* Writer (memtable)\n* Flush queue (list of memtables)\n* Flusher (dumps a memtable to a disk)\n* SSTables storage (main storage for the data)\n* Compaction (background process to remove old keys that were updated)\n\n#### GET process\n\n1. Check memtable\n2. Check memtables in flush queue\n3. Check SSTables\n\nIt checks all these parts in this order to be sure that it returns the latest version of the key.\nEach SSTable has its own index. It can be sparse: it will not keep each key-offset pair in the index,\nbut it will store keys every N bytes. We can do this because SSTable files are sorted and read-only. When we need to find a\nkey, we find its offset or closest minimal to this key. After we can load part of the file into memory and find the value for the key.\n\n#### SET\n\n1. Save value to append only log\n2. Save value to memtable\n\n#### Flush\n\nWhen the memtable becomes bigger than some threshold, the core component puts it to the flush queue and initializes a new memtable. \nThe flusher is a background process that checks the queue and dumps memtables as SSTables to disk.\n\n#### Compaction\n\nIt's a periodical background process that merges small SSTable files into a larger one and removes old key-value pairs that can be removed.\n\n#### SSTables storage\n\nIt's a disk storage. During start-up, mdb checks this folder, registers all files, and builds indexes. \nFiles are read-only; mdb never changes them. It can only merge them into a larger file, but without modifying old files.\n\n#### File format\n\nBinary file format:\n\n```none\n[entry_type: 1byte][key_length: 4bytes][value_length: 4bytes][key][value]\n\nentry_type:\n\n* 0 - value\n\n```\n\n##### Configuration\n\n```none\nCompactionEnabled     bool  // Enable/disable the background compaction process\nMinimumFilesToCompact int   // How many files are needed to start the compaction process\nMaxMemtableSize       int64 // max size for memtable\nMaxCompactFileSize    int64 // Do not compact files bigger than this size\nSSTableReadBufferSize int   // Read buffer size: the database will build indexes every\n                            // \u003cSSTableReadBufferSize\u003e bytes. If you want to have a non-sparse index\n                            // put 1 here\n```\n\n#### performance test mode\n\nStart performance test: insert 10000 keys (`-k 10000`) and then check them (`-c`):\n\n```bash\ngo run cmd/*.go -p -k 10000 -c 2\u003e\u00261 | grep -v 'Adding'\n```\n\nIt will print something like that (it prints `Inserted: \u003ccount\u003e` every second):\n\n```none\n[DEBUG] Read buffer size: 65536\n[DEBUG] Maximum memtable size: 16384\n[DEBUG] Starting lsmt storage\n[DEBUG] Creating dir lsmt_data/sstables\n[DEBUG] Creating dir lsmt_data/aolog_tf\n[DEBUG] Creating dir lsmt_data/tmp\n[DEBUG] Initializing a new SSTable instance...\n[DEBUG] initialized sstables: 16\n[DEBUG] Restoring flush queue...\n[DEBUG] Flush queue has been restored with size= 0\n[DEBUG] AOLog file exists, restoring...\n[DEBUG] Restored 11710 entries\n[DEBUG] Storage ready\n[DEBUG] Started flusher process\n[DEBUG] Started compaction process\n[DEBUG] Started compaction process\n\n[DEBUG] Inserted: 7440\n[DEBUG] Inserted: 2560\n\n[DEBUG] OK. Inserted keys checked: 10000\n```\n\n## TODO\n\n* delete command\n* bloom filter\n* range queries\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Falexander-akhmetov%2Fmdb","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Falexander-akhmetov%2Fmdb","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Falexander-akhmetov%2Fmdb/lists"}