{"id":18830268,"url":"https://github.com/sopherapps/scdb","last_synced_at":"2026-02-17T17:33:41.799Z","repository":{"id":62116653,"uuid":"545519579","full_name":"sopherapps/scdb","owner":"sopherapps","description":"A very simple and fast key-value store but persisting data to disk, with a \"localStorage-like\" API.","archived":false,"fork":false,"pushed_at":"2023-06-01T20:19:51.000Z","size":360,"stargazers_count":8,"open_issues_count":1,"forks_count":0,"subscribers_count":2,"default_branch":"master","last_synced_at":"2025-08-25T10:58:11.263Z","etag":null,"topics":["cache","disk-cache","key-value-store","localstorage"],"latest_commit_sha":null,"homepage":"https://docs.rs/scdb","language":"Rust","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/sopherapps.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":"docs/CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2022-10-04T14:10:48.000Z","updated_at":"2024-10-27T15:26:49.000Z","dependencies_parsed_at":"2025-04-14T03:42:37.237Z","dependency_job_id":"3aeb2c36-2af6-4375-9d93-aa0a37dc6c4d","html_url":"https://github.com/sopherapps/scdb","commit_stats":null,"previous_names":[],"tags_count":8,"template":false,"template_full_name":null,"purl":"pkg:github/sopherapps/scdb","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sopherapps%2Fscdb","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sopherapps%2Fscdb/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sopherapps%2Fscdb/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sopherapps%2Fscdb/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/sopherapps","download_url":"https://codeload.github.com/sopherapps/scdb/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sopherapps%2Fscdb/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":29551257,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-02-17T14:33:00.708Z","status":"ssl_error","status_checked_at":"2026-02-17T14:32:58.657Z","response_time":100,"last_error":"SSL_connect returned=1 errno=0 peeraddr=140.82.121.5:443 state=error: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["cache","disk-cache","key-value-store","localstorage"],"created_at":"2024-11-08T01:48:19.360Z","updated_at":"2026-02-17T17:33:41.783Z","avatar_url":"https://github.com/sopherapps.png","language":"Rust","funding_links":["https://www.buymeacoffee.com/martinahinJ"],"categories":[],"sub_categories":[],"readme":"# scdb\n\n![CI](https://github.com/sopherapps/scdb/actions/workflows/CI.yml/badge.svg)\n\nA very simple and fast key-value store but persisting data to disk, with a \"localStorage-like\" API.\n\n**scdb may not be production-ready yet. It works, quite well but it requires more rigorous testing.**\n\n## Purpose\n\nComing from front-end web\ndevelopment, [localStorage](https://developer.mozilla.org/en-US/docs/Web/API/Window/localStorage) was always\na convenient way of quickly persisting data to be used later by a given application even after a restart.\nIts API was extremely simple i.e. `localStorage.getItem()`, `localStorage.setItem()`, `localStorage.removeItem()`\n, `localStorage.clear()`.\n\nComing to the backend (or even desktop) development, such an embedded persistent data store with a simple API\nwas hard to come by.\n\nscdb is meant to be like the 'localStorage' of backend and desktop (and possibly mobile) systems.\nOf course to make it a little more appealing, it has some extra features like:\n\n- Time-to-live (TTL) where a key-value pair expires after a given time\n- Non-blocking reads from separate processes, and threads.\n- Fast Sequential writes to the store, queueing any writes from multiple processes and threads.\n- Optional searching of keys that begin with a given subsequence. This option is turned on when `scdb::new()` is called.\n\n## Documentation\n\nFind the following documentation sites, depending on the programming language.\n\n- [rust scdb docs](https://docs.rs/scdb)\n- [python scdb docs](https://github.com/sopherapps/py_scdb)\n\n## Quick Start\n\n- Create a new cargo project\n\n  ```shell\n  cargo new hello_scdb \u0026\u0026 cd hello_scdb\n  ```\n\n- Add scdb to your dependencies in `Cargo.toml` file\n\n  ```toml\n  [dependencies]\n  scdb = { version = \"0.1\" }\n  ```\n\n- Update your `src/main.rs` to the following.\n\n```rust\nuse scdb::Store;\nuse std::thread;\nuse std::time::Duration;\n\n/// Converts a byte array to string\nmacro_rules! to_str {\n    ($arr:expr) =\u003e {\n        std::str::from_utf8($arr).expect(\"bytes to str\")\n    };\n}\n\n/// Prints data from store to the screen in a pretty way\nmacro_rules! pprint_data {\n    ($title:expr, $data:expr) =\u003e {\n        println!(\"\\n\");\n        println!(\"{}\", $title);\n        println!(\"===============\");\n\n        for (k, got) in $data {\n            let got_str = match got {\n                None =\u003e \"None\",\n                Some(v) =\u003e to_str!(v),\n            };\n            println!(\"For key: '{}', str: '{}', raw: '{:?}',\", k, got_str, got);\n        }\n    };\n}\n\nfn main() {\n  // Creat the store. You can configure its `max_keys`, `redundant_blocks` etc. The defaults are usable though.\n  // One very important config is `max_keys`. With it, you can limit the store size to a number of keys.\n  // By default, the limit is 1 million keys\n  let mut store =\n          Store::new(\"db\", Some(1000), Some(1), Some(10), Some(1800), true).expect(\"create store\");\n  let records = [\n    (\"hey\", \"English\"),\n    (\"hi\", \"English\"),\n    (\"salut\", \"French\"),\n    (\"bonjour\", \"French\"),\n    (\"hola\", \"Spanish\"),\n    (\"oi\", \"Portuguese\"),\n    (\"mulimuta\", \"Runyoro\"),\n  ];\n  let updates = [\n    (\"hey\", \"Jane\"),\n    (\"hi\", \"John\"),\n    (\"hola\", \"Santos\"),\n    (\"oi\", \"Ronaldo\"),\n    (\"mulimuta\", \"Aliguma\"),\n  ];\n  let keys: Vec\u003c\u0026str\u003e = records.iter().map(|(k, _)| *k).collect();\n\n  // Setting the values\n  println!(\"Let's insert data\\n{:?}]...\", \u0026records);\n  for (k, v) in \u0026records {\n    let _ = store.set(k.as_bytes(), v.as_bytes(), None);\n  }\n\n  // Getting the values (this is similar to what is in `get_all(\u0026mut store, \u0026keys)` function\n  let data: Vec\u003c(\u0026str, Option\u003cVec\u003cu8\u003e\u003e)\u003e = keys\n          .iter()\n          .map(|k| (*k, store.get(k.as_bytes()).expect(\u0026format!(\"get {}\", k))))\n          .collect();\n  pprint_data!(\"After inserting data\", \u0026data);\n\n  // Setting the values with time-to-live\n  println!(\n    \"\\n\\nLet's insert data with 1 second time-to-live (ttl) for keys {:?}]...\",\n    \u0026keys[3..]\n  );\n  for (k, v) in \u0026records[3..] {\n    let _ = store.set(k.as_bytes(), v.as_bytes(), Some(1));\n  }\n\n  println!(\"We will wait for 1 second to elapse...\");\n  thread::sleep(Duration::from_secs(2));\n\n  let data = get_all(\u0026mut store, \u0026keys);\n  pprint_data!(\"After inserting keys with ttl\", \u0026data);\n\n  // Updating the values\n  println!(\"\\n\\nLet's update with data {:?}]...\", \u0026updates);\n  for (k, v) in \u0026updates {\n    let _ = store.set(k.as_bytes(), v.as_bytes(), None);\n  }\n\n  let data = get_all(\u0026mut store, \u0026keys);\n  pprint_data!(\"After updating keys\", \u0026data);\n\n  // Full-text search by key. It returns array of key-value tuples.\n  let data = store\n          .search(\u0026b\"h\"[..], 0, 0)\n          .expect(\"search for keys starting with h\");\n  println!(\"\\nSearching for keys starting with 'h'\");\n  println!(\"=======================================\", );\n  for (k, v) in \u0026data {\n    // note that to_str! is a custom macro changing byte array to UTF-8 string\n    println!(\"{}: {}\", to_str!(k), to_str!(v))\n  }\n\n  // Search with pagination\n  let data = store\n          .search(\u0026b\"h\"[..], 1, 1)\n          .expect(\"search for keys starting with h\");\n  println!(\"\\nPaginated search for keys starting with 'h'\");\n  println!(\"==============================================\", );\n  println!(\"Skipping 1, returning 1 record only\");\n  println!(\"---\");\n  for (k, v) in \u0026data {\n    // note that to_str! is a custom macro changing byte array to UTF-8 string\n    println!(\"{}: {}\", to_str!(k), to_str!(v))\n  }\n\n  // Deleting some values\n  let keys_to_delete = [\"oi\", \"hi\"];\n  println!(\"\\n\\nLet's delete keys{:?}]...\", \u0026keys_to_delete);\n  for k in keys_to_delete {\n    store\n            .delete(k.as_bytes())\n            .expect(\u0026format!(\"delete key {}\", k));\n  }\n\n  let data = get_all(\u0026mut store, \u0026keys);\n  pprint_data!(\"After deleting keys\", \u0026data);\n\n  // Deleting all values\n  println!(\"\\n\\nClear all data...\");\n  store.clear().expect(\"clear store\");\n\n  let data = get_all(\u0026mut store, \u0026keys);\n  pprint_data!(\"After clearing\", \u0026data);\n}\n\n/// Gets all from store for the given keys\nfn get_all\u003c'a\u003e(store: \u0026mut Store, keys: \u0026Vec\u003c\u0026'a str\u003e) -\u003e Vec\u003c(\u0026'a str, Option\u003cVec\u003cu8\u003e\u003e)\u003e {\n  keys.iter()\n          .map(|k| (*k, store.get(k.as_bytes()).expect(\u0026format!(\"get {}\", k))))\n          .collect()\n}\n```\n\n- Run the `main.rs` file\n\n  ```shell\n  cargo run\n  ```\n\n## Contributing\n\nContributions are welcome. The docs have to maintained, the code has to be made cleaner, more idiomatic and faster,\nand there might be need for someone else to take over this repo in case I move on to other things. It happens!\n\nPlease look at the [CONTRIBUTIONS GUIDELINES](./docs/CONTRIBUTING.md)\n\nYou can also look in the [./docs](./docs) folder to get up to speed with the internals of scdb e.g.\n\n- [database file format](./docs/DB_FILE_FORMAT.md)\n- [how it works](./docs/HOW_IT_WORKS.md)\n- [inverted index file format](./docs/INVERTED_INDEX_FILE_FORMAT.md)\n- [how the search works](./docs/HOW_INVERTED_INDEX_WORKS.md)\n\n## Bindings\n\nscdb is meant to be used in multiple languages of choice. However, the bindings for most of them are yet to be\ndeveloped.\nHere are those that have been developed:\n\n- [x] [rust](https://crates.io/crates/scdb)\n- [x] [python](https://github.com/sopherapps/py_scdb)\n- [x] [golang](https://github.com/sopherapps/go-scdb)\n\n### TODO:\n\n- [ ] compare benchmarks with those of redis, sqlite, lmdb etc.\n\n### How to Test\n\n- Make sure you have [rust](https://www.rust-lang.org/tools/install) installed on your computer.\n\n- Clone the repo and enter its root folder\n\n  ```bash\n  git clone https://github.com/sopherapps/scdb.git \u0026\u0026 cd scdb\n  ```\n\n- Run the example\n\n  ```shell\n  cargo run --example hello_scdb\n  ```\n\n- Lint\n\n  ```shell\n  cargo clippy\n  ```\n\n- Run the test command\n\n  ```shell\n  cargo test\n  ```\n\n- Run the bench test command\n\n  ```shell\n  cargo bench\n  ```\n\n## Benchmarks\n\nOn an average PC (i7Core, 16GB RAM):\n\n``` \nset(no ttl): 'foo'      time:   [8.4622 µs 9.3052 µs 10.396 µs]\nset(ttl): 'foo'         time:   [9.0695 µs 9.2830 µs 9.5413 µs]\nset(no ttl) with search: 'foo'\n                        time:   [40.573 µs 41.152 µs 41.825 µs]\nset(ttl) with search: 'foo'\n                        time:   [42.494 µs 43.880 µs 45.353 µs]\nupdate(no ttl): 'foo'   time:   [8.0398 µs 8.1054 µs 8.1814 µs]\nupdate(ttl): 'fenecans' time:   [8.2151 µs 8.3078 µs 8.4137 µs]\nupdate(no ttl) with search: 'foo'\n                        time:   [40.757 µs 40.854 µs 40.960 µs]\nupdate(ttl) with search: 'fenecans'\n                        time:   [40.901 µs 40.985 µs 41.076 µs]\n                        time:   [7.9638 µs 8.0066 µs 8.0609 µs]\nget(no ttl): 'hey'      time:   [209.98 ns 213.70 ns 218.01 ns]\nget(no ttl): 'hi'       time:   [205.34 ns 207.45 ns 209.70 ns]\nget(no ttl): 'salut'    time:   [203.01 ns 204.54 ns 206.45 ns]\nget(no ttl): 'bonjour'  time:   [206.43 ns 208.68 ns 210.97 ns]\nget(no ttl): 'hola'     time:   [268.69 ns 297.50 ns 334.32 ns]\nget(no ttl): 'oi'       time:   [192.04 ns 192.62 ns 193.25 ns]\nget(no ttl): 'mulimuta' time:   [202.74 ns 203.14 ns 203.56 ns]\nget(with ttl): 'hey'    time:   [230.27 ns 230.65 ns 231.06 ns]\nget(with ttl): 'hi'     time:   [229.39 ns 229.89 ns 230.50 ns]\nget(with ttl): 'salut'  time:   [231.72 ns 232.10 ns 232.51 ns]\nget(with ttl): 'bonjour'\n                        time:   [232.30 ns 232.68 ns 233.10 ns]\nget(with ttl): 'hola'   time:   [231.98 ns 232.56 ns 233.16 ns]\nget(with ttl): 'oi'     time:   [228.74 ns 229.30 ns 229.87 ns]\nget(with ttl): 'mulimuta'\n                        time:   [237.61 ns 237.94 ns 238.29 ns]\nget(no ttl) with search: 'hey'\n                        time:   [194.52 ns 194.86 ns 195.25 ns]\nget(no ttl) with search: 'hi'\n                        time:   [195.36 ns 195.61 ns 195.86 ns]\nget(no ttl) with search: 'salut'\n                        time:   [198.78 ns 199.01 ns 199.25 ns]\nget(no ttl) with search: 'bonjour'\n                        time:   [199.74 ns 200.18 ns 200.79 ns]\nget(no ttl) with search: 'hola'\n                        time:   [199.81 ns 200.20 ns 200.60 ns]\nget(no ttl) with search: 'oi'\n                        time:   [191.97 ns 192.37 ns 192.80 ns]\nget(no ttl) with search: 'mulimuta'\n                        time:   [198.39 ns 198.80 ns 199.22 ns]\nget(with ttl) without search: 'hey'\n                        time:   [232.84 ns 234.11 ns 235.46 ns]\nget(with ttl) without search: 'hi'\n                        time:   [230.81 ns 231.25 ns 231.76 ns]\nget(with ttl) without search: 'salut'\n                        time:   [233.56 ns 234.07 ns 234.67 ns]\nget(with ttl) without search: 'bonjour'\n                        time:   [233.81 ns 234.23 ns 234.67 ns]\nget(with ttl) without search: 'hola'\n                        time:   [234.02 ns 234.43 ns 234.86 ns]\nget(with ttl) without search: 'oi'\n                        time:   [228.52 ns 228.84 ns 229.18 ns]\nget(with ttl) without search: 'mulimuta'\n                        time:   [233.36 ns 233.74 ns 234.15 ns]\nsearch (not paged): 'h' time:   [18.156 µs 18.274 µs 18.429 µs]\nsearch (not paged): 'h' #2\n                        time:   [18.093 µs 18.139 µs 18.192 µs]\nsearch (not paged): 's' time:   [8.6507 µs 8.6653 µs 8.6807 µs]\nsearch (not paged): 'b' time:   [8.6318 µs 8.6531 µs 8.6766 µs]\nsearch (not paged): 'h' #3\n                        time:   [18.106 µs 18.147 µs 18.188 µs]\nsearch (not paged): 'o' time:   [8.6288 µs 8.6415 µs 8.6557 µs]\nsearch (not paged): 'm' time:   [8.6453 µs 8.6657 µs 8.6873 µs]\nsearch (paged): 'h'     time:   [16.161 µs 16.230 µs 16.319 µs]\nsearch (paged): 'h' #2  time:   [15.949 µs 16.016 µs 16.093 µs]\nsearch (paged): 's'     time:   [6.0744 µs 6.1114 µs 6.1544 µs]\nsearch (paged): 'b'     time:   [6.2516 µs 6.3119 µs 6.3827 µs]\nsearch (paged): 'h' #3  time:   [15.990 µs 16.026 µs 16.063 µs]\nsearch (paged): 'o'     time:   [6.1061 µs 6.1790 µs 6.2617 µs]\nsearch (paged): 'm'     time:   [6.5727 µs 6.6862 µs 6.7921 µs]\ndelete(no ttl): 'foo'   time:   [51.172 µs 52.554 µs 54.057 µs]\ndelete(ttl): 'foo'      time:   [53.211 µs 54.964 µs 56.804 µs]\ndelete(no ttl) with search: 'foo'\n                        time:   [70.327 µs 70.698 µs 71.226 µs]\ndelete(ttl) with search: 'foo'\n                        time:   [70.753 µs 71.086 µs 71.520 µs]\nclear(no ttl)           time:   [144.05 µs 153.14 µs 170.79 µs]\nclear(ttl)              time:   [142.17 µs 142.68 µs 143.23 µs]\nclear(no ttl) with search\n                        time:   [221.58 µs 223.04 µs 224.52 µs]\nclear(ttl) with search  time:   [218.17 µs 226.53 µs 242.62 µs]\ncompact                 time:   [126.76 ms 128.26 ms 129.86 ms]\ncompact with search     time:   [128.80 ms 131.45 ms 134.50 ms]\n```\n\n## Acknowledgement\n\n- Inspiration was got from [lmdb](https://www.symas.com/lmdb/technical), especially in regard to memory-mapped\n  files. That is until I ran into issues with memory-mapped files...For more details, look\n  at [this paper by Andrew Crotty, Viktor Leis and Andy Pavlo](https://db.cs.cmu.edu/mmap-cidr2022/).\n- A few ideas were picked from [redis](https://redis.io/) and [sqlite](https://www.sqlite.org/index.html) especially to\n  do with the database file format.\n\n## License\n\nCopyright (c) 2022 [Martin Ahindura](https://github.com/Tinitto) Licensed under the [MIT License](./LICENSE)\n\n## Gratitude\n\n\u003e \"For My Father’s will is that everyone who looks to the Son and believes in Him shall have eternal life, and I will\n\u003e raise them up at the last day.\"\n\u003e\n\u003e -- John 6: 40\n\nAll glory be to God.\n\n\u003ca href=\"https://www.buymeacoffee.com/martinahinJ\" target=\"_blank\"\u003e\u003cimg src=\"https://cdn.buymeacoffee.com/buttons/v2/default-yellow.png\" alt=\"Buy Me A Coffee\" style=\"height: 60px !important;width: 217px !important;\" \u003e\u003c/a\u003e\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsopherapps%2Fscdb","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fsopherapps%2Fscdb","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsopherapps%2Fscdb/lists"}