{"id":13822552,"url":"https://github.com/iFaceless/tinkv","last_synced_at":"2025-05-16T17:31:12.436Z","repository":{"id":57669866,"uuid":"270571052","full_name":"iFaceless/tinkv","owner":"iFaceless","description":"[WIP] ✨ TinKV is a simple and fast key-value storage written in Rust, which provides a bultin CLI and a Redis compatible server.","archived":false,"fork":false,"pushed_at":"2022-02-16T15:14:50.000Z","size":170,"stargazers_count":31,"open_issues_count":3,"forks_count":6,"subscribers_count":3,"default_branch":"master","last_synced_at":"2024-11-16T04:46:05.429Z","etag":null,"topics":["bitcask","key-value-store","log-structured","redis-compatiable-storage","redis-protocol","rust","tinkv"],"latest_commit_sha":null,"homepage":"https://ifaceless.github.io/tinkv","language":"Rust","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/iFaceless.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2020-06-08T07:23:16.000Z","updated_at":"2024-02-25T06:53:22.000Z","dependencies_parsed_at":"2022-09-26T20:40:38.997Z","dependency_job_id":null,"html_url":"https://github.com/iFaceless/tinkv","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/iFaceless%2Ftinkv","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/iFaceless%2Ftinkv/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/iFaceless%2Ftinkv/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/iFaceless%2Ftinkv/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/iFaceless","download_url":"https://codeload.github.com/iFaceless/tinkv/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":225442810,"owners_count":17475115,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["bitcask","key-value-store","log-structured","redis-compatiable-storage","redis-protocol","rust","tinkv"],"created_at":"2024-08-04T08:02:05.760Z","updated_at":"2024-11-19T23:30:38.974Z","avatar_url":"https://github.com/iFaceless.png","language":"Rust","funding_links":[],"categories":["Rust"],"sub_categories":[],"readme":"[![Build Status](https://travis-ci.org/iFaceless/tinkv.svg?branch=master)](https://travis-ci.org/iFaceless/tinkv)\n[![tinkv](http://meritbadge.herokuapp.com/tinkv?ver=1)](https://crates.io/crates/tinkv)\n[![License: MIT OR Apache-2.0](https://img.shields.io/crates/l/tinkv.svg)](#license)\n\n# ![TinKV Logo](https://pic3.zhimg.com/80/v2-3a50e948ca1b35f311747591b1e854d1_1440w.png)\n\n[TinKV](https://github.com/iFaceless/tinkv) is a simple and fast key-value storage engine written in Rust. Inspired by [basho/bitcask](https://github.com/basho/bitcask), written after attending the [Talent Plan courses](https://github.com/pingcap/talent-plan). \n\n**Notes**:\n- *Do not use it in production.*\n- *Operations like set/remove/compact are not thread-safe currently.*\n\nHappy hacking~\n\n![Overview.jpg](https://pic4.zhimg.com/80/v2-be29d9457a0d31409649eae4cfe743b4_1440w.png)\n\n![Engine.jpg](https://pic4.zhimg.com/80/v2-f1e850d2af2db3543a3543b0c6e92f59_1440w.png)\n\n# Features\n\n- Embeddable (use `tinkv` as a library);\n- Builtin CLI (`tinkv`);\n- Builtin Redis compatible server;\n- Predictable read/write performance.\n\n# Usage\n## As a library\n\n```shell\n$ cargo add tinkv\n```\n\nFull example usage can be found in [examples/basic.rs](./examples/basic.rs).\n\n```rust\nuse tinkv::{self, Store};\n\nfn main() -\u003e tinkv::Result\u003c()\u003e {\n    pretty_env_logger::init();\n    let mut store = Store::open(\"/path/to/tinkv\")?;\n    store.set(\"hello\".as_bytes(), \"tinkv\".as_bytes())?;\n\n    let value = store.get(\"hello\".as_bytes())?;\n    assert_eq!(value, Some(\"tinkv\".as_bytes().to_vec()));\n\n    store.remove(\"hello\".as_bytes())?;\n\n    let value_not_found = store.get(\"hello\".as_bytes())?;\n    assert_eq!(value_not_found, None);\n\n    Ok(())\n}\n```\n\n### Open with custom options\n\n```rust\nuse tinkv::{self, Store};\n\nfn main() -\u003e tinkv::Result\u003c()\u003e {\n    let mut store = tinkv::OpenOptions::new()\n        .max_data_file_size(1024 * 1024)\n        .max_key_size(128)\n        .max_value_size(128)\n        .sync(true)\n        .open(\".tinkv\")?;\n    store.set(\"hello\".as_bytes(), \"world\".as_bytes())?;\n    Ok(())\n}\n```\n\n### APIs\nPublic APIs of tinkv store are very easy to use:\n| API                      |                   Description                                 |\n|--------------------------|---------------------------------------------------------------|\n|`Store::open(path)`       | Open a new or existing datastore. The directory must be writeable and readable for tinkv store.|`\n|`tinkv::OpenOptions()`    | Open a new or existing datastore with custom options. |\n|`store.get(key)`          | Get value by key from datastore.|\n|`store.set(key, value)`   | Store a key value pair into datastore.|\n|`store.remove(key, value)`| Remove a key from datastore.|\n|`store.compact()`         | Merge data files into a more compact form. drop stale segments to release disk space. Produce hint files after compaction for faster startup.|\n|`store.keys()`            | Return all the keys in database.|\n|`store.len()`             | Return total number of keys in database.|\n|`store.for_each(f: Fn(key, value) -\u003e Result\u003cbool\u003e)`             | Iterate all keys in database and call function `f` for each entry.|\n|`store.stas()`            | Get current statistics of database.|\n|`store.sync()`            | Force any writes to datastore.|\n|`store.close()`           | Close datastore, sync all pending writes to disk.|\n\n### Run examples\n\n```shell\n$ RUST_LOG=trace cargo run --example basic\n```\n\n`RUST_LOG` level can be one of [`trace`, `debug`, `info`, `error`].\n\n\u003cdetails\u003e\n    \u003csummary\u003eCLICK HERE | Example output.\u003c/summary\u003e\n\n```shell\n$ RUST_LOG=info cargo run --example basic\n\n 2020-06-18T10:20:03.497Z INFO  tinkv::store \u003e open store path: .tinkv\n 2020-06-18T10:20:04.853Z INFO  tinkv::store \u003e build keydir done, got 100001 keys. current stats: Stats { size_of_stale_entries: 0, total_stale_entries: 0, total_active_entries: 100001, total_data_files: 1, size_of_all_data_files: 10578168 }\n200000 keys written in 9.98773 secs, 20024.57 keys/s\ninitial: Stats { size_of_stale_entries: 21155900, total_stale_entries: 200000, total_active_entries: 100001, total_data_files: 2, size_of_all_data_files: 31733728 }\nkey_1 =\u003e \"value_1_1592475604853568000_hello_world\"\nafter set 1: Stats { size_of_stale_entries: 21155900, total_stale_entries: 200000, total_active_entries: 100002, total_data_files: 2, size_of_all_data_files: 31733774 }\nafter set 2: Stats { size_of_stale_entries: 21155946, total_stale_entries: 200001, total_active_entries: 100002, total_data_files: 2, size_of_all_data_files: 31733822 }\nafter set 3: Stats { size_of_stale_entries: 21155994, total_stale_entries: 200002, total_active_entries: 100002, total_data_files: 2, size_of_all_data_files: 31733870 }\nafter remove: Stats { size_of_stale_entries: 21156107, total_stale_entries: 200003, total_active_entries: 100001, total_data_files: 2, size_of_all_data_files: 31733935 }\n 2020-06-18T10:20:14.841Z INFO  tinkv::store \u003e compact 2 data files\nafter compaction: Stats { size_of_stale_entries: 0, total_stale_entries: 0, total_active_entries: 100001, total_data_files: 2, size_of_all_data_files: 10577828 }\nkey_1 =\u003e \"value_1_1592475604853568000_hello_world\"\n```\n\u003c/details\u003e\n\n## CLI\n\nInstall `tinkv` executable binaries.\n\n```shell\n$ cargo install tinkv\n```\n\n```shell\n$ tinkv --help\n...\nUSAGE:\n    tinkv [FLAGS] \u003cpath\u003e \u003cSUBCOMMAND\u003e\n\nFLAGS:\n    -h, --help       Prints help information\n    -q, --quiet      Pass many times for less log output\n    -V, --version    Prints version information\n    -v, --verbose    Pass many times for more log output\n\nARGS:\n    \u003cpath\u003e    Path to tinkv datastore\n\nSUBCOMMANDS:\n    compact    Compact data files in datastore and reclaim disk space\n    del        Delete a key value pair from datastore\n    get        Retrive value of a key, and display the value\n    help       Prints this message or the help of the given subcommand(s)\n    keys       List all keys in datastore\n    scan       Perform a prefix scanning for keys\n    set        Store a key value pair into datastore\n    stats      Display statistics of the datastore\n```\n\nExample usages:\n```shell\n$ tinkv /tmp/db set hello world\n$ tinkv /tmp/db get hello\nworld\n\n# Change verbosity level (info).\n$ tinkv /tmp/db -vvv compact\n2020-06-20T10:32:45.582Z INFO  tinkv::store \u003e open store path: tmp/db\n2020-06-20T10:32:45.582Z INFO  tinkv::store \u003e build keydir from data file /tmp/db/000000000001.tinkv.data\n2020-06-20T10:32:45.583Z INFO  tinkv::store \u003e build keydir from data file /tmp/db/000000000002.tinkv.data\n2020-06-20T10:32:45.583Z INFO  tinkv::store \u003e build keydir done, got 1 keys. current stats: Stats { size_of_stale_entries:0, total_stale_entries: 0, total_active_entries: 1,total_data_files: 2, size_of_all_data_files: 60 }\n2020-06-20T10:32:45.583Z INFO  tinkv::store \u003e there are 3 datafiles need to be compacted\n```\n\n## Client \u0026 Server\n\n[`tinkv-server`](./bin/../src/bin/tinkv-server.rs) is a redis-compatible key/value store server. However, not all the redis commmands are supported. The available commands are:\n\n- `get \u003ckey\u003e`\n- `mget \u003ckey\u003e [\u003ckey\u003e...]`\n- `set \u003ckey\u003e \u003cvalue\u003e`\n- `mset \u003ckey\u003e \u003cvalue\u003e [\u003ckey\u003e \u003cvalue\u003e]`\n- `del \u003ckey\u003e`\n- `keys \u003cpattern\u003e`\n- `ping [\u003cmessage\u003e]`\n- `exists \u003ckey\u003e`\n- `info [\u003csection\u003e]`\n- `command`\n- `dbsize`\n- `flushdb/flushall`\n- `compact`: extended command to trigger a compaction manually.\n\nKey/value pairs are persisted in log files under directory `/urs/local/var/tinkv`. The default listening address of server is `127.0.0.1:7379`, and you can connect to it with a redis client.\n\n### Quick Start\n\nIt's very easy to install `tinkv-server`:\n\n```shell\n$ cargo install tinkv\n```\n\nStart server with default config (set log level to `info` mode):\n\n```shell\n$ tinkv-server -vv\n2020-06-24T13:46:49.341Z INFO  tinkv::store \u003e open store path: /usr/local/var/tinkv\n2020-06-24T13:46:49.343Z INFO  tinkv::store \u003e build keydir from data file /usr/local/var/tinkv/000000000001.tinkv.data\n2020-06-24T13:46:49.343Z INFO  tinkv::store \u003e build keydir from data file /usr/local/var/tinkv/000000000002.tinkv.data\n2020-06-24T13:46:49.343Z INFO  tinkv::store \u003e build keydir done, got 0 keys. current stats: Stats { size_of_stale_entries: 0,total_stale_entries: 0, total_active_entries: 0, total_data_files: 2, size_of_all_data_files: 0 }\n2020-06-24T13:46:49.343Z INFO  tinkv::server \u003e TinKV server is listening at '127.0.0.1:7379'\n```\n\nCommunicate with `tinkv-server` by using `reids-cli`:\n\n\u003cdetails\u003e\n    \u003csummary\u003eCLICK HERE\u003c/summary\u003e\n\n```shell\n$ redis-cli -p 7379\n127.0.0.1:7379\u003e ping\nPONG\n127.0.0.1:7379\u003e ping \"hello, tinkv\"\n\"hello, tinkv\"\n127.0.0.1:7379\u003e set name tinkv\nOK\n127.0.0.1:7379\u003e exists name\n(integer) 1\n127.0.0.1:7379\u003e get name tinkv\n(error) ERR wrong number of arguments for 'get' command\n127.0.0.1:7379\u003e get name\n\"tinkv\"\n127.0.0.1:7379\u003e command\n1) \"ping\"\n2) \"get\"\n3) \"set\"\n4) \"del\"\n5) \"dbsize\"\n6) \"exists\"\n7) \"compact\"\n8) \"info\"\n9) \"command\"\n...and more\n127.0.0.1:7379\u003e info\n# Server\ntinkv_version: 0.9.0\nos: Mac OS, 10.15.4, 64-bit\n\n# Stats\nsize_of_stale_entries: 143\nsize_of_stale_entries_human: 143 B\ntotal_stale_entries: 3\ntotal_active_entries: 1109\ntotal_data_files: 5\nsize_of_all_data_files: 46813\nsize_of_all_data_files_human: 46.81 KB\n127.0.0.1:7379\u003e notfound\n(error) ERR unknown command `notfound`\n127.0.0.1:7379\u003e\n```\n\u003c/details\u003e\n\n# About Compaction\n\nCompation process will be triggered if `size_of_stale_entries \u003e= config::COMPACTION_THRESHOLD` after each call of `set/remove`. Compaction steps are very simple and easy to understand:\n1. Freeze current active segment, and switch to another one.\n2. Create a compaction segment file, then iterate all the entries in `keydir` (in-memory hash table), copy related data entries into compaction file and update `keydir`.\n3. Remove all the stale segment files.\n\nHint files (for fast startup) of corresponding data files will be generated after each compaction.\n\nYou can call `store.compact()` method to trigger compaction process if nessesary.\n\n```rust\nuse tinkv::{self, Store};\n\nfn main() -\u003e tinkv::Result\u003c()\u003e {\n    pretty_env_logger::init();\n    let mut store = Store::open(\"/path/to/tinkv\")?;\n    store.compact()?;\n\n    Ok(())\n}\n```\n\n# Structure of Data Directory\n\n```shell\n.tinkv\n├── 000000000001.tinkv.hint -- related index/hint file, for fast startup\n├── 000000000001.tinkv.data -- immutable data file\n└── 000000000002.tinkv.data -- active data file\n```\n\n# Refs\n## Projects\nI'm not familiar with erlang, but I found some implementations in other languages worth learning.\n\n1. Go: [prologic/bitcask](https://github.com/prologic/bitcask)\n2. Go: [prologic/bitraft](https://github.com/prologic/bitraft)\n3. Python: [turicas/pybitcask](https://github.com/turicas/pybitcask)\n4. Rust: [dragonquest/bitcask](https://github.com/dragonquest/bitcask)\n\nFound another simple key-value database based on Bitcask model, please refer [xujiajun/nutsdb](https://github.com/xujiajun/nutsdb).\n\n## Articles and more\n\n- [Implementing a Copyless Redis Protocol in Rust with Parsing Combinators](https://dpbriggs.ca/blog/Implementing-A-Copyless-Redis-Protocol-in-Rust-With-Parsing-Combinators)\n- [Expected type parameter, found struct](https://stackoverflow.com/questions/26049939/expected-type-parameter-found-struct)\n- [Help understanding how trait bounds workd](https://users.rust-lang.org/t/help-understanding-how-trait-bounds-work/19253/3)\n- [Idiomatic way to take ownership of all items in a Vec\u003cString\u003e?](https://users.rust-lang.org/t/idiomatic-way-to-take-ownership-of-all-items-in-a-vec-string/7811/12)\n- [Idiomatic callbacks in Rust](https://stackoverflow.com/questions/41081240/idiomatic-callbacks-in-rust)\n- [What are reasonable ways to store a callback in a struct?](https://users.rust-lang.org/t/what-are-reasonable-ways-to-store-a-callback-in-a-struct/5810)\n- [Things Rust doesn’t let you do](https://medium.com/@GolDDranks/things-rust-doesnt-let-you-do-draft-f596a3c740a5)\n\n# License\n\nLicensed under the [MIT license](./LICENSE).\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FiFaceless%2Ftinkv","html_url":"https://awesome.ecosyste.ms/projects/github.com%2FiFaceless%2Ftinkv","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FiFaceless%2Ftinkv/lists"}