{"id":13504360,"url":"https://github.com/tancehao/ConstDB","last_synced_at":"2025-03-29T21:30:33.370Z","repository":{"id":43152541,"uuid":"469704404","full_name":"tancehao/ConstDB","owner":"tancehao","description":"A redis-like cache store that implements CRDTs and active-active replications. ","archived":false,"fork":false,"pushed_at":"2023-05-29T07:37:48.000Z","size":196,"stargazers_count":40,"open_issues_count":2,"forks_count":1,"subscribers_count":4,"default_branch":"main","last_synced_at":"2025-03-22T06:25:25.718Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Rust","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/tancehao.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null}},"created_at":"2022-03-14T11:31:15.000Z","updated_at":"2025-01-12T08:44:25.000Z","dependencies_parsed_at":"2024-01-13T19:20:06.851Z","dependency_job_id":"f2208b57-4dad-47ff-8204-62c5aaa48778","html_url":"https://github.com/tancehao/ConstDB","commit_stats":null,"previous_names":[],"tags_count":1,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/tancehao%2FConstDB","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/tancehao%2FConstDB/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/tancehao%2FConstDB/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/tancehao%2FConstDB/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/tancehao","download_url":"https://codeload.github.com/tancehao/ConstDB/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":246249130,"owners_count":20747164,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-08-01T00:00:35.313Z","updated_at":"2025-03-29T21:30:32.998Z","avatar_url":"https://github.com/tancehao.png","language":"Rust","funding_links":[],"categories":["Databases"],"sub_categories":["Key-Value Databases"],"readme":"# Introduction\nConstDB is an in-memory cache store which aims at master-master replications. A group of ConstDB instances can be deployed across many regions, and each of them is available for\nreading and writing. Every instance keeps in touch with all other ones. When a write command has been executed in one instance, it is asynchronously replicated to all the other instances. The data structures that are used for storing the client's data implement CRDT(Conflict-Free Replicated Data Types), thus we are sure data in different instances are eventually consistent.  \nConstDB is compatible with redis protocol, and I have implemented a portion of redis's data types and commands. I'll keep developing it and take as much features in redis as possible into ConstDB. Contribution is also welcomed.\n\n# Features\n* **CRDT**: Each write command is associated with an locally increasing 64-bit uuid and an unique node id, therefore the synchronization is idempotent and loopback is avoided. And then the data structures used for storing also track some extra information beside user's data. For example, for some data types, we store the time they are lastly modified. This way, if two commands modifying the same key are executed at different nodes concurrently, we solve the conflicts through `Last-Write-Win` and `Add-Win` policies when they are merged at each node eventually. We also support other crdt types including `PNCounter`, `LWWRegister`, `MultiValueRegister`,`ORSet` and `Sequence` and so on.\n* **Online** scaling: Nodes are free to join or leave a group. When we want to add a node(B) into an already running group, we simply let it to `MEET` a node(A) that is contained in that group. The new node(B) firstly connects to that node. And then they exchange their whole data set through their own snapshot file(this is safe because their data both implement CRDT)——in which all the necessary information of their own replicas are also listed. Once one node has merged the other's snapshot, it then continues a)to accept his write commands happened after the snapshot, and b)connects to the newly found replicas from snapshot and exchange their data. Finnaly the new node(B) keeps pace with all other nodes in this group.\n* **Multiple io threads**: Input requests and output responses are read or written on io threads, and each command is executed sequentially in the main thread. This way the performance is much better.\n\n# Plans\n\n* **S3 storage for snapshot**\n* **Support for zset**  \n* **Global lock**\n\n\n# Try\nYou can experience the master-master replications by the following test.\n1. Download the project and build.\n\u003e git clone https://github.com/tancehao/ConstDB \u0026\u0026 cd ConstDB  \n\u003e cargo build\n \n2. Run 2 processes listening to different local ports. Make 2 copies of the file `constdb.toml`.\n\u003e mkdir server9001 server9002\u003c/p\u003e\n\u003e cp default.toml server9001/constdb.toml \u0026\u0026 cp default.toml server9002/constdb.toml  \n\nAnd then edit the `constdb.toml` config files in each directory(change the id, node_alias, port and other options).\nFinally start 2 process with these different configs.\n\u003e cd server9001 \u0026\u0026 ../target/debug/constdb-server -c constdb.toml  \n\u003e cd ..  \n\u003e cd server9002 \u0026\u0026 ../target/debug/constdb-server -c constdb.toml\n\n3. Now a happy moment has come. We modify different keys in each node and make them keep in touch.\n\u003e redis-cli -p 9001 incr k1             // 1  \n\u003e redis-cli -p 9002 incr k2             // 1  \n\u003e redis-cli -p 9002 meet 127.0.0.1:9001 //  1\n\nwait for some seconds.\n\u003e redis-cli -p 9001 get k2          // 1  \n\u003e redis-cli -p 9002 get k1          // 1  \n\u003e redis-cli -p 9002 incr k2         // 2  \n\u003e redis-cli -p 9001 get k2          // 2  \n\u003e redis-cli -p 9001 hset h1 k1 v1   // +OK  \n\u003e redis-cli -p 9002 hget h1 k1      // \"v1\"\n\nwe can see that the each node stays abreast of the other.\n\n# Commands available\n- **last-write-win registers**  \n    `get`, `set`, `del`  \n\n- **multi-value register**  \n    `mvget`: get the list of node_id -\u003e value pairs.  \n    `mvset`: set the value corresponds to our node. Use the `-HARD` option to delete all other versions. \n\n- **counters**:  \n    `get`, `incr`, `decr`  \n\n- **hash**  \n    `hset`: last write win  \n    `hdel`: add win  \n    `hget`, `hgetall`, `hkeys`, `hlen`, `hvals`, `hrandfield`, `hmget`  \n\n- **set**  \n    `sadd`, `srem`, `spop`, `smembers`:  add win  \n    `sinter`, `sintersotre`, `sintercard`, `sunion`, `sunionstore`,\n    `sdiff`, `sdiffsotre`, `srandmember`, `sismember`, `smismember`, `scard`, `smove`  \n\n- **list**  \n    `llen`, `lindex`, `lrange`, `lpos`, `rpop`, `rpush`, `lpop`, `lpush`, `lrem`, `lset`, `linsert`, `ltrim`  \n\n- **control**  \n    `wait`: wait until the last write has been acked by a configuable count of replicas.  \n\t`desc`: describe the detailed information of a crdt value.  \n    `replicas`: list the replicas in the group currently.  \n    `node`: tell the identification of the server.  \n    `info`: show the stats collected so far.  \n    `meet`: let the server to join another one's group.\n    `forget`: left the server to kick another one out from the group.  \n    `repllog`: print the replicate log with a specified uuid.  \n    `replcheck`: check if the server has received a write command with a specified uuid from another node.  \n\t`del`, `expire` \n\t\n\n# Contact\nEmail: tancehao93@163.com  \nWechat id: xiaotanzi-bien  \n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ftancehao%2FConstDB","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Ftancehao%2FConstDB","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ftancehao%2FConstDB/lists"}