{"id":13614456,"url":"https://github.com/maxmunzel/kvass","last_synced_at":"2025-04-13T18:33:06.936Z","repository":{"id":41570177,"uuid":"510005590","full_name":"maxmunzel/kvass","owner":"maxmunzel","description":"a personal key-value store","archived":false,"fork":false,"pushed_at":"2023-11-28T09:30:14.000Z","size":48,"stargazers_count":883,"open_issues_count":3,"forks_count":24,"subscribers_count":6,"default_branch":"master","last_synced_at":"2024-11-07T22:42:36.520Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/maxmunzel.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2022-07-03T11:37:10.000Z","updated_at":"2024-10-09T09:34:12.000Z","dependencies_parsed_at":"2024-06-18T21:11:24.794Z","dependency_job_id":null,"html_url":"https://github.com/maxmunzel/kvass","commit_stats":{"total_commits":45,"total_committers":7,"mean_commits":6.428571428571429,"dds":0.1777777777777778,"last_synced_commit":"3f861485a75151416436317731230b0427b31259"},"previous_names":[],"tags_count":1,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/maxmunzel%2Fkvass","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/maxmunzel%2Fkvass/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/maxmunzel%2Fkvass/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/maxmunzel%2Fkvass/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/maxmunzel","download_url":"https://codeload.github.com/maxmunzel/kvass/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248760663,"owners_count":21157406,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-08-01T20:01:01.756Z","updated_at":"2025-04-13T18:33:06.666Z","avatar_url":"https://github.com/maxmunzel.png","language":"Go","funding_links":[],"categories":["Go"],"sub_categories":[],"readme":"# kvass: a personal key-value store\n\n![kvass_small](https://user-images.githubusercontent.com/5411096/179968508-5fe1e390-3136-46a6-bb1e-8d329ad231c3.jpeg)\n\n\n```bash\n# simple usage\n$ kvass set hello world\n$ kvass get hello\nworld\n\n# enumerate keys\n$ kvass ls\nhello\n\n# store arbitrary files\n$ kvass set logo \u003c kvass.jpg\n$ kvass get logo \u003e kvass.jpg\n\n\n# Its trivial to set up and operate kvass across multiple devices\n$ ssh you@yourserver.com kvass config show\n\nEncryption Key:  \t5abf59f5f1a2f3c998a4f592ce081a23e14a68fd8a792259c6ec0fc1e8fb1246  # \u003c- copy this for the next step\nProcessID:       \t752176921\nRemote:          \t(None)\n\n$ kvass config key 5abf59f5f1a2f3c998a4f592ce081a23e14a68fd8a792259c6ec0fc1e8fb1246 # set the same key for all your devices\n$ kvass config remote yourserver.com:8000 # tell kvass where to find the server instance\n\n# Run \"kvass serve\" on your server using systemd, screen or the init system of your choice (runit, anyone?). You can specify the interface and port to host at with [--bind].\n\n$ kvass serve --bind=\"0.0.0.0:80\" # host on the default HTTP port (which means you can generate cleaner URLs - just set your remote no port)\n\n# every set will now be broadcasted to the server\n$ kvass set \"hello from the other side\" hello\n$ ssh you@yourserver kvass get \"hello from the other side\"\nhello\n\n# and every get will check the server for updates\n$ ssh you@yourserver kvass set hello 👋\n$ kvass get hello\n👋\n\n# Good to know: All communication between the client and server is authenticated and encrypted using AES-256 GCM.\n\n# remember the file we stored earlier? Let's get a shareable url for it!\n$ kvass url logo\nhttp://demo.maxnagy.com:8000/get?q=OQMwTQmFCz6xiWxFxt4Mkw\n\n# you can also print the corresponding qr code directly to your terminal\nkvass qr logo\n```\n![Screen Shot 2022-07-20 at 13 23 17](https://user-images.githubusercontent.com/5411096/179970204-f1034add-ce07-4f40-b279-0ac25969c069.png)\n\n```\n# run kvass without arguments to get a nice cheat sheet of supported commands\n$ kvass\nkvass [--db=string]\n\nDescription:\n    kvass - a personal KV store\n\nOptions:\n        --db       the database file to use (default: ~/.kvassdb.sqlite)\n\nSub-commands:\n    kvass ls       list keys\n    kvass get      get a value\n    kvass set      set a value\n    kvass rm       remove a key\n    kvass url      show shareable url of an entry\n    kvass qr       print shareable qr code of entry to console\n    kvass config   set config parameters\n    kvass serve    start in server mode [--bind=\"ip:port\" (default: 0.0.0.0:8000)]\n```\n\n# Installation\n\n```bash\ngo install github.com/maxmunzel/kvass@latest\n```\n\n# How Syncing works\n\nTL;DR There is a central server running `kvass serve` with clients\nconnected to it. Key-value pairs overwrite each other based on wall\nclock time 99.999% of the time and using Lamport clocks in the\nremaining .001% of the time. You can mostly forget about this, as long\nas your clocks are mostly in sync.\n\nLet's dive into the details!\n\nEach time we `set` or `rm` a key, kvass creates a new `KvEntry` struct and\nmerges it onto its local state. The local state is a set of `KvEntry`s that\nrepresent a key-value mapping.\nTechnically, `kvass rm key` is `kvass set key \"\"`, so they work the same way.\n\n`KvEntry` is defined as follows:\n```go\ntype KvEntry struct {\n\tKey      string\n\tValue    []byte // empty slice means key deleted\n\tUrlToken string // random token used for url\n\n\t// The following fields are used for state merging\n\tTimestampUnixMicro int64\n\tProcessID          uint32 // randomly chosen for each node\n\tCounter            uint64 // Lamport clock\n}\n```\n\n## Lamport Clocks\n\nLamport Clocks are a common and easy way to order events in a distributed system.\nThe [Wikipedia](https://en.wikipedia.org/wiki/Lamport_timestamp) summarizes them nicely:\n\u003e   The algorithm follows some simple rules:\n\u003e   1. A process increments its counter before each local event (e.g., message sending event);\n\u003e   2. When a process sends a message, it includes its counter value with the message after executing step 1;\n\u003e   3. On receiving a message, the counter of the recipient is updated, if necessary, to the greater of its current counter and the timestamp in the received message. The counter is then incremented by 1 before the message is considered received.\n\nIn kvass, sending a message means `set`ting a key locally and\nreceiving a message means merging a `KvEntry` into the local state.\n\nLamport clocks have a nice property: If an event `a` happens causally\nafter another event `b`, then it follows, that `a.count` \u003e `b.count`:\n\n\n```\nnode1     set foo=bar   -\u003e   set foo=baz    \\  (send KvEntry{Key=\"foo\", Counter=2, Value=\"baz\"} to node2)\ncount=0   count = 1          count = 2       \\\n                                              \\\nnode2                                           -\u003e   rm foo\ncount=0                                              count = 3\n```\n\nnode2 updated its counter upon receiving node1's update, so the counter values nicely reflect the fact,\nthat node2 knew about node1's updates to foo before deleting it. This isn't always helpful though:\n\n```\nnode1     set foo=bar     \\  (send KvEntry{Key=\"foo\", Counter=1, Value=\"bar\"} to node2)\ncount=0   count = 1        \\\n                            \\\nnode2     set foo=baz         \u003e  (What is foo supposed to be know?)\ncount=0   count = 1\n```\n\nThe canonical answer in this case is to resolve conflicts based on node number (`ProcessID`).\nThis introduces a global, consistent and total order of events. However it may not always reflect\nthe order in which a user actually performs the events\n```\nnode1     set foo=bar   -\u003e   set foo=baz                      \\  (send KvEntry{Key=\"foo\", Counter=2, Value=\"baz\"} to node2)\ncount=0   count = 1          count = 2                         \\\n                                                                \\\nnode2                                         rm foo             -\u003e   (foo is not set to \"baz\" again)\ncount=0                                       count = 1\n```\n\nIn kvass, we therefore use wall-clock time to resolve conflicts: The most recent action of a user is probably the\none he intents to persist, independent of the node he triggered it on. Still, we use Lamport timestamps to handle\nidentical timestamps and to keep track of which `KvEntry`s we need to exchange between nodes:\n\n\n```\nnode1     set foo=bar   -\u003e   set foo=baz                      \\  (send KvEntry{Key=\"foo\", Counter=2, Value=\"baz\"} to node2)\ncount=0   count = 1          count = 2                         \\\n                                                                \\\nnode2                                         rm foo             -\u003e   (foo stays removed, its\ncount=0                                       count = 1               count increased to 3)\n```\n\nThe merging of states is actually trivial now:\n```go\nfunc (s *SqlitePersistance) UpdateOn(entry KvEntry) error {\n\n    oldEntry := getCurrentEntryFromDB(entry.Key)\n\n    // update the remote counter\n    s.State.RemoteCounter = mathutil.MaxUint64(s.State.RemoteCounter, entry.Counter)\n\n    // select LUB of old and new entry\n    entry = entry.Max(oldEntry) // returns the entry with the greater (time, counter, - pid) tuple\n\n    // update local counter\n    newCounter := mathutil.MaxUint64(entry.Counter, s.State.Counter) + 1\n    s.State.Counter = newCounter\n\n    // set new entries counter\n    entry.Counter = newCounter\n\n    // write back LUB to db\n}\n```\n\n`LUB` is CRDT-speak for \"least upper bound\" ie the smallest entry that is \u003e= the old and new entry.\nWe convince ourself, that the `KvEntry.Max()` satisfies this LUB property and can therefore derive\nthat it is also commutative, *idempotent* and associative.\n\n\n\n\n# Shoutouts\n\n[Charm skate](https://github.com/charmbracelet/skate) -- the inspiration for this tool\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmaxmunzel%2Fkvass","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fmaxmunzel%2Fkvass","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmaxmunzel%2Fkvass/lists"}