{"id":21964559,"url":"https://github.com/intob/rocketkv","last_synced_at":"2025-07-11T18:02:23.199Z","repository":{"id":43416997,"uuid":"457938133","full_name":"intob/rocketkv","owner":"intob","description":"Minimal KV store","archived":false,"fork":false,"pushed_at":"2022-03-05T16:46:23.000Z","size":10763,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-03-15T08:04:09.254Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/intob.png","metadata":{"files":{"readme":"readme.md","changelog":null,"contributing":null,"funding":null,"license":"license","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2022-02-10T20:33:22.000Z","updated_at":"2022-02-28T09:59:18.000Z","dependencies_parsed_at":"2022-09-14T02:02:09.220Z","dependency_job_id":null,"html_url":"https://github.com/intob/rocketkv","commit_stats":null,"previous_names":["dr-useless/gobkv","intob/gobkv","dr-useless/tcpkv"],"tags_count":6,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/intob%2Frocketkv","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/intob%2Frocketkv/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/intob%2Frocketkv/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/intob%2Frocketkv/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/intob","download_url":"https://codeload.github.com/intob/rocketkv/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":245021301,"owners_count":20548318,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-29T12:24:43.667Z","updated_at":"2025-03-22T21:22:09.838Z","avatar_url":"https://github.com/intob.png","language":"Go","funding_links":[],"categories":[],"sub_categories":[],"readme":"# RocketKV\n\nMinimalisic, highly performant key-value storage, written in Go.\n\n\u003cimg src=\"https://goreportcard.com/badge/github.com/intob/rocketkv\" /\u003e\n\n# Usage\n1. Install `go install github.com/intob/rocketkv`\n2. Run `rocketkv` (if $GOPATH/bin is in your PATH)\n\n## Config\nThis project uses [Viper](https://github.com/spf13/viper).\n\nUnless you explicitly provide a config file with `-c`, the config file should be named `config.`, with an extension. TOML, YAML \u0026 JSON are supported.\n\nThe following directories are searched for a config file:\n- `/etc/rocketkv`\n- `.` (pwd)\n\n```toml\n# /tmp/rocketkv/config.toml\n\n# network \u0026 auth\nnetwork = \"tcp\"\naddress = \":8100\"\nauth = \"supersecretsecret,wait,it'sinthereadme\"\n\n# general\nsegments = 16 # make 256 blocks (16 parts * 16 blocks)\nbuffersize = 2000000 # 2MB\nscanperiod = 10\n\n# persistence\npersist = true\ndir = \"/etc/rocketkv\"\nwriteperiod = 10\n\n[tls]\n  cert = \"path/to/x509/cert.pem\"\n  key = \"path/to/x509/key.pem\"\n```\nFor periords, unit of time is one second. I will add support for parsing time strings.\n\nFor each part, the number of blocks created is equal to the part count. So, 8 parts will result in 64 blocks.\n\n## Play\n1. Install CLI tool, rkteer\n  `go install github.com/intob/rkteer`\n2. Bind to your rocketkv instance\n  `rkteer bind [NETWORK] [ADDRESS] --a [AUTHSECRET]`\n3. Call set, get, del, list, or count\n```\n./rkteer set coffee beans\nstatus: OK\n./rkteer get coffee\nbeans\n```\n\n# In progress\n- Support for horizontal scaling\n\n# To do\n- Re-partitioning\n- Test membership using Bloom filter before GET\n\n# Keys\nKeys can be specified in two forms; bare \u0026 namespaced.\n\n## Bare\nA bare key has no namespace prefix. It is simply a string with no `/` path separator.\n\n## Namespaced\nGrouping keys is acheived by namespacing. This is done by prefixing the key with a path.\n```\n randomnamespaced/examplekey\n|---namespace----|---name---|\n```\nNote that the namespace includes the final `/`.\n\nAll keys for a given namespace will land in the same [block](#blocks). This greatly improves performance for collecting \u0026 listing multiple keys, because only a single block must be searched.\n\n# Segmentation\nTo reduce load on the file system \u0026 and decrease blocking, the dataset is split into 2 layers. Each layer contains the configured number of segments.\n\n## Partitions\nThis is the top layer.\n\nWhen initialising a dataset, the number of partitions (parts) created will be equal to the configured `Parts.Count` property.\n\nIdentifying the partition corresponding to a key is the first step to locating (or placing) a key.\n\n## Blocks\nEach part is split into blocks. The number of blocks in each part is equal to the number of parts. So 8 parts will result in 64 blocks.\n\nEach block has it's own mutex \u0026 map of keys.\n\nWhen a key is written to or deleted, the parent block is flagged as changed.\n\nIf persistence is enabled in the config via `\"Parts.Persist\": true`, then each block is written to the file system periodically, when changed.\n\n## Partition:Block:Key mapping\nDistance from key to a partition or block is calculated using Hamming distance.\nIf the key contains a namespace, only the namespace is hashed.\n```go\nd := hash(key) ^ blockId // or partId\n```\nThe lookup process goes as follows:\n1. Find closest part\n2. Find closest block in part\n\nThis 2-step approach scales well for large datasets where many blocks are desired to reduce blocking.\n\n## Re-partitioning (to do)\nEach time the partition list is loaded, it must be compared to the configured partition count. If they do not match, a re-partitioning process must occur before serving connections.\n\n1. Create new manifest (partition:block list) in sub-directory\n2. Create new Store\n3. For each current part, re-map all keys to their new part\n4. Write each part after all keys are re-mapped\n\n# Key expiry\nThe expires time is evaluated periodically. The period between scans can be configured using `ExpiryScanPeriod`, giving a number of seconds.\n\n# Protocol\n\n## Msg\nA normal operation is transmitted in the serialized form of `protocol.Msg`.\n```go\ntype Msg struct {\n\tOp      byte\n\tStatus  byte\n\tKey     string\n\tValue   []byte\n\tExpires int64\n}\n```\n\n## Serialization\n```\n| 0             | 1             | 2             | 3             |\n|0 1 2 3 4 5 6 7|0 1 2 3 4 5 6 7|0 1 2 3 4 5 6 7|0 1 2 3 4 5 6 7|\n+---------------+---------------+---------------+---------------+\n| \u003c OP        \u003e | \u003c STATUS    \u003e | \u003c EXPIRES UNIX UINT64         |\n|                                                               |\n|                             \u003e | \u003c KEY LEN UINT16            \u003e |\n  KEY ...                                                       \n  VALUE ...                                                     \n```\nAll integers are big endian.\n\n## Op codes\n| Byte | Meaning |\n|------|---------|\n| 0x01 | Close   |\n| 0x02 | Auth    |\n| 0x10 | Ping    |\n| 0x11 | Pong    |\n| 0x20 | Get     |\n| 0x30 | Set     |\n| 0x31 | SetAck  |\n| 0x40 | Del     |\n| 0x41 | DelAck  |\n| 0x50 | List    |\n\n## Status codes\n| Byte | Rune | Meaning      |\n|------|------|--------------|\n| 0x5F | _    | OK           |\n| 0x2F | /    | StreamEnd    |\n| 0x2E | .    | NotFound     |\n| 0x21 | !    | Error        |\n| 0x23 | #    | Unauthorized |\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fintob%2Frocketkv","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fintob%2Frocketkv","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fintob%2Frocketkv/lists"}