{"id":51558154,"url":"https://github.com/vondrakk/stplr","last_synced_at":"2026-07-10T07:32:15.720Z","repository":{"id":366456605,"uuid":"1274853800","full_name":"vondrakk/stplr","owner":"vondrakk","description":"stplr — a self-managing, horizontally-scaling distributed key/value + set-ops store (BSL 1.1)","archived":false,"fork":false,"pushed_at":"2026-06-21T23:48:26.000Z","size":187,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2026-06-22T00:11:36.703Z","etag":null,"topics":["database","distributed-systems","key-value-store","lmdb","rust"],"latest_commit_sha":null,"homepage":"https://stplr.org","language":"Rust","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"other","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/vondrakk.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2026-06-20T00:57:53.000Z","updated_at":"2026-06-21T23:48:30.000Z","dependencies_parsed_at":null,"dependency_job_id":null,"html_url":"https://github.com/vondrakk/stplr","commit_stats":null,"previous_names":["vondrakk/stplr"],"tags_count":null,"template":false,"template_full_name":null,"purl":"pkg:github/vondrakk/stplr","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/vondrakk%2Fstplr","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/vondrakk%2Fstplr/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/vondrakk%2Fstplr/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/vondrakk%2Fstplr/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/vondrakk","download_url":"https://codeload.github.com/vondrakk/stplr/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/vondrakk%2Fstplr/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":35325188,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-26T15:22:16.424Z","status":"online","status_checked_at":"2026-07-10T02:00:06.465Z","response_time":60,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["database","distributed-systems","key-value-store","lmdb","rust"],"created_at":"2026-07-10T07:32:15.156Z","updated_at":"2026-07-10T07:32:15.698Z","avatar_url":"https://github.com/vondrakk.png","language":"Rust","funding_links":[],"categories":[],"sub_categories":[],"readme":"# stplr\n\n**A self-managing, horizontally-scaling distributed key/value + set-ops store.**\n\nstplr is the open substrate underneath higher-level data systems. It knows keys,\ncollections, opaque value objects, opaque set members, and buckets — and nothing\nabout what's stored in them. Any query or correlation semantics layer *on top* of\nstplr and never leak back down.\n\nThe pitch: **durable-tier throughput at cache-tier operational simplicity.** One\nself-managing binary, started in seconds, that on a single node out-reads and\nout-writes the open durable field — while you scale it by adding shards, not by\nhand-tuning a cluster.\n\n## Performance\n\nOn matched single nodes (4 vCPU, LMDB-durable, 128 clients, 64-byte values), stplr's\nbinary protocol leads the open durable field — Cassandra, ScyllaDB, etcd, TiKV,\nFoundationDB — on read and write throughput:\n\n| Op | stplr | ScyllaDB | etcd | Cassandra |\n|----|------:|---------:|-----:|----------:|\n| GET | ~204k op/s | ~75k | ~38k | ~38k |\n| SET (durable) | ~129k op/s | ~77k | ~29k | ~29k |\n\nIn-memory caches (Valkey/Memcached) win raw point ops by design — different tier,\nnot durable. These are **single-node, per-shard** numbers; ScyllaDB is shard-per-core\nand scales with cores on a fat box, where stplr scales **horizontally** (add shards).\nMethodology + the load harness: see the `stplr-bench` project.\n\n## Demo\n\n```sh\n./demo/demo.sh   # builds stplrd, starts a 3-shard durable cluster + coordinator on localhost\n                 # (leader election, durable ingest queue, bearer auth) and walks every feature\n                 # — write/read, set ops, TTL, CAS/INCR, auth, leader, ingest queue, metrics, backup.\n```\n\nPitch deck: \u003chttps://stplr.org/deck/\u003e.\n\n## Quickstart\n\n```sh\n# a durable shard, HTTP API + the fast binary protocol\nstplrd --store lmdb --path /var/lib/stplr --bind 0.0.0.0:8100 --bin-bind 0.0.0.0:8101\n\n# a coordinator routing across shards (replication + failover)\nstplrd --role coordinator --bind 0.0.0.0:8080 \\\n       --shards s0=10.0.0.1:8100,s1=10.0.0.2:8100 --replication 2\n\n# rack/zone-aware replicas: spread each key's copies across distinct racks\nstplrd --role coordinator --bind 0.0.0.0:8080 \\\n       --shards s0=10.0.0.1:8100,s1=10.0.0.2:8100,s2=10.0.1.1:8100 \\\n       --racks s0=zone-a,s1=zone-b,s2=zone-a --replication 2\n\n# HA: run multiple coordinators with leader election (one drives rebalancing)\nstplrd --role coordinator --bind 0.0.0.0:8080 --coordinator-id coord-a \\\n       --shards s0=10.0.0.1:8100,s1=10.0.0.2:8100 --replication 2\n# GET /leader -\u003e {\"self\",\"leader\",\"isLeader\",\"token\"}\n# POST /admin/rebalance {\"nodes\":[...]}  — runs only on the leader (else 409 + leader)\n# POST /admin/drain {\"node\":\"...\"}        — same leader-gating\n```\n\nPass `--coordinator-id` to enable a TTL-lease election: every coordinator races to\nhold one lease (on the shard that owns it), and the holder is the singleton\nrebalance/migration driver. The driver ops (`/admin/rebalance`, `/admin/drain`) are\nleader-gated — fire them at any coordinator and a non-leader replica replies `409`\nwith the current leader's id (leader-redirect). A fencing `token` increments on\neach change of leader so a deposed leader can detect it. It's a lease (Redis/Consul-\nsession model), not partition-tolerant consensus — if the lease shard is\nunreachable no leader is elected (a safe pause) rather than two.\n\nThe HTTP API is `GET /object?coll=\u0026key=`, `POST /write|/setAdd|/setRemove|/deleteObject`.\nAdd `\"ttlMs\": \u003cn\u003e` to a `/write` body for a per-key **TTL** — the key reads as absent once it\nexpires (lazy) and a background sweeper reclaims it; a plain write clears any TTL. A no-TTL\nworkload pays nothing for the feature (the expiry path is guarded behind a \"any TTLs set?\" flag).\n\n**Atomic ops:** `POST /cas {coll,key,expect?,new}` → `{set}` is compare-and-set (omit `expect` for\nset-if-absent); `POST /incr {coll,key,delta}` → `{value}` is an atomic counter (absent = 0, `null`\nif the value isn't an integer). The read-modify-write runs on the key's **primary** owner under the\nshard write lock, then the result replicates to the other owners — one authoritative arbiter, no\ndivergence.\n\n### Authentication \u0026 transport security\n\nSet `--auth-token \u003ctok\u003e` (or `STPLR_AUTH_TOKEN`) on a shard/coordinator to require\n`Authorization: Bearer \u003ctok\u003e` on every HTTP route (except `/health` + `/metrics`, left open for\nprobes/scrapers) and an `OP_AUTH` handshake on the binary protocol. The coordinator presents the\nsame token to its shards; `HttpShardClient::new_authed` / `SmartClient::from_coordinator_authed`\npresent it client-side. One shared cluster secret; no token = open (dev / trusted mesh).\n\n**TLS:** the listeners speak plaintext — terminate TLS at your mesh / ingress / load balancer (the\ncommon in-cluster pattern, and it keeps the hot path free of a TLS dependency). Native in-process\nTLS (rustls) is a roadmap item.\nThe **binary protocol** (`--bin-bind`) is a framed TCP wire (length-prefixed opcodes,\nno HTTP/JSON envelope) — ~3.5× the HTTP throughput on reads, and the recommended hot path.\n\n### Direct-to-shard client\n\n`stplr::smart::SmartClient` routes reads **and writes** straight to the owning shards via the same\nrendezvous placement the coordinator uses — but evaluated client-side, so the coordinator is off the\ndata path (it stays the control plane: membership, rebalance, leader election). One network hop\ninstead of two.\n\n```rust\n// discover the shards from a coordinator, then talk to them directly\nlet client = stplr::smart::SmartClient::from_coordinator(\"http://coord:8080\", 2).await?;\nclient.put(\"kv\", \"alpha\", json!(\"A\")).await;          // -\u003e written to alpha's replica set directly\nlet v = client.get(\"kv\", \"alpha\").await;              // -\u003e read from a live owner, with failover\n```\n\nIt tracks the coordinator's membership **epoch** and refreshes its routing view when it advances\n(`refresh()`, or `spawn_refresher(ms)`). Strongly consistent under stable membership; eventually\nconsistent during a rebalance until the view refreshes.\n\n### Durable ingest queue\n\nFor crash-safe writes, run a coordinator with `--ingest-queue \u003cdir\u003e`: `POST /enqueue` appends the\nwrite to a durable LMDB write-ahead log and acks with its sequence number *before* applying it. A\nbackground drainer applies queued writes to the shards and advances a durable committed cursor, so a\ncoordinator crash replays only the un-applied tail — **at-least-once** (the ops are idempotent).\n`GET /ingest/status` reports `accepted` / `committed` / `pending`.\n\n```sh\nstplrd --role coordinator --bind 0.0.0.0:8080 --shards s0=10.0.0.1:8100 \\\n       --ingest-queue /var/lib/stplr/ingest\ncurl -X POST localhost:8080/enqueue -H 'content-type: application/json' \\\n     -d '{\"coll\":\"kv\",\"key\":\"alpha\",\"obj\":\"A\"}'   # -\u003e {\"ok\":true,\"durable\":true,\"seq\":1}\n```\n\nOn Kubernetes/OpenShift, deploy with the **stplr-operator** (a `StplrCluster` CR → a\nshard StatefulSet + coordinator with auto-sized PVCs).\n\n## Architecture\n\n- **Rendezvous (HRW) partitioning** over a fixed ring of 4096 virtual buckets, with\n  **top-R replication** so every key survives node loss; deterministic placement means\n  minimal data movement on membership change.\n- **Zero-touch operations:** online rebalance, graceful drain, auto-heal, crash-resume,\n  and live membership — no manual reshard, no downtime, no operator babysitting.\n- **Generic data plane:** an object store `(collection, key) -\u003e value` plus server-side\n  posting-list set operations (`set_add` / `set_remove` / `set_members`).\n- **Composable roles, one binary:** `shard` owns/serves a partition; `coordinator` routes\n  and drives rebalancing. For maximum throughput, clients can route **directly to shards**\n  (rendezvous-hash client-side), keeping the coordinator off the data path.\n- **Durable by default:** LMDB-backed via `heed` (group-committed writes, `MDB_NOSYNC` +\n  periodic sync) — the disk *is* the database. In-memory store available for dev/tests.\n- **Replayable change feed:** a durable, value-level log you can tail and replay for\n  recovery or downstream sync.\n\n## Status\n\nThe substrate is built and benchmarked. Coordinator routing + horizontal sharding are in;\nin flight: direct-to-shard write APIs, coordinator leader election, streaming/scheduled\nbackups + restore. See open PRs.\n\n## License\n\nstplr is licensed under the **Business Source License 1.1** (see [LICENSE](LICENSE)).\n\n- **Licensor:** The Von Drakk Corporation\n- Use, copy, modify, and run stplr in production for any purpose — **except** offering it\n  (or a service deriving its value primarily from it) to third parties as a hosted/managed\n  key-value, distributed-storage, or set-operation service. That requires a separate\n  commercial license.\n- **Change Date:** four years after each version is first published, that version converts\n  to the **Apache License 2.0**.\n\nSource-available today, true open source on a rolling per-version clock.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fvondrakk%2Fstplr","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fvondrakk%2Fstplr","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fvondrakk%2Fstplr/lists"}