{"id":51442819,"url":"https://github.com/prof-faustus/memserve","last_synced_at":"2026-07-05T13:01:43.369Z","repository":{"id":364955842,"uuid":"1269798750","full_name":"prof-faustus/memserve","owner":"prof-faustus","description":"In-memory, hash-sharded transaction-lookup fabric over Teranode (BSV). Seen/Mined/Merkle-path/UTXO from memory; pay-per-use BSV payment channels; spend-depth pruning.","archived":false,"fork":false,"pushed_at":"2026-06-15T08:26:40.000Z","size":155,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2026-06-15T09:22:09.391Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"other","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/prof-faustus.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":"SECURITY.md","support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2026-06-15T05:33:43.000Z","updated_at":"2026-06-15T08:26:44.000Z","dependencies_parsed_at":null,"dependency_job_id":null,"html_url":"https://github.com/prof-faustus/memserve","commit_stats":null,"previous_names":["prof-faustus/memserve"],"tags_count":null,"template":false,"template_full_name":null,"purl":"pkg:github/prof-faustus/memserve","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/prof-faustus%2Fmemserve","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/prof-faustus%2Fmemserve/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/prof-faustus%2Fmemserve/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/prof-faustus%2Fmemserve/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/prof-faustus","download_url":"https://codeload.github.com/prof-faustus/memserve/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/prof-faustus%2Fmemserve/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":35154748,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-26T15:22:16.424Z","status":"online","status_checked_at":"2026-07-05T02:00:06.290Z","response_time":100,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2026-07-05T13:01:42.415Z","updated_at":"2026-07-05T13:01:43.356Z","avatar_url":"https://github.com/prof-faustus.png","language":"Go","funding_links":[],"categories":[],"sub_categories":[],"readme":"# MemServe\n\n**An in-memory, hash-sharded transaction-lookup fabric over Teranode (BSV only).**\n\nMemServe is the **proof-serving / status layer** that complements the\n[MF-SPV verification fabric](https://github.com/prof-faustus/mfspv). Verification is\nalready solved and shown to scale linearly (~3.2×10⁸ inclusion-verifications/s per 64-core\nbox; shares-nothing to 10¹⁰–10¹¹ across nodes). The remaining bottleneck is the **pull\nside** — *serving* the Merkle proof and answering *\"is this tx mined?\"* MemServe turns that\ninto a distributed **O(1) in-memory key lookup** backed by Aerospike, pulling its data from\nTeranode.\n\n\u003e **Status: implemented (v1), hardening toward production.** Full pipeline + a\n\u003e commercial-grade HTTP/JSON server daemon, a real Teranode HTTP ingest adapter, pay-per-use\n\u003e payment channels (signing **real BSV transactions** — funding 2-of-2, commitment/settlement/\n\u003e refund, FORKID sighash, DER+low-S) with abuse defenses, spend-depth pruning with a named\n\u003e reorg-horizon correctness floor, a trust-minimizing multi-operator client, and signed-answer\n\u003e accountability (fraud proofs bound to the miner). Tested in CI under `-race`. See\n\u003e [`DESIGN.md`](./DESIGN.md) and [`SECURITY.md`](./SECURITY.md).\n\u003e\n\u003e **Remaining for production (live-infra only):** broadcast-validate the channel txs on **BSV\n\u003e testnet** (§10.6); point ingest at a **live Teranode** (thin endpoint mapping); run against an\n\u003e **Aerospike cluster** (`aerospike` tag); and validate the **GPU verify kernel** (`cuda` tag,\n\u003e gated by `accel.Validate`). The off-chain accounting, sighash, and tx construction are built\n\u003e and tested; only consensus acceptance on a real network/cluster/GPU remains.\n\n## What it answers\n\nFor a `txid` (and optionally `outpoint = txid:vout`):\n\n- **Seen?** — known to the network? → bool + first-seen time\n- **Mined?** — in a block? → `{blockHash, height, blockTime}` (when)\n- **Merkle path** — the inclusion proof (the same `fabric.Proof` the verifier consumes)\n- **UTXO** — `{unspent, value}` for double-spend / value-conservation checks\n\n## Key ideas\n\n- **Hash-prefix sharding.** A txid is a uniform hash, so its **leading bits** partition the\n  key space evenly: **k bits → 2ᵏ shards**. Uniform load, stateless prefix routing, elastic\n  split (raise k by 1 to double capacity). Shares-nothing, like the verifier.\n- **Ingest from Teranode.** A run server pulls subtrees, sealed blocks, and UTXO deltas and\n  indexes them into Aerospike, sharded by prefix. It does **not** build or store full blocks\n  — consensus stays in Teranode; MemServe is a lookup/index, not a node.\n- **Pay-per-use (BSV payment channel).** Native unidirectional micropayment channel,\n  **prepay-then-serve** (the server can never lose an access), **per-shard channels** (more\n  private, no central hub), **one signature per access**, configurable pricing (flat or\n  metered), settle on **n accesses and/or time x**, with the **settlement fee built in**\n  (covers the on-chain mining fee). See `DESIGN.md §10`.\n- **Spend-depth pruning.** Not an archive: once a spend is buried **D blocks deep** (a\n  per-server policy, e.g. \"run 10 deep\") the record is **pruned and freed**. Live (unspent)\n  outputs are never pruned. Bounds memory to the live set + recent spends. See `DESIGN.md §11`.\n\n## Layout\n\n```\ncmd/memserved  the production daemon (HTTP/JSON server; -mock or -teranode \u003curl\u003e)\ncmd/memserve   demo + benchmark (ingest -\u003e serve -\u003e verify -\u003e pay -\u003e accel -\u003e shard scale)\ncmd/ingest     standalone ingest demo: pulls a Teranode source, drives spend-depth pruning\nserver/        commercial HTTP/JSON service: health, metrics, rate limit, admin, signed answers\nshard/         hash-prefix sharding + stateless routing (k bits -\u003e 2^k shards)\nstore/         Store interface + records; store/mem (striped, default), store/aerospike (tag)\nteranode/      ingest source + Merkle-consistent mock; teranode/httpsource (real HTTP adapter)\ningest/        indexes blocks (with anti-poisoning validation), reorg rollback, prune driver\nproof/         Merkle-path assembly + verification (reuses the commitment fold)\napi/           Seen / Mined / MerklePath / UTXO, with honest post-prune semantics\nprune/         spend-depth retention with the named reorg-horizon correctness floor\npayment/       pay-per-use BSV channel (prepay-then-serve) + abuse defenses + alert path\nattest/        signed answers, miner endorsement, fraud proofs (accountability)\nclient/        trust-minimizing multi-operator client (verify proofs; quorum; fraud detection)\naccel/         batch secp256k1 verify (CPU + cuda tag), batching gate, differential validator\ncommitment/    SHA-256d Merkle core (vendored from MF-SPV)\ncrypto/        secp256k1 ECDSA, RFC 6979, low-S, constant-time scalar mult (vendored from MF-SPV)\n```\n\n## Run the server (miner sidecar)\n\n```sh\ngo run ./cmd/memserved -mock -addr :8080                      # demo against the built-in mock\ngo run ./cmd/memserved -teranode http://teranode:8090 \\\n   -operator-seed \u003chex32\u003e -admin-token \u003ctok\u003e -min-deposit 100000 -rate 500\n# probes: /healthz /readyz /metrics ; queries: /v1/seen|mined|merklepath|utxo ;\n# payment: /v1/channel/open /v1/quote /v1/paid/query ; admin: /admin/stats (Bearer token)\n```\n\n### Deployment fronts (built to the boundary)\n\n```sh\nmake aerospike-up \u0026\u0026 make aerospike-test   # local Aerospike + run the store conformance suite\ngo run -tags aerospike ./cmd/memserved -mock -store aerospike -aerospike-host 127.0.0.1\nmake accelcheck                            # validate the CPU verify backend vs the reference\nmake cuda \u0026\u0026 make cuda-check               # build + validate the GPU kernel (needs nvcc + GPU)\n```\n\nRemaining live-infra steps: point `-teranode` at a real node (two endpoint templates),\nrun the conformance suite against your Aerospike cluster, and validate the CUDA kernel on\na GPU. Everything else is built and tested.\n\n## Build / test / run\n\n```sh\nmake            # fmtcheck + vet + build + test\nmake race       # tests under the race detector\nmake demo       # cmd/memserve: end-to-end + throughput + shard extrapolation\nmake ingest     # cmd/ingest: spend-depth pruning bounding memory as the chain grows\nmake aerospike  # compile the Aerospike-backed store (pulls the client library)\n```\n\nMeasured on a 64-core box (`go run ./cmd/memserve`): in-memory lookups scale across cores\n(Seen/Mined ~1.8×10⁸ answers/s, UTXO ~1.4×10⁸; the store is internally striped by hash\nprefix so it does not serialize on one lock). MerklePath is CPU-bound (the path is rebuilt\non read — the honest pull cost). The paid path is bound by secp256k1 verification (the true\ncost of metered access, reported, not hidden). Aggregate scales shares-nothing as\nper-shard × 2ᵏ shards.\n\n## License\n\n[Open BSV License version 4](./LICENSE). BSV / Teranode only — no BTC code, parameters, or\nassumptions.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fprof-faustus%2Fmemserve","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fprof-faustus%2Fmemserve","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fprof-faustus%2Fmemserve/lists"}