{"id":48093973,"url":"https://github.com/nostrability/outbox","last_synced_at":"2026-04-04T15:34:07.002Z","repository":{"id":339618653,"uuid":"1162636867","full_name":"nostrability/outbox","owner":"nostrability","description":"outbox performance measured","archived":false,"fork":false,"pushed_at":"2026-03-14T14:24:15.000Z","size":2495,"stargazers_count":1,"open_issues_count":12,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2026-03-15T01:30:52.555Z","etag":null,"topics":["benchmarking","maximum-coverage","maximum-covering-problems","nip-65","nip-66","nostr","outbox"],"latest_commit_sha":null,"homepage":"","language":"TypeScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/nostrability.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":"AGENTS.md","dco":null,"cla":null}},"created_at":"2026-02-20T14:08:23.000Z","updated_at":"2026-03-14T14:09:01.000Z","dependencies_parsed_at":null,"dependency_job_id":null,"html_url":"https://github.com/nostrability/outbox","commit_stats":null,"previous_names":["nostrability/outbox"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/nostrability/outbox","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/nostrability%2Foutbox","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/nostrability%2Foutbox/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/nostrability%2Foutbox/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/nostrability%2Foutbox/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/nostrability","download_url":"https://codeload.github.com/nostrability/outbox/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/nostrability%2Foutbox/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":31403959,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-04-04T10:20:44.708Z","status":"ssl_error","status_checked_at":"2026-04-04T10:20:06.846Z","response_time":60,"last_error":"SSL_read: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["benchmarking","maximum-coverage","maximum-covering-problems","nip-65","nip-66","nostr","outbox"],"created_at":"2026-04-04T15:34:06.752Z","updated_at":"2026-04-04T15:34:06.903Z","avatar_url":"https://github.com/nostrability.png","language":"TypeScript","readme":"# Outbox Model: What Actually Works\n\nThe relay that \"should\" have the event often doesn't — due to retention, downtime, silent write failures, or auth restrictions. We tested 27 relay selection algorithms against 12 real profiles to find what actually works.\n\n**Full report:** [OUTBOX-REPORT.md](OUTBOX-REPORT.md) | **Code examples:** [IMPLEMENTATION-GUIDE.md](IMPLEMENTATION-GUIDE.md) | **Reproduce results:** [Benchmark-recreation.md](Benchmark-recreation.md) | **Parent issue:** [nostrability#69](https://github.com/nostrability/nostrability/issues/69)\n\n## Using a client library? Start here.\n\n| If you use… | You're at step… | Next upgrade | Details |\n|---|:---:|---|---|\n| **Welshman/Coracle** | Stochastic scoring | Add Thompson Sampling — +9pp at 1yr (paired); cross-profile baseline ~24% | [cheat sheet](analysis/clients/welshman-coracle.md) |\n| **NDK** | Priority-based | Add NDK+Thompson CG3 — ~16% → ~27% (65% increase), fixes fiatjaf regression | [cheat sheet](analysis/clients/ndk-applesauce-nostrudel.md) |\n| **Applesauce/noStrudel** | Greedy set-cover | Stochastic scoring then Thompson — ~16% → ~40% (140% increase), two steps | [cheat sheet](analysis/clients/ndk-applesauce-nostrudel.md) |\n| **Gossip** | Greedy set-cover | Stochastic scoring then Thompson — ~16% → ~40% (140% increase), two steps | [cheat sheet](analysis/clients/gossip.md) |\n| **rust-nostr** | Filter decomposition | Add FD+Thompson — ~25% → ~37% (50% increase) | [cheat sheet](analysis/clients/rust-nostr-voyage-nosotros-wisp-shopstr.md) |\n| **Amethyst** | Direct mapping | Add NIP-66 filtering — cuts load time by ~45% | [cheat sheet](analysis/clients/amethyst.md) |\n| **Nostur** | Coverage sort | Remove skipTopRelays, add stochastic factor — recovers 5–12% lost coverage | [cheat sheet](analysis/clients/nostur-yakihonne-notedeck.md) |\n| **Ditto-Mew** | 4 app relays | Add hybrid outbox — ~10% → ~23% (130% increase), ~80 LOC | [details below](#full-outbox-vs-hybrid-outbox) |\n| **Nothing yet** | — | Start with hybrid outbox or big relays, then add full outbox when ready | [IMPLEMENTATION-GUIDE.md](IMPLEMENTATION-GUIDE.md) |\n\n## If you're building from scratch\n\n- **Filter dead relays** ([NIP-66](https://github.com/nostr-protocol/nips/blob/master/66.md)) — nearly half of declared relays are offline. Removing them halves your load time.\n- **Add randomness** — deterministic algorithms pick the same popular relays that prune old events. Stochastic selection finds ~50% more events at 1yr.\n- **Learn from delivery** (Thompson Sampling) — track which relays actually return events and feed it back. +9pp at 1yr (~30% → ~39%, paired benchmark), ~80 LOC.\n- **EOSE-race with 2s grace** — query 20 relays in parallel, stop 2s after the first finishes. 86–99% completeness in under 3s.\n\n## What each step buys you\n\n| Step | What you do | 1yr recall | Effort |\n|:---:|---|:---:|---|\n| 0 | **Hardcode big relays** (damus + nos.lol) | ~8% | Zero |\n| 1a | **Basic outbox** (greedy set-cover from NIP-65) | ~16% | Medium — ~200 LOC |\n| 1b | **Hybrid outbox** (keep app relays + add author write relays) | ~23% | Low — ~80 LOC |\n| 2 | **Stochastic scoring** (Welshman's random factor) | ~24% | Low — ~50 LOC |\n| 3 | **Filter dead relays** (NIP-66 liveness) | neutral recall, −45% latency | Low — ~30 LOC |\n| 4 | **Learn from delivery** (Thompson Sampling) | ~40% | Low — ~80 LOC + DB table |\n| 4+ | **Learn relay speed** (latency discount) | same recall, faster feed fill | 1 line on top of Step 4 |\n\nSteps 1a/1b are alternative entry points. 1a replaces your routing layer, 1b augments it. Steps 2–4 are incremental on the 1a (full outbox) path. The 1b (hybrid) path gets Thompson Sampling directly — see [Hybrid+Thompson](OUTBOX-REPORT.md#85-hybrid-outbox-app-relay-broadcast--per-author-thompson) for gains on that path. See [OUTBOX-REPORT.md](OUTBOX-REPORT.md) for per-profile data and methodology.\n\n## Full outbox vs hybrid outbox\n\n\u003cdetails\u003e\n\u003csummary\u003eExpand comparison\u003c/summary\u003e\n\n**Full outbox** — replace your relay selection layer. Route queries to each author's NIP-65 write relays. Used by Welshman/Coracle, rust-nostr, NDK, Gossip.\n\n**Hybrid outbox** — keep app relays for the main feed, add outbox queries for profile views, event lookups, and threads. ~80 LOC, no routing layer changes.\n\n| | Full outbox | Hybrid outbox |\n|---|---|---|\n| **1yr recall** | ~40% | ~23% |\n| **Feed latency** | Depends on per-author relay quality | Unchanged (app relays) |\n| **What changes** | Routing layer | Individual hooks (profile, event, thread) |\n| **Connections** | 20+ (budgeted across follows) | 4 app relays + 3 per viewed profile |\n| **Engineering effort** | ~200–500 LOC | ~80 LOC |\n| **Best for** | Clients building relay routing from scratch | Clients with fixed app relays |\n\n**Decision tree:**\n\n```text\nDo you have a routing layer that selects relays per-author?\n├─ Yes → Add Thompson Sampling to it (Step 4)\n│\n└─ No (fixed app relays / broadcast)\n   ├─ Can you rewrite your routing layer?\n   │  └─ Yes → Implement full outbox (Step 1a → Step 4)\n   │\n   └─ No, or need to preserve feed latency?\n      └─ Add hybrid outbox (Step 1b) — ~80 LOC, no routing changes\n```\n\n\u003c/details\u003e\n\n\u003cdetails\u003e\n\u003csummary\u003eAlgorithm quick reference (27 algorithms)\u003c/summary\u003e\n\n**Deployed in clients:**\n\n| Algorithm | Used by | 1yr recall | 7d recall | Verdict |\n|---|---|:---:|:---:|---|\n| **Welshman+Thompson** | *not yet deployed* | ~40% | ~83% | Upgrade path for Coracle — learns from delivery |\n| **FD+Thompson** | *not yet deployed* | ~37% | ~84% | Upgrade path for rust-nostr — learns from delivery |\n| **Hybrid+Thompson** | *not yet deployed* | ~23% | — | Upgrade path for app-relay clients |\n| **Filter Decomposition** | rust-nostr | ~25% | ~77% | Per-author top-N write relays |\n| **Welshman Stochastic** | Coracle | ~24% | ~83% | Best stateless deployed algorithm |\n| **Greedy Set-Cover** | Gossip, Applesauce, Wisp | ~16% | ~84% | Best on-paper coverage; degrades for history |\n| **NDK+Thompson CG3** | *not yet deployed* | ~27% | — | Recommended NDK variant — fixes regressions |\n| **NDK Priority** | NDK | ~16% | ~83% | Similar to Greedy |\n| **Coverage Sort** | Nostur | ~16% | ~65% | skipTopRelays costs 5–12% coverage |\n\n**Baselines:**\n\n| Baseline | 1yr recall | 7d recall | What it is |\n|---|:---:|:---:|---|\n| Direct Mapping | ~30% | ~88% | All declared write relays — unlimited connections |\n| Ditto-Mew | ~6% | ~62% | 4 hardcoded app relays |\n| Big Relays | ~8% | ~61% | Just damus + nos.lol |\n\nAll values are 6-profile means. See [OUTBOX-REPORT.md § 8](OUTBOX-REPORT.md#8-benchmark-results) for per-profile data, confidence intervals, and the full 25-algorithm table.\n\n\u003c/details\u003e\n\n\u003cdetails\u003e\n\u003csummary\u003eKey findings detail\u003c/summary\u003e\n\n**1. Learning beats static optimization.** Greedy set-cover (Gossip, Applesauce) picks the \"best\" relays on paper but never learns whether they actually deliver. Thompson Sampling tracks delivery and reaches ~40% at 1yr vs ~16% for greedy. [Report § 8.2](OUTBOX-REPORT.md#82-approximating-real-world-conditions-event-verification)\n\n**2. Dead relay filtering saves your connection budget.** Nearly half of declared relays are offline. NIP-66 filtering removes them, cutting load time by 45% (protocol-level benchmark — see [Benchmark-recreation.md](Benchmark-recreation.md#nip-66-liveness-comparison) for methodology). Recall impact is roughly neutral. [Report § 5.3](OUTBOX-REPORT.md#53-misconfigured-relay-lists)\n\n**3. Per-author relay diversity beats popularity.** Algorithms that give each author their own relay picks (Filter Decomposition, Welshman stochastic) find 1.5× more events at 1yr than popularity-based selection. [Report § 8.2](OUTBOX-REPORT.md#82-approximating-real-world-conditions-event-verification)\n\n**4. EOSE-race with 2s grace is the latency sweet spot.** First event arrives in 530–670ms regardless of algorithm. At +2s grace, you have 86–99% of events. [Report § 8.17](OUTBOX-REPORT.md#817-latency-simulation)\n\n**5. Latency-aware scoring helps small follow graphs.** A 1-line latency discount gives +10pp completeness at 2s for \u003c500 follows, with negligible recall cost. Steep tradeoff for 1000+ follows. [Report § 8.16](OUTBOX-REPORT.md#816-latency-aware-thompson-sampling)\n\n**6. 20 relay connections is enough for most users.** Small graphs saturate at 10–15 relays. Medium graphs benefit from 20. Beyond 20 shows diminishing returns. [Report § 8.13](OUTBOX-REPORT.md#813-adaptive-connection-limits)\n\n\u003c/details\u003e\n\n## How to implement\n\nSee **[IMPLEMENTATION-GUIDE.md](IMPLEMENTATION-GUIDE.md)** for Thompson Sampling, hybrid outbox, NIP-66 filtering, FD+Thompson, and latency-aware scoring — all with code examples and integration guides.\n\n## Running the benchmark\n\nPrerequisites: [Deno](https://deno.com/) v2+\n\n```bash\ncd bench\n\n# Assignment coverage (fast, no network after initial fetch)\ndeno task bench \u003cnpub_or_hex\u003e\n\n# Event retrieval — connects to real relays\ndeno task bench \u003cnpub_or_hex\u003e --verify\n\n# With NIP-66 liveness filter\ndeno task bench \u003cnpub_or_hex\u003e --verify --nip66-filter liveness\n\n# Multi-session Thompson Sampling (5 learning sessions)\nbash run-benchmark-batch.sh\n```\n\nRun `deno task bench --help` for all options. See [Benchmark-recreation.md](Benchmark-recreation.md) for full reproduction instructions.\n\n## Help wanted: benchmark from your location\n\nAll data was collected from a single observer. Relay latency and success rates are location-dependent. We need runs from different locations to validate generalizability.\n\n**What to run** (~30 min, needs Deno v2+):\n\n```bash\ncd bench\ndeno task bench 3bf0c63fcb93463407af97a5e5ee64fa883d107ef9e558472c4eb9aaaefa459d \\\n  --verify --verify-window 604800 \\\n  --nip66-filter liveness --no-phase2-cache \\\n  --output both\n```\n\n**What to share:** Open an issue with your JSON file from `bench/results/`, your approximate location, and connection type.\n\n\u003cdetails\u003e\n\u003csummary\u003eRepo structure\u003c/summary\u003e\n\n```text\nOUTBOX-REPORT.md              Full analysis report (methodology + all data)\nIMPLEMENTATION-GUIDE.md       How to implement the recommendations above\nBenchmark-recreation.md       Step-by-step reproduction instructions\nbench/                        Benchmark tool (Deno/TypeScript)\n  main.ts                     CLI entry point\n  src/algorithms/             25 algorithm implementations (+2 latency-aware variants)\n  src/phase2/                 Event verification + baseline cache\n  src/nip66/                  NIP-66 relay liveness filter\n  src/relay-scores.ts         Thompson Sampling score persistence\n  probe-nip11.ts              NIP-11 relay classification probe\n  run-benchmark-batch.sh      Multi-session batch runner\n  results/                    JSON benchmark outputs\nanalysis/\n  clients/                    Per-client cheat sheets (6 files)\n  cross-client-comparison.md  Cross-client comparison by decision point\n```\n\n\u003c/details\u003e\n\n## Links\n\n- [Full Analysis Report](OUTBOX-REPORT.md) — 15-client cross-analysis + complete benchmark data\n- [Implementation Guide](IMPLEMENTATION-GUIDE.md) — Detailed recommendations with code examples\n- [Cross-Client Comparison](analysis/cross-client-comparison.md) — How 15 clients make each decision\n- [Benchmark Recreation](Benchmark-recreation.md) — Reproduce all results\n- [nostrability#69](https://github.com/nostrability/nostrability/issues/69) — Parent issue\n- [NIP-65](https://github.com/nostr-protocol/nips/blob/master/65.md) — Relay List Metadata specification\n- [Building Nostr](https://building-nostr.coracle.social) — Protocol architecture guide\n- [replicatr](https://github.com/coracle-social/replicatr) — Event replication daemon for relay list changes\n\n*Benchmark data collected February 2026. Relay state changes continuously — relative algorithm rankings should be stable; absolute recall percentages will vary on re-run.*\n","funding_links":[],"categories":[],"sub_categories":[],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fnostrability%2Foutbox","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fnostrability%2Foutbox","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fnostrability%2Foutbox/lists"}