{"id":51129737,"url":"https://github.com/malbeclabs/edge-multicast-ref","last_synced_at":"2026-06-25T11:01:18.125Z","repository":{"id":362980395,"uuid":"1193812312","full_name":"malbeclabs/edge-multicast-ref","owner":"malbeclabs","description":"Reference designs for consuming DoubleZero Edge multicast feeds over kernel sockets and XDP","archived":false,"fork":false,"pushed_at":"2026-06-14T21:47:38.000Z","size":40449,"stargazers_count":2,"open_issues_count":4,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2026-06-14T22:10:23.925Z","etag":null,"topics":["doublezero","ebpf","market-data","multicast","networking","shreds","solana","xdp"],"latest_commit_sha":null,"homepage":"https://doublezero.xyz","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/malbeclabs.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2026-03-27T15:51:42.000Z","updated_at":"2026-06-14T20:12:21.000Z","dependencies_parsed_at":null,"dependency_job_id":null,"html_url":"https://github.com/malbeclabs/edge-multicast-ref","commit_stats":null,"previous_names":["malbeclabs/edge-multicast-ref"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/malbeclabs/edge-multicast-ref","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/malbeclabs%2Fedge-multicast-ref","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/malbeclabs%2Fedge-multicast-ref/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/malbeclabs%2Fedge-multicast-ref/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/malbeclabs%2Fedge-multicast-ref/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/malbeclabs","download_url":"https://codeload.github.com/malbeclabs/edge-multicast-ref/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/malbeclabs%2Fedge-multicast-ref/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":34771664,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-26T15:22:16.424Z","status":"online","status_checked_at":"2026-06-25T02:00:05.521Z","response_time":101,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["doublezero","ebpf","market-data","multicast","networking","shreds","solana","xdp"],"created_at":"2026-06-25T11:01:14.612Z","updated_at":"2026-06-25T11:01:18.089Z","avatar_url":"https://github.com/malbeclabs.png","language":"Go","funding_links":[],"categories":[],"sub_categories":[],"readme":"# DoubleZero Edge Multicast Reference Designs\n\nReference implementations for consuming [DoubleZero](https://doublezero.xyz) edge multicast feeds.\n\nDoubleZero delivers data as GRE-encapsulated UDP multicast — the kernel handles GRE de-encapsulation, so applications see clean UDP. This repo covers two kinds of feed that arrive over that transport, each with its own reference code:\n\n- **[Solana shreds](#shred-receivers)** — raw shred packets received directly off the wire. High-throughput, receive-only receivers in Rust and Go, over both kernel sockets and XDP. *This is the repo's original focus.*\n- **[Market data](#market-data-pipelines)** — DoubleZero's binary market-data feeds (see [edge-feed-spec](https://github.com/malbeclabs/edge-feed-spec)), decoded and monitored through a parse → build → store → visualize demo stack.\n\n## Transport\n\nThe feed arrives on a GRE tunnel interface (e.g. `doublezero1`) as UDP multicast. The DoubleZero client handles tunnel setup and heartbeat responses — these reference designs are receive-only.\n\n```\nPhysical NIC:  Eth → Outer IP → GRE → Inner IP → UDP → payload\nGRE interface: Inner IP (148.51.x.x → 233.84.178.1) → UDP → payload\n```\n\nExample interface:\n\n```\n$ ip a s doublezero1\n26: doublezero1@NONE: \u003cPOINTOPOINT,NOARP,UP,LOWER_UP\u003e mtu 1476 qdisc noqueue state UNKNOWN\n    link/gre 64.130.37.175 peer 4.42.212.122\n    inet 169.254.10.233/31 scope link doublezero1\n```\n\n**GRE decapsulator (optional).** [gre-decap](gre-decap/) is a standalone XDP program that strips GRE encapsulation inline on the physical NIC. After decap, the kernel sees plain multicast UDP — no tunnel interface or application changes needed. Useful when you want existing socket-based applications to receive the feed without a GRE tunnel.\n\n## Shred Receivers\n\nReceive Solana shreds directly off the multicast group. Two packet types are present on the shred feed:\n\n- **Shred packets** (port 7733) — Solana shreds (~1247–1272 bytes each)\n- **Heartbeat packets** (port 5765) — 4-byte DoubleZero liveness probes\n\n| Language | Kernel Sockets | XDP |\n|----------|----------------|-----|\n| **Rust** | [rust/kernel-receiver](rust/kernel-receiver/) | [rust/xdp-receiver](rust/xdp-receiver/) |\n| **Go**   | [go/kernel-receiver](go/kernel-receiver/)     | [go/xdp-receiver](go/xdp-receiver/) |\n| **C**    | planned | planned |\n\nKernel-socket receivers are the simple path; XDP receivers are the high-performance path.\n\n## Market Data Pipelines\n\nDecode DoubleZero's binary market-data feeds (defined in [edge-feed-spec](https://github.com/malbeclabs/edge-feed-spec)) and monitor them live. Each feed has a multicast subscriber (**parser**) that decodes the wire format and republishes it on a Unix socket, and a reference **bot** that builds state and persists to ClickHouse.\n\n| Feed | Spec | Implementation |\n|------|------|----------------|\n| Top-of-Book \u0026 Trades | [spec](https://github.com/malbeclabs/edge-feed-spec/blob/main/top-of-book/spec.md) | `go/topofbook-parser`, `go/topofbook-bot` |\n| Market-by-Order | [spec](https://github.com/malbeclabs/edge-feed-spec/blob/main/market-by-order/spec.md) | `go/marketbyorder-parser`, `go/marketbyorder-bot` |\n\n### Top-of-Book \u0026 Trades\n\nConsumes the [Top-of-Book \u0026 Trades feed](https://github.com/malbeclabs/edge-feed-spec/blob/main/top-of-book/spec.md).\n\n| Component | Description |\n|---|---|\n| [go/topofbook-parser](go/topofbook-parser/) | Multicast subscriber. Decodes frames, writes JSON/CSV to a file or Unix socket, exposes Prometheus metrics |\n| [go/topofbook-bot](go/topofbook-bot/) | Reads the parser's Unix socket, filters by symbol, exposes per-symbol top-of-book state as Prometheus metrics, and optionally writes every tick to ClickHouse |\n\n### Market-by-Order\n\nConsumes the [Market-by-Order feed](https://github.com/malbeclabs/edge-feed-spec/blob/main/market-by-order/spec.md).\n\n| Component | Description |\n|---|---|\n| [go/marketbyorder-parser](go/marketbyorder-parser/) | Three-port multicast subscriber + binary wire decoder, broadcasts decoded JSONL on a Unix socket |\n| [go/marketbyorder-bot](go/marketbyorder-bot/) | Book builder + persistor. Maintains in-memory market-by-order books and writes per-event rows + coalesced top-N level snapshots to ClickHouse |\n\n### Demo Stack\n\n[demo/](demo/) is a one-command Docker Compose stack — parsers + bots + ClickHouse + Grafana — with pre-provisioned dashboards for both feeds: per-symbol top-of-book state, and an order-book view with ladder, depth heatmap, spread, trade tape, and event-rate panels.\n\n## Sample Captures\n\nThe `pcaps/` directory contains sample packet captures from a live DoubleZero edge feed. These can be inspected with Wireshark using the `solana.shreds` dissector (decode as UDP port 7733 → `solana.shreds`).\n\n## Target Audience\n\nTraders and operators already familiar with tools like the [jito shredstream-proxy](https://github.com/jito-labs/shredstream-proxy) who want to consume DoubleZero edge multicast feeds directly.\n\n## License\n\nLicensed under the **Apache License 2.0**.\n\nSee [LICENSE](./LICENSE) for details.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmalbeclabs%2Fedge-multicast-ref","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fmalbeclabs%2Fedge-multicast-ref","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmalbeclabs%2Fedge-multicast-ref/lists"}