{"id":42213624,"url":"https://github.com/goceleris/benchmarks","last_synced_at":"2026-04-01T17:44:38.754Z","repository":{"id":331676701,"uuid":"1126715799","full_name":"goceleris/benchmarks","owner":"goceleris","description":"Official reproducible benchmark suite for comparing Go HTTP server throughput and latency. Tests production frameworks against theoretical maximum implementations using raw syscalls.","archived":false,"fork":false,"pushed_at":"2026-03-25T17:20:35.000Z","size":26714,"stargazers_count":0,"open_issues_count":3,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2026-03-25T22:59:59.551Z","etag":null,"topics":["benchmarks","docker","latency","load-testing","performance","throughput"],"latest_commit_sha":null,"homepage":"https://goceleris.dev/","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/goceleris.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2026-01-02T12:57:35.000Z","updated_at":"2026-03-25T07:33:37.000Z","dependencies_parsed_at":null,"dependency_job_id":null,"html_url":"https://github.com/goceleris/benchmarks","commit_stats":null,"previous_names":["goceleris/benchmarks"],"tags_count":19,"template":false,"template_full_name":null,"purl":"pkg:github/goceleris/benchmarks","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/goceleris%2Fbenchmarks","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/goceleris%2Fbenchmarks/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/goceleris%2Fbenchmarks/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/goceleris%2Fbenchmarks/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/goceleris","download_url":"https://codeload.github.com/goceleris/benchmarks/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/goceleris%2Fbenchmarks/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":31290571,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-04-01T13:12:26.723Z","status":"ssl_error","status_checked_at":"2026-04-01T13:12:25.102Z","response_time":53,"last_error":"SSL_read: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["benchmarks","docker","latency","load-testing","performance","throughput"],"created_at":"2026-01-27T01:08:02.267Z","updated_at":"2026-04-01T17:44:38.748Z","avatar_url":"https://github.com/goceleris.png","language":"Go","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Celeris Benchmarks\n\nReproducible HTTP server benchmarks on dedicated bare-metal hardware with 10GbE point-to-point networking. Compares production Go frameworks against theoretical maximum performance using raw Linux syscalls.\n\n## Why This Exists\n\nMost HTTP benchmarks run on shared VMs with noisy neighbors, variable network hops, and throttled I/O — making results unreliable and non-reproducible. This suite runs on dedicated bare-metal machines with direct 10GbE links, automated kernel tuning, and CPU pinning, so every release gets consistent, comparable numbers.\n\nWe measure three categories of servers:\n- **Baseline**: Production Go frameworks (Gin, Fiber, Echo, Chi, Iris, Hertz, FastHTTP, stdlib)\n- **Celeris**: The [Celeris](https://github.com/goceleris/celeris) HTTP engine with io_uring, epoll, and adaptive backends\n- **Theoretical**: Raw epoll/io_uring implementations showing the syscall performance ceiling\n\n## Hardware\n\nThree dedicated Minisforum mini PCs connected via 10GbE point-to-point links:\n\n| Machine | Role | CPU | Cores/Threads | RAM | Network |\n|---------|------|-----|---------------|-----|---------|\n| MS-A2 | Client (self-hosted runner) | AMD Ryzen 9 9955HX (Zen 5) | 16C/32T | 32 GB DDR5 | 10GbE SFP+ |\n| MS-A2 | x86 Server | AMD Ryzen 7 7745HX (Zen 4) | 8C/16T | 32 GB DDR5 | 10GbE SFP+ |\n| MS-R1 | ARM64 Server | CIX CP8180 | 12C/12T | 64 GB LPDDR5 | Dual 10GbE RJ45 (RTL8127) |\n\nAll machines run Debian 13 (Trixie) with kernel 6.12+ for full io_uring support. The client machine is the GitHub Actions self-hosted runner that orchestrates everything via SSH.\n\n## Benchmark Types\n\n### Standard Level (7 types, ~66 min per architecture)\n\n| Type | Endpoint | What It Tests |\n|------|----------|---------------|\n| `simple` | `GET /` | Plain text — pure framework overhead |\n| `json` | `GET /json` | JSON serialization |\n| `path` | `GET /users/:id` | Path parameter extraction + routing |\n| `body` | `POST /upload` | 2 KB request body read |\n| `headers` | `GET /users/:id` | Realistic API headers (~850 bytes: JWT, cookies, tracing) |\n| `json-64k` | `GET /json-64k` | 64 KB JSON response — I/O throughput, efficiency metric |\n| `churn` | `GET /` | New TCP connection per request — tests `accept()`, `SO_REUSEPORT` |\n\n### Full Level (15 types, ~142 min per architecture)\n\nAdds a **concurrency sweep** that scales connections from 1 to 10,000 on the `simple` endpoint:\n\n```\nsimple@1  simple@10  simple@50  simple@100  simple@500  simple@1000  simple@5000  simple@10000\n```\n\nThis produces scaling curves that show where goroutine-based frameworks plateau and where event-loop servers keep climbing.\n\n## Servers Tested\n\n### Production Frameworks (Baseline)\n\n| Server | Protocols | Framework |\n|--------|-----------|-----------|\n| stdhttp | H1, H2C, Hybrid | Go stdlib `net/http` |\n| gin | H1, H2C, Hybrid | [Gin](https://github.com/gin-gonic/gin) |\n| echo | H1, H2C, Hybrid | [Echo](https://github.com/labstack/echo) |\n| chi | H1, H2C, Hybrid | [Chi](https://github.com/go-chi/chi) |\n| iris | H1, H2C, Hybrid | [Iris](https://github.com/kataras/iris) |\n| hertz | H1, H2C, Hybrid | [Hertz](https://github.com/cloudwego/hertz) |\n| fiber | H1 | [Fiber](https://github.com/gofiber/fiber) (fasthttp-based) |\n| fasthttp | H1 | [FastHTTP](https://github.com/valyala/fasthttp) |\n\n### Celeris\n\n| Server | Protocols | Engine |\n|--------|-----------|--------|\n| celeris-iouring | H1, H2C, Hybrid | io_uring (Linux 5.10+) |\n| celeris-epoll | H1, H2C, Hybrid | epoll (Linux 2.6+) |\n| celeris-adaptive | H1, H2C, Hybrid | Runtime engine selection |\n\nEach engine runs with three resource profiles: `latency`, `throughput`, and `balanced`.\n\n### Theoretical Maximum\n\n| Server | Protocols | Implementation |\n|--------|-----------|----------------|\n| epoll | H1, H2C, Hybrid | Raw epoll with SO_REUSEPORT, SIMD header parsing, zero-alloc response path |\n| iouring | H1, H2C, Hybrid | io_uring with SQPOLL, multishot accept, linked SQEs |\n\n## Dashboard \u0026 Results\n\nResults are published to [goceleris/docs](https://github.com/goceleris/docs) as dashboard-format JSON (schema v4.0), keyed by Celeris version:\n\n- `results/latest/{arch}.json` — most recent run\n- `results/{version}/{arch}.json` — per-version archive\n\nDashboard data includes:\n- **RPS and latency percentiles** (P50, P75, P90, P99, P999, P9999) per server per benchmark type\n- **Concurrency scaling curves** — RPS at each concurrency level (full level only)\n- **Efficiency metric** — RPS / Server CPU% per server, normalizing across core counts\n- **System metrics** — server CPU, memory RSS, GC pauses (Go servers only)\n- **Timeseries** — per-second RPS and P99 latency snapshots\n\n## Running Benchmarks\n\nBenchmarks are designed to run through GitHub Actions workflows. The self-hosted runner on the client machine handles everything: SSH into servers, deploy binaries, tune kernels, run benchmarks, collect results.\n\n### Via GitHub Actions (Primary Method)\n\n- **Release benchmarks**: Trigger automatically on every release, or manually via the `benchmark.yml` workflow dispatch. Releases run at `full` level (includes concurrency sweep).\n- **PR benchmarks**: Add the `benchmark` label to a pull request. Runs at `standard` level.\n\n### Local Development\n\nFor local development and testing (not full benchmarks):\n\n```bash\n# Build server and bench binaries\nmage build\n\n# Run a quick local smoke test (5s per server, localhost)\nmage benchmarkQuick\n```\n\n## CI/CD\n\n| Workflow | Trigger | Level | Timeout |\n|----------|---------|-------|---------|\n| `benchmark.yml` | Release (auto) or manual dispatch | `full` on release, configurable on manual | 480 min |\n| `benchmark-pr.yml` | PR with `benchmark` label | `standard` | 240 min |\n\nBoth workflows SSH to the bare-metal servers, deploy the server binary, run benchmarks, and collect results. Release runs also publish to the docs repository and trigger a site rebuild.\n\n## Project Structure\n\n```\ncmd/bench/          Benchmark runner CLI (specs, runner, checkpoint)\ncmd/server/         Server binary (all implementations + control daemon)\nservers/\n  baseline/         Production frameworks (gin, echo, chi, iris, etc.)\n  celeris/          Celeris HTTP engine\n  theoretical/      Raw epoll/iouring implementations\n  common/           Shared types, payload generators, SIMD helpers\ninternal/\n  dashboard/        Dashboard JSON format (schema v4.0)\n  metrics/          Prometheus metrics definitions\n  version/          Version info\nconfig/\n  hosts.json        Machine addresses and hardware metadata\n```\n\n## Contributing\n\n### Requirements\n\n- **Go 1.24+**: [Download](https://go.dev/dl/)\n- **Mage**: `go install github.com/magefile/mage@latest`\n\n### Development\n\n```bash\nmage check    # deps + lint + vet + build\nmage test     # run tests\nmage fmt      # format code\n```\n\n### Adding a Server\n\n1. Create a package under `servers/baseline/` (or `servers/theoretical/`)\n2. Implement all benchmark endpoints: `GET /`, `GET /json`, `GET /json-1k`, `GET /json-64k`, `GET /users/:id`, `POST /upload`\n3. Register the server type in `cmd/server/main.go`\n4. Add to the server list in `cmd/bench/main.go`\n\n### Adding a Benchmark Type\n\n1. Add the endpoint to all server implementations\n2. Add a `BenchmarkSpec` entry in `cmd/bench/main.go`\n3. Update dashboard format if new fields are needed (`internal/dashboard/format.go`)\n\n## License\n\nApache 2.0\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fgoceleris%2Fbenchmarks","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fgoceleris%2Fbenchmarks","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fgoceleris%2Fbenchmarks/lists"}