{"id":50937918,"url":"https://github.com/wallarm/gateway-benchmarks","last_synced_at":"2026-06-17T11:03:19.947Z","repository":{"id":353056343,"uuid":"1217761390","full_name":"wallarm/gateway-benchmarks","owner":"wallarm","description":null,"archived":false,"fork":false,"pushed_at":"2026-05-19T14:08:04.000Z","size":1325,"stargazers_count":1,"open_issues_count":1,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2026-05-19T17:08:20.323Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/wallarm.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":"SECURITY.md","support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2026-04-22T07:35:12.000Z","updated_at":"2026-05-19T14:17:43.000Z","dependencies_parsed_at":null,"dependency_job_id":null,"html_url":"https://github.com/wallarm/gateway-benchmarks","commit_stats":null,"previous_names":["wallarm/gateway-benchmarks"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/wallarm/gateway-benchmarks","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/wallarm%2Fgateway-benchmarks","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/wallarm%2Fgateway-benchmarks/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/wallarm%2Fgateway-benchmarks/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/wallarm%2Fgateway-benchmarks/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/wallarm","download_url":"https://codeload.github.com/wallarm/gateway-benchmarks/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/wallarm%2Fgateway-benchmarks/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":34445186,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-26T15:22:16.424Z","status":"online","status_checked_at":"2026-06-17T02:00:05.408Z","response_time":127,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2026-06-17T11:03:09.264Z","updated_at":"2026-06-17T11:03:19.941Z","avatar_url":"https://github.com/wallarm.png","language":"Go","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Gateway Benchmarks\n\n\u003e Reproducible, vendor-neutral performance benchmarks for production API gateways under a **policy × protocol × load** matrix.\n\n[![License: Apache 2.0](https://img.shields.io/badge/License-Apache_2.0-blue.svg)](./LICENSE)\n\n\u003c!-- v0.1.0 ANNOUNCEMENT — replace once the canonical AWS run is captured. Source: docs/release-notes/v0.1.0.md § Announcement snippet --\u003e\n\n---\n\n## TL;DR\n\nWe compare seven API gateways under **identical conditions** and let any independent reviewer:\n\n1. Re-run the benchmark locally or on AWS with a single command.\n2. Obtain a **byte-for-byte equivalent ranking** (within tolerance — see [docs/REPRODUCIBILITY.md](./docs/REPRODUCIBILITY.md)).\n3. Verify that all gateways treat the same request the same way (**parity attestation**).\n4. Inspect the full run manifest: image digests, git SHA, RNG seed, host info.\n\n## Neutrality Disclaimer\n\nThis project is developed and maintained by **Wallarm, Inc.** — the author of one of the gateways under test. To neutralise the conflict of interest, we follow strict rules:\n\n- All gateway configs, k6 scenarios, and infrastructure are **open and frozen at report release** (the git SHA is pinned in `manifest.json`).\n- **Parity attestation** runs before every cell: the same request, the same JWT seed, the same rate-limit window — gateways either behave identically, or the cell is marked as a `deviation` and excluded from the aggregate.\n- Reasonable external tuning of a competing gateway is accepted as a PR — see [`CONTRIBUTING.md § Gateway-tuning PRs`](./CONTRIBUTING.md#gateway-tuning-prs) and the [PR template](./.github/PULL_REQUEST_TEMPLATE.md).\n- All deviations are documented in [docs/GATEWAYS.md](./docs/GATEWAYS.md) with a reason and an upstream reference.\n\n## Gateways Under Test\n\n| Gateway | Language | Role |\n|---------|----------|------|\n| [Wallarm API Gateway](https://github.com/wallarm/wallarm-api-gateway) | Rust | subject under test |\n| NGINX                   | C                  | baseline |\n| Envoy                   | C++                | baseline |\n| HAProxy *(candidate)*   | C                  | baseline |\n| Kong                    | Lua/OpenResty      | baseline |\n| Apache APISIX           | Lua/OpenResty      | baseline |\n| Traefik                 | Go                 | baseline |\n| Tyk                     | Go                 | baseline |\n\n**(11 ranking policy profiles × 4 load profiles) + (2 HTTPS scenarios × 4 load profiles) = 52 cells per gateway × 7 gateways = 364 cells per run** — plus the supplemental `p03-jwks-rs256-basic` capability scenario (parity-only, off-grid). See [TASK.md §7](./TASK.md).\n\n## Quick Start — Local mode\n\n\u003e Requirements: Linux/macOS host, Docker ≥ 24, 8+ physical cores, 16 GB RAM, `make`, `go ≥ 1.23`.\n\nTwo independent scenarios. Pick one — don't mix them. The Makefile has a\npreflight check that refuses to boot a second stack while another one is up\nand tells you exactly which command clears it.\n\n**Run the full matrix** (the default workflow):\n\n```bash\ngit clone https://github.com/wallarm/gateway-benchmarks\ncd gateway-benchmarks\n\nmake prereqs-check          # verify the environment\nmake perf-local-run         # parity → load → aggregate → manifest → report\n```\n\nThe orchestrator brings per-cell stacks up and down by itself — no separate\n`perf-local-up` is needed. Result: `reports/\u003crun-id\u003e/report.html`\n(`bench run` calls `bench report` automatically; `make bench-report\nBENCH_RUN_ID=\u003crun-id\u003e` and `make bench-report\nBENCH_REPORT_COMBINE=run-a,run-b` are also available).\n\n**Long-running smoke stack** (only when you want to poke a live gateway\nby hand — parity, curl, logs):\n\n```bash\nmake perf-local-up          # bring loadgen + gateway + backend up in separate namespaces\nmake perf-local-parity      # parity-check against localhost:9080\nmake perf-local-cycle-smoke # HTTP + HTTPS round-trip through the stack\nmake perf-local-down        # tear down (also cleans up any orphan gwb-\u003cgw\u003e* per-cell stacks)\n```\n\n## Quick Start — AWS mode\n\n\u003e Requirements: AWS credentials, `tofu` ≥ 1.7 (or `terraform` ≥ 1.6), ~$15 per full run.\n\n```bash\ncd infra/aws\ncp terraform.tfvars.example terraform.tfvars    # set your CIDR and region\ntofu init \u0026\u0026 tofu apply -auto-approve\n\ncd ../..\nmake perf-aws-deploy        # provision the stack on all 3 EC2 hosts\nmake perf-aws-run           # run the matrix (same orchestrator)\nmake perf-aws-report        # render reports/\u003crun-id\u003e/report.html (bench report)\nmake perf-aws-down          # tear down EC2 (edits tfvars and runs apply)\n```\n\n## Repository Layout\n\n```\n.\n├── TASK.md           # PRD — what we measure and why\n├── Makefile          # single entry point\n├── backend/          # forked go-httpbin — a predictable upstream\n├── gateways/         # per-gateway configs × policy profile\n├── k6/               # load profiles and scenarios\n├── orchestrator/     # Go binary driving the run\n├── infra/\n│   ├── local/        # docker-compose + resource pins\n│   └── aws/          # Terraform (3 EC2 cluster PG)\n├── scripts/          # prereqs, parity, deploy, fetch\n├── reports/          # output runs (local-only, never tracked — see docs/REPORT.md)\n└── docs/             # ARCHITECTURE / POLICIES / LOAD-PROFILES / GATEWAYS / REPRODUCIBILITY / REPORT\n```\n\n## Documentation\n\n- [TASK.md](./TASK.md) — PRD (mandatory properties of the benchmark)\n- [CHANGELOG.md](./CHANGELOG.md) — versioned release notes\n- [CONTRIBUTING.md](./CONTRIBUTING.md) — how to submit tuning PRs, what we review\n- [SECURITY.md](./SECURITY.md) — security policy + what is and isn't a secret in this tree\n- [docs/ARCHITECTURE.md](./docs/ARCHITECTURE.md) — local/AWS topology and network path\n- [docs/POLICIES.md](./docs/POLICIES.md) — 11 ranking + 1 supplemental policy profiles, parity requirements\n- [docs/LOAD-PROFILES.md](./docs/LOAD-PROFILES.md) — 4 load profiles\n- [docs/GATEWAYS.md](./docs/GATEWAYS.md) — versions, digests, deviations\n- [docs/REPORT.md](./docs/REPORT.md) — how to read the HTML report\n- [docs/REPRODUCIBILITY.md](./docs/REPRODUCIBILITY.md) — manifest, seeds, tolerance, `bench compare-runs` gate\n- [docs/CANONICAL-RUN-HANDOFF.md](./docs/CANONICAL-RUN-HANDOFF.md) — executable playbook for the AWS canonical sweep\n- [docs/RELEASE.md](./docs/RELEASE.md) — maintainer release process\n- [orchestrator/README.md](./orchestrator/README.md) — `bench` Go binary (Phases 6 + 7 + 8)\n\n## License\n\nApache 2.0 — see [LICENSE](./LICENSE).\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fwallarm%2Fgateway-benchmarks","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fwallarm%2Fgateway-benchmarks","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fwallarm%2Fgateway-benchmarks/lists"}