{"id":49360418,"url":"https://github.com/ambient-code/gh-api-benchmark","last_synced_at":"2026-04-27T16:02:16.485Z","repository":{"id":353654230,"uuid":"1220337869","full_name":"ambient-code/gh-api-benchmark","owner":"ambient-code","description":"GitHub REST API benchmark: measures latency cost of using GitHub as state coordination for agentic workflows","archived":false,"fork":false,"pushed_at":"2026-04-24T20:27:41.000Z","size":160,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2026-04-24T22:29:34.745Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"TypeScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/ambient-code.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2026-04-24T19:48:19.000Z","updated_at":"2026-04-24T20:27:43.000Z","dependencies_parsed_at":null,"dependency_job_id":null,"html_url":"https://github.com/ambient-code/gh-api-benchmark","commit_stats":null,"previous_names":["ambient-code/gh-api-benchmark"],"tags_count":null,"template":false,"template_full_name":null,"purl":"pkg:github/ambient-code/gh-api-benchmark","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ambient-code%2Fgh-api-benchmark","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ambient-code%2Fgh-api-benchmark/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ambient-code%2Fgh-api-benchmark/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ambient-code%2Fgh-api-benchmark/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/ambient-code","download_url":"https://codeload.github.com/ambient-code/gh-api-benchmark/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ambient-code%2Fgh-api-benchmark/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":32343571,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-04-26T23:26:28.701Z","status":"online","status_checked_at":"2026-04-27T02:00:06.769Z","response_time":128,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2026-04-27T16:02:13.885Z","updated_at":"2026-04-27T16:02:16.479Z","avatar_url":"https://github.com/ambient-code.png","language":"TypeScript","funding_links":[],"categories":[],"sub_categories":[],"readme":"# GitHub-as-State-Machine Benchmark\n\nMeasures the latency cost of using GitHub's REST API as a state coordination\nmechanism in the [Amber issue-handler workflow](https://ambient-code.ai).\n\n## Thesis\n\nUsing GitHub (issues, PRs, labels, checks) as the state machine for agentic\nworkflows introduces measurable overhead. This benchmark quantifies:\n\n1. **Per-step API latency** — how long each GitHub REST call takes\n2. **Polling tax** — time wasted waiting for GitHub to compute state (e.g. PR\n   mergeable)\n3. **Rate limit pressure** — how fast you burn through 5,000 req/hr under\n   concurrent agents\n4. **Concurrency degradation** — how latency changes as parallel agents increase\n5. **Event propagation delay** — the gap between \"state changed\" and \"state\n   visible via API\"\n\nThe goal is to prove that **timer-based polling should be replaced with\nevent-driven webhooks** and that GitHub's API is a bottleneck worth measuring.\n\n## Workflow Mapping\n\n```\nMermaid Step              → GitHub API Call                    → Type\n─────────────────────────────────────────────────────────────────────\nA: Issue + label          → POST /issues + POST /labels       → WRITE\nB: Workflow dispatch      → POST /dispatches                  → WRITE\nC: ambient-action         → (simulated delay)                 → COMPUTE\nD: ACP session            → (simulated delay)                 → COMPUTE\nE: Amber2 workflow        → (simulated delay)                 → COMPUTE\nF: Load context           → GET /repos + GET /issues          → READ\nG: Spec Kit               → (simulated delay)                 → COMPUTE\nH: Reproduce              → (simulated delay)                 → COMPUTE\nI: Implement fix          → (simulated delay)                 → COMPUTE\nJ: Lint/test/coverage     → POST + PATCH /check-runs          → WRITE\nK: Emit report            → POST /issues/comments             → WRITE\nL: Push branch            → POST /git/refs + PUT /contents    → WRITE\nM: Create PR              → POST /pulls                       → WRITE\nM: Poll mergeable         → GET /pulls (repeated)             → POLL ← event gap\nN: Review                 → POST /pulls/reviews               → WRITE\nO: Merge                  → PUT /pulls/merge                  → WRITE\nP: Post-merge learning    → POST /issues/comments             → WRITE\nQ: Close issue            → PATCH /issues                     → WRITE\nZ: Cleanup                → DELETE /git/refs                  → WRITE\n```\n\nEach WRITE call costs 5 points against the secondary rate limit (900 pts/min).\nA single cycle makes ~14 write calls = 70 points. At 900 pts/min, you hit the\nsecondary rate limit at ~12 concurrent cycles/minute.\n\n## Quick Start\n\n```bash\n# Install\nuv sync\n\n# Set up test repo (creates repo, labels, workflow)\nGITHUB_TOKEN=ghp_xxx GITHUB_ORG=your-test-org \\\n  uv run python setup_repo.py\n\n# Run benchmark (sequential then parallel ramp)\nGITHUB_TOKEN=ghp_xxx GITHUB_ORG=your-test-org GITHUB_REPO=gh-api-benchmark \\\n  uv run python benchmark.py\n\n# Analyze results\nuv run python analyze.py benchmark_results.db ./reports/\n```\n\n## Configuration\n\nEnvironment variables:\n\n| Variable            | Default              | Description                    |\n|---------------------|----------------------|--------------------------------|\n| `GITHUB_TOKEN`      | (required)           | PAT with repo+workflow+checks  |\n| `GITHUB_ORG`        | (required)           | Test org or username           |\n| `GITHUB_REPO`       | `gh-api-benchmark`   | Test repo name                 |\n| `SEQUENTIAL_CYCLES` | `5`                  | Baseline cycles (no concurrency) |\n| `PARALLEL_CYCLES`   | `20`                 | Total parallel cycles          |\n| `MAX_CONCURRENCY`   | `5`                  | Max simultaneous cycles        |\n\n## Required PAT Scopes\n\n- `repo` — full control of private repos\n- `workflow` — update GitHub Action workflows\n- `write:checks` — create and update check runs\n\n## Output\n\nSQLite database (`benchmark_results.db`) with three tables:\n\n- **`step_results`** — every individual API call or simulated step\n- **`cycle_summary`** — per-cycle aggregates (wall time, API time, sim time)\n- **`rate_limit_snapshots`** — periodic rate limit state\n\nAnalysis script produces:\n- `benchmark_report.txt` — human-readable summary\n- `step_latency_stats.csv` — per-step p50/p90/p99\n- `rate_limit_timeline.csv` — rate limit consumption over time\n- `all_step_results.csv` — raw data for external tools\n\n## Key Metrics to Watch\n\n1. **Poll Tax %** — what fraction of cycle time is spent polling for state\n2. **GitHub Tax %** — what fraction of non-compute time is API calls\n3. **P99 latency per step** — tail latency tells the real story\n4. **Rate limit burn rate** — projected req/hr extrapolated from the run\n5. **Concurrency knee** — at what parallelism level does latency spike\n\n## Dashboard\n\nA Next.js app for launching runs, monitoring progress, and comparing results.\n\n```bash\ncd dashboard\nnpm install\n\n# Required: pass the same GitHub credentials\nGITHUB_TOKEN=ghp_xxx GITHUB_ORG=your-test-org GITHUB_REPO=gh-api-benchmark \\\n  npx next dev -p 3001\n```\n\nSee `dashboard/.env.example` for required variables. The dashboard reads from\nthe same `benchmark_results.db` in the project root.\n\nFeatures:\n- **Launch** benchmark runs with preset or custom configurations\n- **Live monitoring** — polls every 3s while a run is active\n- **Run detail** — per-step latency charts, percentile distributions, rate limit timeline\n- **Compare** — side-by-side metrics across multiple runs\n\n## Architecture\n\n```\nbenchmark.py          — orchestrator + instrumented GitHub client\n├── Phase 1           — sequential baseline (N cycles, 1 at a time)\n├── Phase 2           — parallel ramp (1→MAX_CONCURRENCY workers)\n└── SQLite writer     — WAL mode, commit-per-cycle for crash safety\n\nanalyze.py            — reads SQLite, produces stats + CSVs\nsetup_repo.py         — one-time test repo initialization\n\ndashboard/            — Next.js app for visualization and run management\n├── src/app/api/      — API routes (runs CRUD, benchmark launcher, compare)\n└── src/app/page.tsx  — single-page app with runs list, detail, and compare views\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fambient-code%2Fgh-api-benchmark","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fambient-code%2Fgh-api-benchmark","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fambient-code%2Fgh-api-benchmark/lists"}