{"id":50416948,"url":"https://github.com/viplismism/benchmark_suite","last_synced_at":"2026-05-31T06:30:39.600Z","repository":{"id":346981016,"uuid":"1172495804","full_name":"viplismism/benchmark_suite","owner":"viplismism","description":"Universal LLM evaluation framework — HumanEval, SWE-bench, GPQA, BigCodeBench, τ-Bench, Terminal-Bench via a single make command","archived":false,"fork":false,"pushed_at":"2026-03-26T07:51:25.000Z","size":109,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2026-03-27T02:42:56.770Z","etag":null,"topics":["benchmarks","evaluation","gpqa","humaneval","llm","swe-bench"],"latest_commit_sha":null,"homepage":null,"language":"Shell","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/viplismism.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2026-03-04T11:25:38.000Z","updated_at":"2026-03-26T07:51:28.000Z","dependencies_parsed_at":null,"dependency_job_id":null,"html_url":"https://github.com/viplismism/benchmark_suite","commit_stats":null,"previous_names":["viplismism/benchmark_suite"],"tags_count":null,"template":false,"template_full_name":null,"purl":"pkg:github/viplismism/benchmark_suite","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/viplismism%2Fbenchmark_suite","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/viplismism%2Fbenchmark_suite/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/viplismism%2Fbenchmark_suite/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/viplismism%2Fbenchmark_suite/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/viplismism","download_url":"https://codeload.github.com/viplismism/benchmark_suite/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/viplismism%2Fbenchmark_suite/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":33722156,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-26T15:22:16.424Z","status":"online","status_checked_at":"2026-05-31T02:00:06.040Z","response_time":95,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["benchmarks","evaluation","gpqa","humaneval","llm","swe-bench"],"created_at":"2026-05-31T06:30:39.285Z","updated_at":"2026-05-31T06:30:39.590Z","avatar_url":"https://github.com/viplismism.png","language":"Shell","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Benchmark Suite\n\nUniversal evaluation framework for LLM models. Standardized interface for running code generation, agentic, and reasoning benchmarks through a single `make` command.\n\n## Benchmarks\n\n| Benchmark | What it tests | Tasks | Requirements |\n|-----------|--------------|-------|-------------|\n| **HumanEval-Rust** | Rust code generation | 161 | Rust toolchain |\n| **τ-Bench** | Tool-calling agents | 50-115 | — |\n| **Terminal-Bench** | Terminal-based tasks | 89 | Docker |\n| **SWE-bench Verified** | Real GitHub bug fixing | 500 | Docker, ~120GB disk |\n| **BigCodeBench** | Multi-language code gen | 150-1140 | — |\n| **GPQA Diamond** | Graduate-level reasoning | 198 | HuggingFace token |\n| **CodeSearchNet** | Code retrieval (NDCG@10) | 573 | QMD, Node.js |\n\n## Quick Start\n\n```bash\n# 1. Clone and configure\ngit clone \u003crepo-url\u003e \u0026\u0026 cd benchmark_suite\ncp config.env.example config.env\ncp litellm_config.yaml.example litellm_config.yaml\n# Edit both files with your API keys and model configurations\n\n# 2. Install and start LiteLLM proxy\npython3 -m venv .venv \u0026\u0026 .venv/bin/pip install 'litellm[proxy]'\nmake litellm-start\n\n# 3. Run a single benchmark\nmake humaneval\nmake gpqa\nmake swe-bench\nmake codesearch    # Code retrieval (no LLM needed)\n\n# 4. Or run all benchmarks\nmake all\n\n# 5. View results\nmake results\n```\n\n## Example Output\n\n### Running a benchmark\n\n```\n═══════════════════════════════════════════════════════════════════════\n  GPQA Diamond Evaluation\n═══════════════════════════════════════════════════════════════════════\n  Model:    claude-haiku\n  Subset:   diamond (198 questions)\n  Results:  ./benchmark_results/gpqa/gpqa_20260304_152831_claude-haiku\n═══════════════════════════════════════════════════════════════════════\n\n→ Installing dependencies...\n→ Running GPQA evaluation...\nLoading GPQA diamond from HuggingFace...\nLoaded 198 questions\n\n[1/198] ✓ predicted=B correct=B\n[2/198] ✗ predicted=A correct=C\n[3/198] ✓ predicted=D correct=D\n...\n\n======================================================================\n  GPQA RESULTS\n======================================================================\n  Model:     claude-haiku\n  Subset:    diamond\n  Accuracy:  72/198 (36.36%)\n  Tokens:    284,190\n  Random baseline: 25.0%\n======================================================================\n```\n\n### HumanEval-Rust\n\n```\n═══════════════════════════════════════════════════════════════════════\n  HumanEval-Rust Evaluation\n═══════════════════════════════════════════════════════════════════════\n  Model:    claude-haiku\n  Tasks:    161 problems\n═══════════════════════════════════════════════════════════════════════\n\n→ Running evaluation...\nLoading HumanEval-Rust dataset from HuggingFace...\nLoaded 161 problems\n[1/161] HumanEval_0_has_close_elements ✓\n[2/161] HumanEval_1_separate_paren_groups ✓\n[3/161] HumanEval_2_truncate_number ✓\n...\n\n══════════════════════════════════════════════════════════════════════\n  HUMANEVAL-RUST RESULTS\n══════════════════════════════════════════════════════════════════════\n  Model:     claude-haiku\n  Tasks:     118/161 passed (73.3%)\n══════════════════════════════════════════════════════════════════════\n```\n\n### SWE-bench Verified\n\n```\n═══════════════════════════════════════════════════════════════════════\n  SWE-bench Verified Evaluation\n═══════════════════════════════════════════════════════════════════════\n  Model:        claude-haiku\n  Dataset:      princeton-nlp/SWE-bench_Verified\n  Agent:        simple\n  Max Workers:  4\n  Instances:    all (500)\n═══════════════════════════════════════════════════════════════════════\n\n→ Checking prerequisites...\n  ✓ Docker is running (socket: unix:///Users/dev/.orbstack/run/docker.sock)\n  ✓ ARM architecture detected — will use --namespace ''\n\n→ Phase 1: Inference (generating patches)...\n[1/500] astropy__astropy-12907\n  ✓ Patch generated (1284 chars)\n[2/500] django__django-11099\n  ✓ Patch generated (856 chars)\n...\n\n→ Phase 2: Evaluation (running tests in Docker)...\n...\n\n══════════════════════════════════════════════════════════════════════\n  SWE-BENCH VERIFIED RESULTS\n══════════════════════════════════════════════════════════════════════\n  Model:       claude-haiku\n  Instances:   500\n  Resolved:    23/500 (4.6%)\n══════════════════════════════════════════════════════════════════════\n```\n\n### Viewing results across runs\n\n```\n$ make results\n═══════════════════════════════════════════════════════════════════════\n  Benchmark Results\n═══════════════════════════════════════════════════════════════════════\n\n  Benchmark              Model                        Score            Date\n  ────────────────────── ──────────────────────────── ──────────────── ───────────────────\n  gpqa                   claude-haiku                 72/198 (36.36%) 2026-03-04T15:28:31\n  humaneval-rust         claude-haiku                 118/161 (73.3%) 2026-03-04T15:15:02\n  swe-bench-verified     claude-haiku                 23/500 (4.6%)   2026-03-04T15:38:52\n\n═══════════════════════════════════════════════════════════════════════\n```\n\n## Available Commands\n\n```bash\nmake help                    # Show all commands\n\n# LiteLLM Proxy\nmake litellm                 # Start proxy (foreground)\nmake litellm-start           # Start proxy (background)\nmake litellm-stop            # Stop proxy\nmake litellm-status          # Check status\n\n# Benchmarks\nmake humaneval               # Run HumanEval-Rust evaluation\nmake tau-bench               # Run τ-Bench evaluation\nmake terminal-bench          # Run Terminal-Bench evaluation\nmake terminal-bench-2        # Run Terminal-Bench 2.0 evaluation\nmake swe-bench               # Run SWE-bench Verified evaluation\nmake bigcodebench            # Run BigCodeBench evaluation\nmake gpqa                    # Run GPQA Diamond evaluation\nmake codesearch              # Run CodeSearchNet retrieval evaluation\n\n# Suite\nmake all                     # Run all benchmarks sequentially\nmake results                 # Show results from all runs\n\n# Utilities\nmake swe-bench-clean         # Remove SWE-bench Docker images\nmake clean                   # Clean up Docker containers\nmake docker-clean            # Remove ALL Docker resources\n```\n\n## Configuration\n\n### config.env\n\nAll benchmarks read from a single `config.env` file. See `config.env.example` for the full template.\n\n```bash\n# LiteLLM Proxy\nMODEL_ENDPOINT=\"http://localhost:8001\"\nLITELLM_PROXY_KEY=\"your-key\"\n\n# Each benchmark has its own MODEL variable\nHUMANEVAL_MODEL=\"your-model\"\nTAU_BENCH_AGENT_MODEL=\"your-model\"\nTERMINAL_BENCH_MODEL=\"your-model\"\nSWE_BENCH_MODEL=\"your-model\"\nBIGCODEBENCH_MODEL=\"your-model\"\nGPQA_MODEL=\"your-model\"\n```\n\n### CodeSearchNet (non-LLM benchmark)\n\nUnlike other benchmarks, CodeSearchNet evaluates a **local retrieval tool** (QMD), not an LLM:\n\n```bash\nCODESEARCH_QMD_DIR=\"/path/to/qmd\"     # QMD repo root\nCODESEARCH_LANGUAGE=\"all\"             # or a specific language\n```\n\n### litellm_config.yaml\n\nRoutes all benchmark API calls through a single proxy. Configure your models and backends:\n\n```yaml\nmodel_list:\n  - model_name: your-model-name\n    litellm_params:\n      model: openai/your-model       # or anthropic/claude-..., etc.\n      api_base: http://your-server/v1\n      api_key: your-api-key\n\ngeneral_settings:\n  master_key: sk-litellm-proxy-key-123\n```\n\n## Results\n\nAll results are saved in `benchmark_results/` with timestamped directories:\n\n```\nbenchmark_results/\n├── humaneval/\n│   └── humaneval_YYYYMMDD_HHMMSS_MODEL/\n│       ├── summary.json\n│       ├── results.json\n│       └── run.log\n├── tau_bench/\n│   └── tau_bench_YYYYMMDD_HHMMSS_MODEL/\n│       ├── evaluation_results.json\n│       └── run.log\n├── terminal_bench/\n│   └── terminal_bench_YYYYMMDD_HHMMSS_MODEL/\n│       ├── summary.json\n│       └── session_*/\n├── swe_bench/\n│   └── swe_bench_YYYYMMDD_HHMMSS_MODEL/\n│       ├── predictions.jsonl\n│       ├── summary.json\n│       └── run.log\n├── bigcodebench/\n│   └── bigcodebench_YYYYMMDD_HHMMSS_MODEL/\n│       └── summary.json\n└── gpqa/\n    └── gpqa_YYYYMMDD_HHMMSS_MODEL/\n        └── summary.json\n├── codesearch/\n│   └── codesearch_YYYYMMDD_HHMMSS/\n│       ├── summary.json\n│       ├── detailed_results.json\n│       └── run.log\n```\n\nUse `make results` to view a summary table across all runs.\n\n## Project Structure\n\n```\nbenchmark_suite/\n├── benchmarks/          # Benchmark scripts only\n│   ├── humaneval.sh\n│   ├── gpqa.sh\n│   ├── bigcodebench.sh\n│   ├── swe_bench.sh\n│   ├── tau_bench.sh\n│   ├── terminal_bench.sh\n│   └── terminal_bench_2.sh\n├── config.env.example   # Configuration template\n├── litellm_config.yaml.example\n├── Makefile             # All commands\n├── benchmark_results/   # Output (created on run)\n└── .cache/              # Venvs \u0026 cloned repos (created on run)\n```\n\n## Requirements\n\n- Python 3.10+ (HumanEval, τ-Bench, SWE-bench, BigCodeBench, GPQA)\n- Python 3.12+ (Terminal-Bench)\n- Rust toolchain (HumanEval — auto-installed if missing)\n- Docker (Terminal-Bench, SWE-bench)\n- LiteLLM (`pip install 'litellm[proxy]'`)\n- HuggingFace token (GPQA — gated dataset)\n- ~120GB+ free disk space (SWE-bench Docker images)\n\n## License\n\nMIT\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fviplismism%2Fbenchmark_suite","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fviplismism%2Fbenchmark_suite","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fviplismism%2Fbenchmark_suite/lists"}