{"id":48789247,"url":"https://github.com/chenhaodev/medqa-auto-eval-skills","last_synced_at":"2026-04-13T19:02:35.611Z","repository":{"id":350693399,"uuid":"1207726119","full_name":"chenhaodev/medqa-auto-eval-skills","owner":"chenhaodev","description":null,"archived":false,"fork":false,"pushed_at":"2026-04-11T16:16:12.000Z","size":1611,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":0,"default_branch":"master","last_synced_at":"2026-04-11T17:11:52.212Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/chenhaodev.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2026-04-11T10:11:41.000Z","updated_at":"2026-04-11T16:16:15.000Z","dependencies_parsed_at":null,"dependency_job_id":null,"html_url":"https://github.com/chenhaodev/medqa-auto-eval-skills","commit_stats":null,"previous_names":["chenhaodev/medqa-auto-eval-skills"],"tags_count":null,"template":false,"template_full_name":null,"purl":"pkg:github/chenhaodev/medqa-auto-eval-skills","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/chenhaodev%2Fmedqa-auto-eval-skills","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/chenhaodev%2Fmedqa-auto-eval-skills/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/chenhaodev%2Fmedqa-auto-eval-skills/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/chenhaodev%2Fmedqa-auto-eval-skills/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/chenhaodev","download_url":"https://codeload.github.com/chenhaodev/medqa-auto-eval-skills/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/chenhaodev%2Fmedqa-auto-eval-skills/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":31766482,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-04-13T15:25:13.801Z","status":"ssl_error","status_checked_at":"2026-04-13T15:25:09.162Z","response_time":93,"last_error":"SSL_read: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2026-04-13T19:02:34.614Z","updated_at":"2026-04-13T19:02:35.602Z","avatar_url":"https://github.com/chenhaodev.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# medbench-eval\n\nUse an **LLM judge** to score answers on **MedBench-Agent-95** (390 items, **13 task types**). Each criterion is Likert **1–5**; scores normalize to **0–100**.\n\n| You are… | Start here |\n|----------|------------|\n| **A person** using Claude / Cursor with the skill | Read **[`SKILL.md`](SKILL.md)** — contract for `/medbench-eval`. This README covers **repo + CLI**. |\n| **An agent** (Cursor, CI, etc.) | **[Automation checklist](#automation-checklist-for-agents)** below. **Claude Code** also loads **[`CLAUDE.md`](CLAUDE.md)**. |\n| **A maintainer** changing rubrics, capabilities, or dependencies | **[Maintainers](#maintainers)**, [`references/PROTOCOL.md`](references/PROTOCOL.md), [`references/README.md`](references/README.md). |\n\n---\n\n## What you need\n\n1. **Python 3.11+**\n2. **Repository root** as the working directory (the folder that contains `pyproject.toml`).\n3. **API key** for the default judge: set **`ANTHROPIC_API_KEY`** (model `claude-haiku-4-5`). Other backends need extra env vars — see [`judge/llm_client.py`](judge/llm_client.py).\n4. A **`.env`** file in the repo root is loaded automatically. **Do not commit `.env`.**\n\n---\n\n## Install dependencies\n\n**Recommended ([uv](https://docs.astral.sh/uv/))** — reproducible installs from **`uv.lock`**:\n\n```bash\ncd /path/to/repo   # must contain pyproject.toml\nuv sync            # runtime only; creates .venv/\n\n# include pytest (for tests/README “smoke”)\nuv sync --group dev\n```\n\n**Using pip only** (no dev tools):\n\n```bash\npip install -r requirements.txt\n```\n\n`requirements.txt` is **exported** from the lockfile (`uv export --format requirements-txt -o requirements.txt`). After you edit **`pyproject.toml`** dependencies, run **`uv lock`**, then re-export if pip users need an updated file.\n\n---\n\n## Automation checklist (for agents)\n\nExecute **in order**. Assume the shell’s cwd is the repository root.\n\n| Step | Action | Command or file |\n|------|--------|-------------------|\n| 1 | Obtain the tree | `git clone \u003crepo-url\u003e` then `cd` into the root, **or** unpack a tarball and `cd` there. |\n| 2 | Install **uv** if missing | `curl -LsSf https://astral.sh/uv/install.sh \\| sh` — then ensure `uv` is on `PATH` (often `~/.local/bin`). |\n| 3 | Install deps + dev | `uv sync --group dev` |\n| 4 | Configure secrets | Create `.env` with at least `ANTHROPIC_API_KEY=...` |\n| 5 | Verify (no paid API calls in these checks) | `uv run --group dev pytest tests/ -q` then `python -m scripts.validate --help`, `python eval.py tasks`, and `python eval.py generate --help` |\n| 6 | Run real judging | Needs a valid key — see **[Run evaluations](#run-evaluations)** |\n\nFor conversational scoring conventions (wizard flow, rubric names, output template), read **`SKILL.md`**. This README does not duplicate that protocol.\n\n---\n\n## Run evaluations\n\n**Always run commands from the repository root** so paths like `references/medbench-agent-95` resolve correctly (`judge/refs.py`).\n\n### 1. See what the CLI can do\n\n```bash\npython eval.py --help\npython eval.py tasks\n```\n\n### 2. Batch (typical automation)\n\n```bash\npython eval.py batch \\\n  --capability reasoning \\\n  --dut YOUR_MODEL_NAME \\\n  --responses-dir path/to/responses/ \\\n  --samples 5 \\\n  --output results/run-001/\n```\n\n- **`--benchmark`** defaults to the shipped gold data: `references/medbench-agent-95/`.\n- **Outputs** under `--output`: `summary.json`, `details.jsonl`, `report.md`.\n\n### 2b. Generate answers (DeepSeek, etc.) then batch-judge\n\nUse an **answer model** (default `deepseek-chat`; set `DEEPSEEK_API_KEY` in `.env`) to answer gold questions. By default, repo **`SKILL.md`** is injected as **system** context (use `--no-skill` to disable). Sample selection uses the **same shared RNG across tasks** as **`eval.py batch`** when you use the same **`--samples`**, **`--seed`**, and the same task list (e.g. same `--task` or `--capability`).\n\n```bash\npython eval.py generate -o generated/dut/ --task MedCOT --samples 3 --seed 42 --answer-model deepseek-chat\npython eval.py batch --task MedCOT --dut deepseek-chat \\\n  --responses-file generated/dut/MedCOT.jsonl --samples 3 --seed 42 --output results/ds-001/\n```\n\nJudge model is still **`--model`** on `batch` (default Haiku); configure `ANTHROPIC_API_KEY` unless you pass `--model minimax-m2.5` / `deepseek-chat`.\n\n### 3. Single item (one question + one response)\n\n```bash\npython eval.py single \\\n  --task MedCOT \\\n  --dut YOUR_MODEL_NAME \\\n  --question \"...\" \\\n  --response \"...\" \\\n  [--gold-answer \"...\"]\n```\n\n### 4. Interactive wizard (human)\n\n```bash\npython eval.py\n```\n\nWith **no arguments**, this starts prompts on stdin — **unsuitable for unattended agents**.\n\n### Options you will need from docs\n\n| Topic | Where to read |\n|-------|----------------|\n| Capability keys (`reasoning`, `long_context`, …) and task lists | `python eval.py tasks` and [`SKILL.md`](SKILL.md) |\n| DUT file layouts (`--responses-dir`, `--responses-file`) | [`references/README.md`](references/README.md) |\n| **`--calibrate-n`** / **`--calibrate-mode`** (few-shot anchors, BM25, etc.) | `--help` on `eval.py batch` and [`judge/README.md`](judge/README.md) |\n\n---\n\n## Judge alignment (validate)\n\nOptional sanity check that the judge ranks gold above weak tiers (uses the API — not “offline”):\n\n```bash\npython -m scripts.validate --task MedCOT --samples 3\n# or\npython -m scripts.validate --all-tasks --samples 5\n```\n\nSame entry point: `python scripts/validate.py …`.\n\n---\n\n## Project layout\n\n```\npyproject.toml          # project metadata + runtime deps\nuv.lock                 # locked versions (commit this)\nrequirements.txt        # pip export of runtime deps (commit this)\ntests/                  # pytest smoke (CLI + gold files + BM25 index)\neval.py                 # CLI: wizard | batch | single | tasks\nscripts/\n  validate.py           # alignment CLI (prefer: python -m scripts.validate)\njudge/                  # refs | llm_client | scoring | runner — see judge/README.md\nreferences/\n  PROTOCOL.md           # what is “data/spec” vs executable code\n  capabilities.json     # capability groups (wizard / --capability); edit for task lists\n  rubrics.yaml          # task rubrics (source); edit criteria here\n  rubrics.md            # generated from rubrics.yaml; do not edit by hand\n  medbench-agent-95/    # gold JSONL per task\nSKILL.md                # skill /medbench-eval behavior\nCLAUDE.md               # Claude Code agent notes (this file is optional for humans)\n```\n\n---\n\n## More documentation\n\n| File | Contents |\n|------|----------|\n| [`references/README.md`](references/README.md) | Gold file format, rubric regeneration, skill-oriented notes |\n| [`references/PROTOCOL.md`](references/PROTOCOL.md) | Protocol vs Python in this repo; exact batch vs wizard sampling |\n| [`judge/DESIGN.md`](judge/DESIGN.md) | Plain-language overview of each `judge/` module (refs, llm_client, scoring, runner) |\n| [`judge/README.md`](judge/README.md) | Module map and implementation notes |\n\n---\n\n## Maintainers\n\n- Commit **`pyproject.toml`**, **`uv.lock`**, and **`requirements.txt`** together when dependencies change.\n- After editing the **`[project]` `dependencies`** list in `pyproject.toml`: run `uv lock`, then optionally `uv export …` for pip users.\n- Regenerate **`references/rubrics.md`** after changing **`references/rubrics.yaml`**: `python -m judge.refs`.\n- Edit **`references/capabilities.json`** to change capability → task grouping (no Python edit required for that).\n- **OpenCode skill copy:** `references/` may be hardlinked to your local skill install; **`SKILL.md` is not.** After editing repo **`SKILL.md`**, sync the installed skill, e.g. `cp SKILL.md ~/.config/opencode/skills/medbench-eval/SKILL.md` (adjust path if your OpenCode config dir differs).\n\n---\n\n## License\n\nMIT\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fchenhaodev%2Fmedqa-auto-eval-skills","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fchenhaodev%2Fmedqa-auto-eval-skills","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fchenhaodev%2Fmedqa-auto-eval-skills/lists"}