https://github.com/kunyuan/allen-dynes-tc-benchmark
Allen-Dynes Tc agent benchmark (Harbor): L1 basic + L2 strong-coupling. Formula withheld; code submission graded on held-out cases. Derived from Paper2ARM WF-6.
https://github.com/kunyuan/allen-dynes-tc-benchmark
agentic-ai allen-dynes benchmark electron-phonon harbor llm-evaluation physics scientific-computing superconductivity
Last synced: 16 days ago
JSON representation
Allen-Dynes Tc agent benchmark (Harbor): L1 basic + L2 strong-coupling. Formula withheld; code submission graded on held-out cases. Derived from Paper2ARM WF-6.
- Host: GitHub
- URL: https://github.com/kunyuan/allen-dynes-tc-benchmark
- Owner: kunyuan
- License: mit
- Created: 2026-06-06T07:19:57.000Z (about 1 month ago)
- Default Branch: main
- Last Pushed: 2026-06-06T09:00:57.000Z (about 1 month ago)
- Last Synced: 2026-06-06T09:15:44.790Z (about 1 month ago)
- Topics: agentic-ai, allen-dynes, benchmark, electron-phonon, harbor, llm-evaluation, physics, scientific-computing, superconductivity
- Language: Python
- Size: 32.2 KB
- Stars: 0
- Watchers: 0
- Forks: 0
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
- License: LICENSE
Awesome Lists containing this project
README
# allen-dynes-tc-benchmark — Allen-Dynes Tc Harbor benchmark
[](https://github.com/kunyuan/allen-dynes-tc-benchmark/actions/workflows/ci.yml)
Three [Harbor](https://github.com/harbor-framework/harbor) tasks derived from
**WF-6 (McMillan–Allen-Dynes Tc estimation), sub-family 6a** of the Paper2ARM
superconductivity workflow cluster. The **formula is withheld** in all three —
the agent must recall the theory, not transcribe a given expression — and each
level tests a distinct closed-form **subroutine / variant** of the Tc computation.
| Task | Subroutine / variant tested | Inputs | Cases | Gold |
|---|---|---|---|---|
| **L0** `tc-allendynes-L0-mcmillan` | the **original McMillan (1968)** form (Θ_D/**1.45** prefactor, *not* Allen-Dynes ω_log/1.2) | λ, **Θ_D**, μ\* | weak-moderate | McMillan output |
| **L1** `tc-allendynes-L1-basic` | basic **Allen-Dynes (1975)** exponential (ω_log/**1.2** prefactor) | λ, ω_log, μ\* | papers that used the basic form | basic Allen-Dynes output |
| **L2** `tc-allendynes-L2-strongcoupling` | strong-coupling **f1** + shape **f2** corrections | λ, ω_log, **ω̄₂**, μ\* | papers that used f1/f2 | full Allen-Dynes (f1·f2) output |
Ordered by the physics lineage: McMillan (1968) → Allen-Dynes basic (1975) →
Allen-Dynes + strong-coupling corrections.
- **Formula withheld**: no `instruction.md` contains the expression; it names the
method + regime (L0: the *original McMillan* Debye-temperature form, explicitly
**not** Allen-Dynes; L2: that f1/f2 are required) and the agent must supply the
coefficients.
- **Code submission + held-out test** (anti-cheat): each level ships public
development cases (with gold, in `environment/packet/`) and **concealed**
held-out cases (`tests/hidden/`, gold root-only). The agent submits a generic
`run_tc.py`; it is run on cases it never saw.
- **Cross-formula discrimination**: at L0 the McMillan prefactor (Θ_D/1.45) is ~21%
below Allen-Dynes (Θ_D/1.2), so an agent who reaches for the Allen-Dynes form
**fails** L0; at L2 the f1·f2 factors raise Tc by ~7–27%, so an L1 basic-only
implementation **fails** L2. The three variants are genuinely separated.
- **Gold = deterministic formula output** (NOT paper-reported experimental Tc).
L2's ω̄₂ = ⟨ω²⟩^(1/2) is an assigned representative second moment (papers rarely
report it); the task grades implementation of the f1/f2 subroutine, not ω̄₂.
- **Pure closed-form** — no DFT/HPC/Julia; sub-second verifiers.
## Data & provenance
All cases derive from **WF-6 (243 papers)** of the Paper2ARM superconductivity
cluster. The method and numbers for **every paper were read directly from its LKM
knowledge-graph nodes** (verbatim claim/conclusion text) by **parallel subagents**
— not regex over a summary. That read distinguishes each paper's **computed** Tc
from its **experimental** Tc, recovers the paper's **own μ\***, and — for the 183
multi-condition papers — **splits each pressure / doping / SOC / phase into its own
self-consistent data point** (a second subagent pass).
A data point enters the benchmark only if the formula the paper used, **with the
paper's own μ\***, reproduces that point's **computed** Tc (within 15%). The funnel,
layer by layer:
```
243 WF-6 papers
└─ 232 found in LKM (11 not found)
→ read per-paper; multi-condition papers split per condition
341 self-consistent (λ, ω/Θ_D, μ*, Tc) data points (deduped: one per paper × λ × freq)
├─ 268 formula reproduces that point's COMPUTED Tc (≤15%) → BENCHMARK
│ = L0 McMillan 13 + L1 basic 236 + L2 f1f2 19 (from 123 distinct papers)
└─ 65 formula does NOT reproduce it → REPRODUCTION.md §B
(papers with no closed-form method / missing a param are not counted above)
independent axis — theory vs experiment:
85 papers report BOTH a computed and an experimental Tc
├─ 61 computed ≈ experiment
└─ 24 computed ≠ experiment (>30% apart) → REPRODUCTION.md §A
```
Benchmark cases, by the formula the paper used (dev + hidden):
| Level | Formula the paper used | Benchmark cases (dev + hidden) |
|---|---|---|
| **L0** | McMillan (Θ_D/1.45) | 13 (9 + 4) |
| **L1** | basic Allen-Dynes (ω_log/1.2) | 236 (157 + 79) |
| **L2** | Allen-Dynes + f1/f2 | 19 (13 + 6) |
| **total** | | **268 cases** |
Inputs are abstract parameters only (no material identity), so Tc cannot be recalled
from a known compound. L2's ω̄₂ = ⟨ω²⟩^(1/2) is the paper's value where reported, else
an assigned representative second moment.
**Two kinds of failure are catalogued — not discarded — in
[`REPRODUCTION.md`](REPRODUCTION.md):** *(A)* **24 papers whose computed Tc is
reproducible but disagrees with experiment** (the closed form is inadequate — very
strong coupling needs full Eliashberg — or pair-breaking suppresses the real Tc);
*(B)* **65 per-condition data points whose computed Tc we could not reproduce** with
the reported method + μ\* (non-standard/custom-fit coefficients, a full-Eliashberg
paper, a Debye-vs-log frequency mismatch, or a residual mis-pairing). *Failing to
reproduce a reported result is itself a finding.*
### Per-case sources
The source `paper_id` for **every** case — all 268 across L0/L1/L2, with the
condition label, dev/hidden split and parameters — is recorded in
[`SOURCES.md`](SOURCES.md). Each L0 case is a paper that used the original McMillan
Θ_D/1.45 form and whose McMillan Tc reproduces its computed value; each L1/L2 case
likewise reproduces its paper's computed Tc with the matching Allen-Dynes form.
## Layout (standard Harbor, per level)
```
tc-allendynes-L{0,1,2}-*/
├── task.toml # metadata + verifier/agent/build timeouts
├── instruction.md # agent-facing task (formula WITHHELD; I/O contract; scoring)
├── environment/
│ ├── Dockerfile # python:3.12-slim + numpy
│ └── packet/ # public dev data (params + gold)
├── solution/ # oracle (run_tc.py + solve.sh) — not shown to agent
└── tests/ # verifier — not shown to agent
├── test.sh # run /app/run_tc.py as `nobody` on hidden params; reward.txt
├── score.py # shape guard + per-material tolerance; result.json; exit 0 iff PASS
├── hidden/params.csv
└── gold/gold.csv # root-only
```
## Run
```bash
harbor run --agent oracle --path tc-allendynes-L1-basic # confirm solvable
harbor run --agent claude-code --path tc-allendynes-L2-strongcoupling
```
PASS iff every concealed case is within `max(0.05 K, 1%·|Tc_gold|)`, exactly one
prediction each. The full host-side self-check (every oracle on dev+hidden, plus
the L1-basic→L2 and Allen-Dynes-prefactor→L0 discrimination checks) runs in CI via
`scripts/selfcheck.sh` and is also reproduced by the agent solutions in
`experiments/`.
## Harder variants (future)
L4 — withhold even the *method name* (only the physical setup + α²F spectral data),
requiring derivation of the Allen-Dynes form from Eliashberg theory; or require the
agent to compute ω_log / λ / ω̄₂ from a provided α²F(ω) spectrum (adds a
moment-integration subroutine).