{"id":51412896,"url":"https://github.com/kunyuan/allen-dynes-tc-benchmark","last_synced_at":"2026-07-04T16:01:50.868Z","repository":{"id":362855619,"uuid":"1261056073","full_name":"kunyuan/allen-dynes-tc-benchmark","owner":"kunyuan","description":"Allen-Dynes Tc agent benchmark (Harbor): L1 basic + L2 strong-coupling. Formula withheld; code submission graded on held-out cases. Derived from Paper2ARM WF-6.","archived":false,"fork":false,"pushed_at":"2026-06-06T09:00:57.000Z","size":33,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2026-06-06T09:15:44.790Z","etag":null,"topics":["agentic-ai","allen-dynes","benchmark","electron-phonon","harbor","llm-evaluation","physics","scientific-computing","superconductivity"],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/kunyuan.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2026-06-06T07:19:57.000Z","updated_at":"2026-06-06T09:01:00.000Z","dependencies_parsed_at":null,"dependency_job_id":null,"html_url":"https://github.com/kunyuan/allen-dynes-tc-benchmark","commit_stats":null,"previous_names":["kunyuan/allen-dynes-tc-benchmark"],"tags_count":null,"template":false,"template_full_name":null,"purl":"pkg:github/kunyuan/allen-dynes-tc-benchmark","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/kunyuan%2Fallen-dynes-tc-benchmark","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/kunyuan%2Fallen-dynes-tc-benchmark/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/kunyuan%2Fallen-dynes-tc-benchmark/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/kunyuan%2Fallen-dynes-tc-benchmark/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/kunyuan","download_url":"https://codeload.github.com/kunyuan/allen-dynes-tc-benchmark/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/kunyuan%2Fallen-dynes-tc-benchmark/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":35127443,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-26T15:22:16.424Z","status":"online","status_checked_at":"2026-07-04T02:00:05.987Z","response_time":113,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["agentic-ai","allen-dynes","benchmark","electron-phonon","harbor","llm-evaluation","physics","scientific-computing","superconductivity"],"created_at":"2026-07-04T16:01:50.125Z","updated_at":"2026-07-04T16:01:50.863Z","avatar_url":"https://github.com/kunyuan.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# allen-dynes-tc-benchmark — Allen-Dynes Tc Harbor benchmark\n\n[![oracle-selfcheck](https://github.com/kunyuan/allen-dynes-tc-benchmark/actions/workflows/ci.yml/badge.svg)](https://github.com/kunyuan/allen-dynes-tc-benchmark/actions/workflows/ci.yml)\n\nThree [Harbor](https://github.com/harbor-framework/harbor) tasks derived from\n**WF-6 (McMillan–Allen-Dynes Tc estimation), sub-family 6a** of the Paper2ARM\nsuperconductivity workflow cluster. The **formula is withheld** in all three —\nthe agent must recall the theory, not transcribe a given expression — and each\nlevel tests a distinct closed-form **subroutine / variant** of the Tc computation.\n\n| Task | Subroutine / variant tested | Inputs | Cases | Gold |\n|---|---|---|---|---|\n| **L0** `tc-allendynes-L0-mcmillan` | the **original McMillan (1968)** form (Θ_D/**1.45** prefactor, *not* Allen-Dynes ω_log/1.2) | λ, **Θ_D**, μ\\* | weak-moderate | McMillan output |\n| **L1** `tc-allendynes-L1-basic` | basic **Allen-Dynes (1975)** exponential (ω_log/**1.2** prefactor) | λ, ω_log, μ\\* | papers that used the basic form | basic Allen-Dynes output |\n| **L2** `tc-allendynes-L2-strongcoupling` | strong-coupling **f1** + shape **f2** corrections | λ, ω_log, **ω̄₂**, μ\\* | papers that used f1/f2 | full Allen-Dynes (f1·f2) output |\n\nOrdered by the physics lineage: McMillan (1968) → Allen-Dynes basic (1975) →\nAllen-Dynes + strong-coupling corrections.\n\n- **Formula withheld**: no `instruction.md` contains the expression; it names the\n  method + regime (L0: the *original McMillan* Debye-temperature form, explicitly\n  **not** Allen-Dynes; L2: that f1/f2 are required) and the agent must supply the\n  coefficients.\n- **Code submission + held-out test** (anti-cheat): each level ships public\n  development cases (with gold, in `environment/packet/`) and **concealed**\n  held-out cases (`tests/hidden/`, gold root-only). The agent submits a generic\n  `run_tc.py`; it is run on cases it never saw.\n- **Cross-formula discrimination**: at L0 the McMillan prefactor (Θ_D/1.45) is ~21%\n  below Allen-Dynes (Θ_D/1.2), so an agent who reaches for the Allen-Dynes form\n  **fails** L0; at L2 the f1·f2 factors raise Tc by ~7–27%, so an L1 basic-only\n  implementation **fails** L2. The three variants are genuinely separated.\n- **Gold = deterministic formula output** (NOT paper-reported experimental Tc).\n  L2's ω̄₂ = ⟨ω²⟩^(1/2) is an assigned representative second moment (papers rarely\n  report it); the task grades implementation of the f1/f2 subroutine, not ω̄₂.\n- **Pure closed-form** — no DFT/HPC/Julia; sub-second verifiers.\n\n## Data \u0026 provenance\n\nAll cases derive from **WF-6 (243 papers)** of the Paper2ARM superconductivity\ncluster. The method and numbers for **every paper were read directly from its LKM\nknowledge-graph nodes** (verbatim claim/conclusion text) by **parallel subagents**\n— not regex over a summary. That read distinguishes each paper's **computed** Tc\nfrom its **experimental** Tc, recovers the paper's **own μ\\***, and — for the 183\nmulti-condition papers — **splits each pressure / doping / SOC / phase into its own\nself-consistent data point** (a second subagent pass).\n\nA data point enters the benchmark only if the formula the paper used, **with the\npaper's own μ\\***, reproduces that point's **computed** Tc (within 15%). The funnel,\nlayer by layer:\n\n```\n243  WF-6 papers\n└─ 232  found in LKM                                  (11 not found)\n   → read per-paper; multi-condition papers split per condition\n341  self-consistent (λ, ω/Θ_D, μ*, Tc) data points  (deduped: one per paper × λ × freq)\n├─ 268  formula reproduces that point's COMPUTED Tc (≤15%)   →  BENCHMARK\n│        = L0 McMillan 13 + L1 basic 236 + L2 f1f2 19   (from 123 distinct papers)\n└─  65  formula does NOT reproduce it                        →  REPRODUCTION.md §B\n   (papers with no closed-form method / missing a param are not counted above)\n\nindependent axis — theory vs experiment:\n 85  papers report BOTH a computed and an experimental Tc\n ├─ 61  computed ≈ experiment\n └─ 24  computed ≠ experiment (\u003e30% apart)                    →  REPRODUCTION.md §A\n```\n\nBenchmark cases, by the formula the paper used (dev + hidden):\n\n| Level | Formula the paper used | Benchmark cases (dev + hidden) |\n|---|---|---|\n| **L0** | McMillan (Θ_D/1.45) | 13 (9 + 4) |\n| **L1** | basic Allen-Dynes (ω_log/1.2) | 236 (157 + 79) |\n| **L2** | Allen-Dynes + f1/f2 | 19 (13 + 6) |\n| **total** | | **268 cases** |\n\nInputs are abstract parameters only (no material identity), so Tc cannot be recalled\nfrom a known compound. L2's ω̄₂ = ⟨ω²⟩^(1/2) is the paper's value where reported, else\nan assigned representative second moment.\n\n**Two kinds of failure are catalogued — not discarded — in\n[`REPRODUCTION.md`](REPRODUCTION.md):** *(A)* **24 papers whose computed Tc is\nreproducible but disagrees with experiment** (the closed form is inadequate — very\nstrong coupling needs full Eliashberg — or pair-breaking suppresses the real Tc);\n*(B)* **65 per-condition data points whose computed Tc we could not reproduce** with\nthe reported method + μ\\* (non-standard/custom-fit coefficients, a full-Eliashberg\npaper, a Debye-vs-log frequency mismatch, or a residual mis-pairing). *Failing to\nreproduce a reported result is itself a finding.*\n\n### Per-case sources\n\nThe source `paper_id` for **every** case — all 268 across L0/L1/L2, with the\ncondition label, dev/hidden split and parameters — is recorded in\n[`SOURCES.md`](SOURCES.md). Each L0 case is a paper that used the original McMillan\nΘ_D/1.45 form and whose McMillan Tc reproduces its computed value; each L1/L2 case\nlikewise reproduces its paper's computed Tc with the matching Allen-Dynes form.\n\n## Layout (standard Harbor, per level)\n\n```\ntc-allendynes-L{0,1,2}-*/\n├── task.toml          # metadata + verifier/agent/build timeouts\n├── instruction.md     # agent-facing task (formula WITHHELD; I/O contract; scoring)\n├── environment/\n│   ├── Dockerfile     # python:3.12-slim + numpy\n│   └── packet/        # public dev data (params + gold)\n├── solution/          # oracle (run_tc.py + solve.sh) — not shown to agent\n└── tests/             # verifier — not shown to agent\n    ├── test.sh        #   run /app/run_tc.py as `nobody` on hidden params; reward.txt\n    ├── score.py       #   shape guard + per-material tolerance; result.json; exit 0 iff PASS\n    ├── hidden/params.csv\n    └── gold/gold.csv  #   root-only\n```\n\n## Run\n\n```bash\nharbor run --agent oracle --path tc-allendynes-L1-basic            # confirm solvable\nharbor run --agent claude-code --path tc-allendynes-L2-strongcoupling\n```\n\nPASS iff every concealed case is within `max(0.05 K, 1%·|Tc_gold|)`, exactly one\nprediction each. The full host-side self-check (every oracle on dev+hidden, plus\nthe L1-basic→L2 and Allen-Dynes-prefactor→L0 discrimination checks) runs in CI via\n`scripts/selfcheck.sh` and is also reproduced by the agent solutions in\n`experiments/`.\n\n## Harder variants (future)\n\nL4 — withhold even the *method name* (only the physical setup + α²F spectral data),\nrequiring derivation of the Allen-Dynes form from Eliashberg theory; or require the\nagent to compute ω_log / λ / ω̄₂ from a provided α²F(ω) spectrum (adds a\nmoment-integration subroutine).\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fkunyuan%2Fallen-dynes-tc-benchmark","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fkunyuan%2Fallen-dynes-tc-benchmark","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fkunyuan%2Fallen-dynes-tc-benchmark/lists"}