{"id":49504289,"url":"https://github.com/leanprover/lean-eval","last_synced_at":"2026-05-01T14:01:18.483Z","repository":{"id":354777950,"uuid":"1206705574","full_name":"leanprover/lean-eval","owner":"leanprover","description":"Comparator-based Lean formal mathematics eval","archived":false,"fork":false,"pushed_at":"2026-04-30T06:57:27.000Z","size":298,"stargazers_count":1,"open_issues_count":3,"forks_count":1,"subscribers_count":0,"default_branch":"main","last_synced_at":"2026-04-30T07:14:28.295Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"https://leanprover.github.io/lean-eval-leaderboard/","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/leanprover.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":"AGENTS.md","dco":null,"cla":null}},"created_at":"2026-04-10T07:11:10.000Z","updated_at":"2026-04-30T06:57:30.000Z","dependencies_parsed_at":null,"dependency_job_id":null,"html_url":"https://github.com/leanprover/lean-eval","commit_stats":null,"previous_names":["kim-em/lean-eval","leanprover/lean-eval"],"tags_count":null,"template":false,"template_full_name":null,"purl":"pkg:github/leanprover/lean-eval","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/leanprover%2Flean-eval","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/leanprover%2Flean-eval/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/leanprover%2Flean-eval/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/leanprover%2Flean-eval/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/leanprover","download_url":"https://codeload.github.com/leanprover/lean-eval/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/leanprover%2Flean-eval/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":32499691,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-04-30T13:12:12.517Z","status":"online","status_checked_at":"2026-05-01T02:00:05.856Z","response_time":64,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2026-05-01T14:01:15.682Z","updated_at":"2026-05-01T14:01:18.339Z","avatar_url":"https://github.com/leanprover.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Lean Eval\n\n**[View the leaderboard →](https://lean-lang.org/eval/)**\n\nThis repository is a comparator-based Lean benchmark for formal mathematics.\nBenchmark authors write trusted problem statements once in shared Lean modules, and the\ntooling generates one comparator workspace per problem under `generated/`.\n\nA submission is scored entirely by comparator results: a problem counts as solved iff\ncomparator accepts the submitted solution.\n\nThe main user-facing entrypoint is:\n\n```bash\nlake exe lean-eval --help\n```\n\n## Quick Start For Benchmark Problem Contributors\n\nUse this path if you are adding or editing benchmark problems.\n\n### 1. Install and fetch dependencies\n\n```bash\nlake exe cache get\nlake build\n```\n\n### 2. Add or edit a trusted theorem\n\nPut the statement in one of the shared modules under `LeanEval/` and mark it with\n`@[eval_problem]`.\n\n```lean\n@[eval_problem]\ntheorem my_new_problem : ... := by\n  sorry\n```\n\nCurrent source modules live in topic folders such as:\n\n- `LeanEval/NumberTheory/`\n- `LeanEval/Topology/`\n- `LeanEval/ComplexAnalysis/`\n- `LeanEval/EasyProblems.lean`\n\n### 3. Add the manifest entry\n\nEach tagged declaration must be listed by exactly one entry in\n[`manifests/problems.toml`](manifests/problems.toml). The `holes` array\nnames every `@[eval_problem]`-tagged declaration in the module that the\nproblem owns; for the common single-theorem case it has one element.\n\n```toml\n[[problem]]\nid = \"my_new_problem\"\ntitle = \"My new problem\"\ntest = false\nmodule = \"LeanEval.SomeModule\"\nholes = [\"my_new_problem\"]\nsubmitter = \"Your Name\"\nnotes = \"Optional notes.\"\nsource = \"Optional citation or URL.\"\ninformal_solution = \"Optional proof sketch or reference.\"\n```\n\nThe required fields are:\n\n- `id`\n- `title`\n- `test`\n- `module`\n- `holes`\n- `submitter`\n\n#### Multi-hole problems\n\nA problem may bundle several `def`s, `instance`s and `theorem`s — list\nthem all in `holes`. Comparator then asks the participant to fill every\nlisted declaration in their `Submission.lean`. Two conventions:\n\n- **Name every instance.** The generator addresses holes by their\n  declaration name, and Lean's auto-generated names for anonymous\n  `instance`s (e.g. `instTopologicalSpaceJacobian`) are not stable. Use\n  `instance instAddCommGroup : ... := sorry` rather than `instance : ...\n  := sorry`.\n- **Use `holes` even for one declaration.** There is no `theorem = \"...\"`\n  shorthand; a singleton hole is just `holes = [\"my_thm\"]`.\n\nSee [`LeanEval/Sandbox/DefHoleExample.lean`](LeanEval/Sandbox/DefHoleExample.lean)\nand [`LeanEval/Sandbox/InstanceHoleExample.lean`](LeanEval/Sandbox/InstanceHoleExample.lean)\nfor the smallest working examples, or\n[`LeanEval/Geometry/JacobianChallenge.lean`](LeanEval/Geometry/JacobianChallenge.lean)\nfor a realistic multi-hole problem.\n\n### 4. Validate the authored source of truth (optional)\n\n```bash\nlake exe lean-eval validate-manifest\nlake exe lean-eval check-problem-build\n```\n\n`validate-manifest` checks that `@[eval_problem]` declarations and manifest entries\nmatch. `check-problem-build` builds the problem modules so warning-producing Lean changes\ndo not slip through. Both are cheap and catch the most common mistakes before a CI\nroundtrip.\n\n### 5. Open a PR\n\nThat's it — push your branch and open a PR. CI regenerates the comparator workspaces\nunder `generated/` and verifies they build, so you do not need to commit anything\nunder `generated/` yourself. Once the PR merges, a separate workflow regenerates\n`generated/` on `main` and pushes the result.\n\nIf CI fails with a generation or build error, you'll need to fix the source. The\nfastest local equivalent is:\n\n```bash\nlake exe lean-eval generate --problem my_new_problem\nlake exe lean-eval check-generated-builds --problem my_new_problem\n```\n\n## Quick Start For Solvers\n\nUse this path if you want to prove benchmark problems without touching the trusted\nbenchmark files.\n\n### 1. Pick a problem\n\nGenerated workspaces live under `generated/`, one directory per problem. The current\ncatalog is summarized in [`generated/index.json`](/home/kim/lean-evals/generated/index.json).\n\nExamples:\n\n- `generated/two_plus_two/`\n- `generated/list_append_singleton_length/`\n- `generated/cyclotomic_integer_house_le_two/`\n\n### 2. Create your local workspace\n\nCopy a clean starter workspace:\n\n```bash\nlake exe lean-eval start-problem two_plus_two\n```\n\nThat creates `workspaces/two_plus_two/` by default. You can also pass a destination:\n\n```bash\nlake exe lean-eval start-problem two_plus_two /tmp/two_plus_two\n```\n\n### 3. Install workspace dependencies\n\n```bash\ncd workspaces/two_plus_two\nlake update\n```\n\n### 4. Write your proof\n\nSolver-owned files are:\n\n- `Submission.lean`\n- `Submission/Helpers.lean`\n- any additional Lean files you add under `Submission/`\n\nTrusted files you should not edit in the normal solver workflow are:\n\n- `Challenge.lean`\n- `Solution.lean`\n- `config.json`\n- `lakefile.toml`\n\n`Challenge.lean` contains the benchmark statement. `Solution.lean` is the fixed bridge\nthat tells comparator to check your theorem from the `Submission` namespace.\n\n### 5. Run comparator locally\n\n```bash\nlake test\n```\n\nIf the comparator binary is not on your `PATH`, set it explicitly:\n\n```bash\nCOMPARATOR_BIN=/path/to/comparator lake test\n```\n\nYou can verify the installation against the starter problem from the repo root with:\n\n```bash\nlake exe lean-eval check-comparator-installation\n```\n\nComparator setup also requires the upstream external tools, including `landrun` and\n`lean4export`. Install `landrun` from its git `main` branch\n(`go install github.com/zouuup/landrun/cmd/landrun@main`); the latest tagged\nrelease (v0.1.15) is missing fixes that comparator's sandbox relies on.\n\nCI pins `lean4export` to tag `v4.30.0-rc2` and `comparator` to commit\n`71b52ec29e06d4b7d882726553b1ceb99a2499e0` (which adds support for\n`def`-shaped holes).\n\n### 6. Check your local score\n\nFrom the repo root:\n\n```bash\nlake exe lean-eval run-eval\nlake exe lean-eval run-eval --json\n```\n\nThe scorer prefers `workspaces/\u003cproblem-id\u003e/` when present and falls back to\n`generated/\u003cproblem-id\u003e/` otherwise.\n\n## Submission Rules\n\nParticipants may use Mathlib freely.\n\nIf a proof needs helper code that is not already in Mathlib, that code must be included\ninside the submission workspace itself. Multi-file submissions are allowed through\n`Submission.lean` and extra local modules under `Submission/`.\n\nFor benchmark-repo submissions, validate changed paths with:\n\n```bash\nlake exe lean-eval validate-submission --file generated/two_plus_two/Solution.lean\n```\n\nThe current validator accepts:\n\n- modifications to `generated/\u003cproblem-id\u003e/Solution.lean` and\n  `generated/\u003cproblem-id\u003e/Submission.lean`\n- additions, modifications, deletions, renames, or copies of `.lean` files under\n  `generated/\u003cproblem-id\u003e/Submission/`\n- additions (only) of markdown files anywhere inside\n  `generated/\u003cproblem-id\u003e/`, other than the generated `README.md`\n- additions (only) of a top-level `generated/\u003cproblem-id\u003e/LICENCE` or\n  `generated/\u003cproblem-id\u003e/LICENSE` file\n\nIn practice, solvers should normally work in `Submission.lean` and `Submission/`.\n\n## Repository Layout\n\n- [`LeanEval/`](/home/kim/lean-evals/LeanEval): trusted authored problem statements\n- [`manifests/problems.toml`](/home/kim/lean-evals/manifests/problems.toml): problem metadata\n- [`generated/`](/home/kim/lean-evals/generated): generated comparator workspaces\n- [`scripts/`](/home/kim/lean-evals/scripts): generation, validation, and scoring helpers\n- [`PLAN.md`](/home/kim/lean-evals/PLAN.md): deferred design and roadmap notes\n\n## End-To-End Repo Checks\n\nFor a local health pass over the repository:\n\n```bash\nlake exe lean-eval validate-manifest\nlake exe lean-eval check-problem-build\nlake exe lean-eval generate --check\nlake exe lean-eval check-generated-builds\nlake exe lean-eval run-eval\n```\n\nThere is also an end-to-end workflow self-check:\n\n```bash\nlake exe lean-eval check-eval-workflow\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fleanprover%2Flean-eval","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fleanprover%2Flean-eval","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fleanprover%2Flean-eval/lists"}