{"id":49504171,"url":"https://github.com/leanprover/lean-eval-leaderboard","last_synced_at":"2026-05-01T14:00:37.441Z","repository":{"id":350656444,"uuid":"1207745090","full_name":"leanprover/lean-eval-leaderboard","owner":"leanprover","description":"Results for the lean-eval benchmark (https://github.com/leanprover/lean-eval)","archived":false,"fork":false,"pushed_at":"2026-04-30T07:55:46.000Z","size":110,"stargazers_count":3,"open_issues_count":1,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2026-04-30T08:16:36.896Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"https://leanprover.github.io/lean-eval-leaderboard/","language":"Lean","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/leanprover.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2026-04-11T10:45:48.000Z","updated_at":"2026-04-30T07:19:16.000Z","dependencies_parsed_at":null,"dependency_job_id":null,"html_url":"https://github.com/leanprover/lean-eval-leaderboard","commit_stats":null,"previous_names":["kim-em/lean-eval-leaderboard","leanprover/lean-eval-leaderboard"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/leanprover/lean-eval-leaderboard","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/leanprover%2Flean-eval-leaderboard","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/leanprover%2Flean-eval-leaderboard/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/leanprover%2Flean-eval-leaderboard/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/leanprover%2Flean-eval-leaderboard/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/leanprover","download_url":"https://codeload.github.com/leanprover/lean-eval-leaderboard/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/leanprover%2Flean-eval-leaderboard/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":32499691,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-04-30T13:12:12.517Z","status":"online","status_checked_at":"2026-05-01T02:00:05.856Z","response_time":64,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2026-05-01T14:00:23.988Z","updated_at":"2026-05-01T14:00:37.422Z","avatar_url":"https://github.com/leanprover.png","language":"Lean","funding_links":[],"categories":[],"sub_categories":[],"readme":"# lean-eval-leaderboard\n\n**[View the leaderboard →](https://lean-lang.org/eval/)**\n\nResults store and public website data source for the\n[lean-eval](https://github.com/leanprover/lean-eval) benchmark.\n\nThis repository holds machine-written artifacts produced by the lean-eval CI\nand by the leaderboard-site build pipeline.\n\n- `results/` is the append-only public success log written by lean-eval CI.\n- `site-data/` is the derived, presentation-oriented data consumed by the\n  public website.\n\n**Do not edit generated files here by hand.**\n\nSuccesses are **sticky**: once a problem is marked solved for a given user,\nthat record is never modified or removed, even if a later submission from the\nsame user no longer proves it.\n\n\u003e Submitter-facing instructions live on the Verso website at\n\u003e [`LeaderboardSite/Pages/Submit.lean`](LeaderboardSite/Pages/Submit.lean).\n\u003e This README is for people building or operating the leaderboard repository,\n\u003e not for people trying to submit proofs.\n\n## File layout\n\n```\nresults/\n  \u003cgithub-login\u003e.json\n\nsite-data/\n  problems.json\n  leaderboard.json\n```\n\nOne file per submitter. Users without any successful submission have no file.\nFilenames use the user's GitHub login, lowercased, since GitHub logins are\ncase-insensitive.\n\nThe `site-data/` directory is generated from the raw `results/` files together\nwith benchmark metadata imported from the benchmark repository. Its schema is\ndocumented in [docs/site-data-schema.md](docs/site-data-schema.md). Its files\nare derived (`.gitignore`d) and produced by `scripts/generate_site_data.py`,\nwhich the deploy workflow runs before every site build, and which local\ndevelopers invoke via `lake script run generate` (the `generate` script in\n`lakefile.lean` runs the generator and then builds the site). The deploy pins\nthe benchmark sibling clone to the commit recorded in\n`benchmark-snapshot/.benchmark-commit`, so the regenerated site-data and the\nchecked-in snapshot's catalog stay in lockstep.\n\n## Record schema (v2)\n\n```json\n{\n  \"schema_version\": 2,\n  \"user\": \"kim-em\",\n  \"solved\": {\n    \"Claude Opus 4.6\": {\n      \"two_plus_two\": {\n        \"solved_at\": \"2026-04-11T10:45:00Z\",\n        \"benchmark_commit\": \"8e1b9cf5e1d3c2b1a0f9e8d7c6b5a4938271605f\",\n        \"submission_repo\": \"kim-em/my-lean-eval-proofs\",\n        \"submission_ref\": \"deadbeefcafef00dbaadc0de1234567890abcdef\",\n        \"submission_public\": true,\n        \"issue_number\": 42\n      },\n      \"list_append_singleton_length\": {\n        \"solved_at\": \"2026-04-12T08:15:30Z\",\n        \"benchmark_commit\": \"8e1b9cf5e1d3c2b1a0f9e8d7c6b5a4938271605f\",\n        \"submission_repo\": \"kim-em/my-lean-eval-proofs\",\n        \"submission_ref\": \"deadbeefcafef00dbaadc0de1234567890abcdef\",\n        \"submission_public\": false,\n        \"issue_number\": 43\n      }\n    },\n    \"GPT-5.5\": {\n      \"two_plus_two\": {\n        \"solved_at\": \"2026-04-12T11:00:00Z\",\n        \"benchmark_commit\": \"8e1b9cf5e1d3c2b1a0f9e8d7c6b5a4938271605f\",\n        \"submission_repo\": \"kim-em/another-gist\",\n        \"submission_ref\": \"0123456789abcdef0123456789abcdef01234567\",\n        \"submission_public\": true,\n        \"issue_number\": 44\n      }\n    }\n  }\n}\n```\n\n### Top-level fields\n\n| Field            | Type    | Description                                              |\n| ---------------- | ------- | -------------------------------------------------------- |\n| `schema_version` | integer | Currently `2`.                                           |\n| `user`           | string  | GitHub login with original case preserved.               |\n| `solved`         | object  | Map from model name to per-problem solution records. Never empty. |\n\n### Per-model bucket\n\nThe keys of `solved` are free-form model identifiers as supplied on the\nsubmission form (e.g. `\"Claude Opus 4.7\"`, `\"GPT-5.5\"`, `\"Aristotle (Harmonic)\"`).\nEach value is an object mapping `\u003cproblem_id\u003e` to a solution record.\n\n### Per-problem solution record\n\n| Field              | Type    | Description                                                                            |\n| ------------------ | ------- | -------------------------------------------------------------------------------------- |\n| `solved_at`        | string  | ISO 8601 UTC timestamp of when the record was first written.                           |\n| `benchmark_commit` | string  | 40-character SHA of the `leanprover/lean-eval` commit evaluated against.               |\n| `submission_repo`  | string  | Identifier of the submission source: `owner/repo` for a GitHub repository, or `user/gist-id` for a gist. |\n| `submission_ref`   | string  | 40-character SHA pinning the submission at evaluation time. For repositories this is a commit SHA; for gists this is the gist revision SHA. |\n| `submission_public`| boolean | `true` if the submission source was public at evaluation time, `false` otherwise. The leaderboard site uses this to decide whether to link to the solution. |\n| `issue_number`     | integer | Issue number in `leanprover/lean-eval` that triggered the evaluation.                  |\n| `production_description` | string \\| absent | Optional free-form description of how the solution was produced. |\n\nThe `model` field has moved out of the per-problem record and become the\nkey of the surrounding bucket.\n\n## Write semantics\n\nWhen the lean-eval CI records a successful submission:\n\n1. It reads `results/\u003clogin\u003e.json`, or starts from an empty `solved` map.\n2. The submission carries one model name (the value of the issue form's\n   `Model` field). The CI uses that as the bucket key.\n3. For each problem that passed in the submission:\n   - If `solved[\u003cmodel\u003e][\u003cproblem_id\u003e]` already exists, **do nothing** (sticky no-op).\n   - Otherwise, create the model bucket if needed, and add a new record with\n     the fields above.\n4. If at least one new record was added, the CI commits and pushes the\n   updated file. If no new records were added, it makes no commit.\n\nThe map only ever grows: existing buckets keep their problems, and existing\nproblem records keep their original `solved_at` timestamps and audit trail\n(`benchmark_commit`, `submission_ref`, `issue_number`).\n\nA single user can therefore claim the same problem under several model\nnames — one record per `(user, model, problem)` triple. This is the change\nfrom v1, which keyed sticky records on `(user, problem)` only and so could\nrecord at most one model per problem per user.\n\n## Commit convention\n\nCommits by the lean-eval CI use the message form:\n\n```\nrecord: \u003clogin\u003e solved \u003cproblem_id\u003e[, \u003cproblem_id\u003e...] using \u003cmodel\u003e @ \u003cbenchmark_short_sha\u003e\n```\n\nOne commit per submission, grouping all newly-recorded problems for a\nsingle model together.\n\n## Schema evolution\n\nBreaking changes bump `schema_version`. Consumers should refuse to parse a\nfile whose `schema_version` they do not know. Non-breaking additive changes\n(new optional fields) keep the version number stable.\n\n## Website build direction\n\nThe public website should live in this repository and consume derived artifacts\nfrom `site-data/`, not the raw `results/` tree directly. The site itself will\nbe implemented in Verso.\n\n- Raw results preserve audit information and sticky solve history.\n- Derived site data resolves aggregation rules, ranking, and problem metadata.\n- The frontend should render `site-data/leaderboard.json` and\n  `site-data/problems.json` without re-implementing benchmark logic in the\n  browser.\n\nSee:\n\n- [docs/site-data-schema.md](docs/site-data-schema.md)\n- [docs/website-plan.md](docs/website-plan.md)\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fleanprover%2Flean-eval-leaderboard","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fleanprover%2Flean-eval-leaderboard","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fleanprover%2Flean-eval-leaderboard/lists"}