{"id":13754339,"url":"https://github.com/openai/miniF2F","last_synced_at":"2025-05-09T22:32:01.303Z","repository":{"id":37785421,"uuid":"364208834","full_name":"openai/miniF2F","owner":"openai","description":"Formal to Formal Mathematics Benchmark","archived":false,"fork":false,"pushed_at":"2023-08-16T09:36:09.000Z","size":712,"stargazers_count":341,"open_issues_count":10,"forks_count":46,"subscribers_count":116,"default_branch":"main","last_synced_at":"2025-04-05T23:06:18.549Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":"Objective-C++","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/openai.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null}},"created_at":"2021-05-04T09:46:25.000Z","updated_at":"2025-04-02T06:37:53.000Z","dependencies_parsed_at":"2024-01-18T04:17:36.122Z","dependency_job_id":"cf7820d9-27b4-4179-a8bf-183e03d09326","html_url":"https://github.com/openai/miniF2F","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/openai%2FminiF2F","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/openai%2FminiF2F/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/openai%2FminiF2F/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/openai%2FminiF2F/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/openai","download_url":"https://codeload.github.com/openai/miniF2F/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":253335827,"owners_count":21892745,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-08-03T09:01:55.419Z","updated_at":"2025-05-09T22:31:56.287Z","avatar_url":"https://github.com/openai.png","language":"Objective-C++","funding_links":[],"categories":["NLP语料和数据集","Objective-C++","Benchmarks"],"sub_categories":["其他_文本生成、文本对话","Interactive theorem proving"],"readme":"# MiniF2F\n\nMiniF2F is a formal mathematics benchmark (translated across multiple formal systems) consisting of\nexercise statements from olympiads (AMC, AIME, IMO) as well as high-school and undergraduate maths\nclasses.\n\nThe goal of the project is to provide a shared benchmark to evaluate and directly compare automated\ntheorem proving systems based on the formal systems targeted, initially **Lean**, and **Metamath**\n(targeting also **Hol Light** and\n**Isabelle**).\n\nThe benchmark (released under permissive licenses (MIT for Metamath, Apache for Lean)) is a work in\nprogress and contributions are welcome and encouraged through pull requests.\n\n## Citation\n\nThe benchmark is described in detail in the following [pre-print](https://arxiv.org/abs/2109.00110):\n```\n@article{zheng2021minif2f,\n  title={MiniF2F: a cross-system benchmark for formal Olympiad-level mathematics},\n  author={Zheng, Kunhao and Han, Jesse Michael and Polu, Stanislas},\n  journal={arXiv preprint arXiv:2109.00110},\n  year={2021}\n}\n```\n\n## Statistics\n\n|           | Test | Valid |\n|:---------:|:----:|:-----:|\n|   Lean    |  244 |  244  |\n| Metamath  |  244 |  244  |\n| Isabelle  |  244 |  244  |\n| Hol Light |  165 |  165  |\n\n## Structure\n\nEach problem is represented by a unique name and a file for each of the formal systems we target.\nEach file consists at minima in the problem statement and optionally one or more example proofs\nassociated with it. The benchmark is divided in two splits:\n\n- `valid`: validation set that can be used while designing automated theorem proving systems\n  (early-stopping, reinforcement learning, data-augmentation, curriculum design, ...).\n- `test`: held-out test set reserved for final evaluation.\n\nNaming conventions are still a work in progress. Olympiads problems are generally named after their\ncompetition year and problem number (eg. `imo-1990-p3` or `aime-1983-p2`). Problems coming from a\nparticular dataset (eg the [MATH](https://arxiv.org/abs/2103.03874) dataset) are named to ease their\nretrieval (eg. `mathd-algebra-125`). Other problems are prefixed by a category hint and a unique\nname in the style of Metamath naming conventions (eg. `induction-11div10tonmn1ton`).\n\nEach exercise file complies to the following system-specific conventions.\n\n### Lean\n\nTo install the project make sure you have [elan](https://github.com/leanprover/elan) installed,\nthen in the directory where you want the project installed run:\n\n```\ngit clone https://github.com/openai/miniF2F\ncd miniF2F\nleanpkg configure\nleanpkg build\n```\n\nSince having one file per statement causes slowness in Lean parsing stage, all Lean statements are\nexceptionally aggregated in two files (`valid.lean` and `test.lean`). These files contain a list of\nthe problem statements defined as `theorem`s. Optionally, proofs for these statements are provided\nas well as potential lemmas to support the ground-truth proof.\n\nNo `theorem` should appear that do not correspond to a problem statement; use `lemma` instead.\n\nPlease use `lean/scripts/lint_style.py` to check all the statements pass the linter. You can also\nmake use of `lean/scripts/simple_formatter.sh` to enforce a few basic formatting rules.\n\nThe `lean` folder is released under the Apache License (so that it is aligned with Lean's mathlib\nlicense).\n\n### Metamath\n\nEach file contains the problem statement with the same name as the problem unique name. The\nstatement is commented (using Metamath convention) if provided without proof.\n\nThe `metamath` folder is released under the MIT License.\n\n### HOL Light\n\nEach file contains the problem statement defined as a HOL Light term\nwhose name must match the file name.\n\nThe `hollight` folder is released under the FreeBSD License.\n\n### Isabelle\n\nEach file contains the problem statement defined as a theorem\nwhose name must match the file name, optionally with a proof for it as well as the necessary\nimports.\n\nThe `isabelle` folder is released under the Apache License.\n\n## Code of Conduct and Contributions\n\nMiniF2F is meant to serve as a shared and useful resource for the machine learning community working\non formal mathematics. \n\nThere is no obligation tied with the use and reporting of a result based on miniF2F. But if you're\nusing it and discovering new proofs (manually or automatically) please contribute them back to the\nbenchmark.\n\nAll contributions, such as new statements for later versions, addition of missing statements for\nexisting versions, bug fixes, additional proofs are all welcome.\n\n## Versioning\n\nA version of miniF2F is defined by a frozen set of statements. The goal for each version is to get\nfull coverage on all formal systems for that version even if that might not be the case when the\nversion is frozen.\n\nWhen reporting a result based on miniF2F please always specify the version you used. The current\nversion is `v1`, frozen as of August 2021, including 244 statements (fully translated to Lean and\nMetamath but still WIP in other formal systems).\n\nEach version will live in its own branch to allow later additions of translated statements or fixes\nto existing statements as needed. The `main` branch remains reserved for active development and\nshould not be used when reporting results.\n\n### Active version\n\n- Version: `v1`\n- Freeze date: August 2021\n- Branch: [v1](https://github.com/openai/miniF2F/tree/v1)\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fopenai%2FminiF2F","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fopenai%2FminiF2F","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fopenai%2FminiF2F/lists"}