{"id":17573272,"url":"https://github.com/atomind-ai/mlip-arena","last_synced_at":"2026-04-02T14:32:19.246Z","repository":{"id":260195659,"uuid":"776930320","full_name":"atomind-ai/mlip-arena","owner":"atomind-ai","description":"Fair and transparent benchmark of machine-learned interatomic potentials (MLIPs), beyond basic error metrics","archived":false,"fork":false,"pushed_at":"2025-03-04T06:30:42.000Z","size":25523,"stargazers_count":56,"open_issues_count":11,"forks_count":2,"subscribers_count":0,"default_branch":"main","last_synced_at":"2025-03-04T07:32:05.134Z","etag":null,"topics":["benchmark-framework","interatomic-potentials","machine-learning","materials","molecules","quantum-chemistry"],"latest_commit_sha":null,"homepage":"https://huggingface.co/spaces/atomind/mlip-arena","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/atomind-ai.png","metadata":{"files":{"readme":".github/README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":"CITATION.cff","codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-03-24T20:36:55.000Z","updated_at":"2025-02-01T07:52:34.000Z","dependencies_parsed_at":"2024-12-15T07:26:03.301Z","dependency_job_id":"75841093-775f-4771-8485-ad60879a4839","html_url":"https://github.com/atomind-ai/mlip-arena","commit_stats":null,"previous_names":["atomind-ai/mlip-arena"],"tags_count":3,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/atomind-ai%2Fmlip-arena","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/atomind-ai%2Fmlip-arena/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/atomind-ai%2Fmlip-arena/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/atomind-ai%2Fmlip-arena/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/atomind-ai","download_url":"https://codeload.github.com/atomind-ai/mlip-arena/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":242487506,"owners_count":20136651,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["benchmark-framework","interatomic-potentials","machine-learning","materials","molecules","quantum-chemistry"],"created_at":"2024-10-21T21:00:38.527Z","updated_at":"2026-04-02T14:32:19.234Z","avatar_url":"https://github.com/atomind-ai.png","language":"Python","funding_links":[],"categories":["Universal Potentials"],"sub_categories":[],"readme":"\u003cdiv align=\"center\"\u003e\n    \u003ch1\u003e⚔️ MLIP Arena ⚔️\u003c/h1\u003e\n    \u003ca href=\"https://huggingface.co/spaces/atomind/mlip-arena\"\u003e\u003cimg src=\"https://img.shields.io/badge/%F0%9F%A4%97%20Hugging%20Face-Space-blue\" alt=\"Hugging Face\"\u003e\u003c/a\u003e\n    \u003ca href=\"https://neurips.cc/virtual/2025/poster/121648\"\u003e\u003cimg alt=\"Static Badge\" src=\"https://img.shields.io/badge/NeurIPS-Spotlight-magenta\"\u003e\u003c/a\u003e\n    \u003ca href=\"https://arxiv.org/abs/2509.20630\"\u003e\u003cimg src=\"https://img.shields.io/badge/arXiv-2509.20630-b31b1b\"\u003e\u003c/a\u003e\n    \u003ca href=\"https://openreview.net/forum?id=ysKfIavYQE#discussion\"\u003e\u003cimg alt=\"Static Badge\" src=\"https://img.shields.io/badge/ICLR AI4Mat-Spotlight-purple\"\u003e\u003c/a\u003e\n    \u003cbr\u003e\n    \u003ca href=\"https://github.com/atomind-ai/mlip-arena/actions\"\u003e\u003cimg alt=\"GitHub Actions Workflow Status\" src=\"https://img.shields.io/github/actions/workflow/status/atomind-ai/mlip-arena/test.yaml\"\u003e\u003c/a\u003e\n    \u003ca href=\"https://pypi.org/project/mlip-arena/\"\u003e\u003cimg alt=\"PyPI - Version\" src=\"https://img.shields.io/pypi/v/mlip-arena\"\u003e\u003c/a\u003e\n    \u003ca href=\"https://pypi.org/project/mlip-arena/\"\u003e\u003cimg alt=\"PyPI - Downloads\" src=\"https://img.shields.io/pypi/dm/mlip-arena\"\u003e\u003c/a\u003e\n    \u003ca href=\"https://zenodo.org/doi/10.5281/zenodo.13704399\"\u003e\u003cimg src=\"https://zenodo.org/badge/776930320.svg\" alt=\"DOI\"\u003e\u003c/a\u003e\n    \u003c!-- \u003ca href=\"https://discord.gg/W8WvdQtT8T\"\u003e\u003cimg alt=\"Discord\" src=\"https://img.shields.io/discord/1299613474820984832?logo=discord\"\u003e --\u003e\n\u003c/a\u003e\n\u003c/div\u003e\n\n![Thumnail](../serve/assets/workflow.png)\n\nFoundation machine learning interatomic potentials (MLIPs), trained on extensive databases containing millions of density functional theory (DFT) calculations, have revolutionized molecular and materials modeling, but existing benchmarks suffer from data leakage, limited transferability, and an over-reliance on error-based metrics tied to specific DFT references.\n\nWe introduce MLIP Arena, a unified benchmark platform for evaluating foundation MLIP performance beyond conventional error metrics. It focuses on revealing the physical soundness learned by MLIPs and assessing their utilitarian performance agnostic to underlying model architecture and training dataset.\n\n***By moving beyond static DFT references and revealing the important failure modes*** of current foundation MLIPs in real-world settings, MLIP Arena provides a reproducible framework to guide the next-generation MLIP development toward improved predictive accuracy and runtime efficiency while maintaining physical consistency.\n\nMLIP Arena leverages modern pythonic workflow orchestrator 💙\n [Prefect](https://www.prefect.io/) 💙\n to enable advanced task/flow chaining and caching.\n\n![Prefect](../serve/assets/prefect.png)\n\n\u003c!-- - [Announcement](#announcement)\n- [Installation](#installation)\n  - [From PyPI (prefect workflow only, *without* pretrained models)](#from-pypi-prefect-workflow-only-without-pretrained-models)\n  - [From source (with integrated pretrained models)](#from-source-with-integrated-pretrained-models)\n- [Quickstart](#quickstart)\n- [Workflow Overview](#workflow-overview)\n  - [✅ The first Prefect workflow: molecular dynamics](#-the-first-prefect-workflow-molecular-dynamics)\n  - [🚀 Parallelize benchmarks at scale](#-parallelize-benchmarks-at-scale)\n  - [List of modular tasks](#list-of-modular-tasks)\n- [Contribute and Development](#contribute-and-development)\n  - [Add new MLIP models](#add-new-mlip-models)\n    - [External ASE Calculator (easy)](#external-ase-calculator-easy)\n    - [Hugging Face Model (recommended, difficult)](#hugging-face-model-recommended-difficult)\n  - [Add new benchmark](#add-new-benchmark)\n- [Citation](#citation) --\u003e\n\n\u003e [!NOTE]\n\u003e Contributions of new tasks through PRs are very welcome! See [project page](https://github.com/orgs/atomind-ai/projects/1) for some outstanding tasks, or propose new feature requests in [Discussion](https://github.com/atomind-ai/mlip-arena/discussions/new?category=ideas).\n\n## Announcement\n\n- **[Sep 18, 2025]** [🎊 **MLIP Arena is accepted as a Spotlight (top 3.5%) at NeurIPS!** 🎊](https://neurips.cc/virtual/2025/poster/121648)\n- **[Apr 8, 2025]** [🎉 **MLIP Arena is accepted as an ICLR AI4Mat Spotlight!** 🎉](https://openreview.net/forum?id=ysKfIavYQE#discussion) Huge thanks to all co-authors for their contributions!\n\n\n## Installation\n\n### From PyPI (prefect workflow only, *without* pretrained models)\n\n```bash\npip install mlip-arena\n```\n\n### From source (with integrated pretrained models)\n\n\u003e [!CAUTION]\n\u003e We strongly recommend clean build in a new virtual environment due to the compatibility issues between multiple popular MLIPs. We provide a single installation script using `uv` for minimal package conflicts and fast installation!\n\n\u003e [!CAUTION]\n\u003e To automatically download fairchem model checkpoints, please make sure you have gained downloading access to their HuggingFace [***model repo (e.g. OMAT24)***](https://huggingface.co/facebook/OMAT24) (not dataset repo), and login locally on your machine through `hf auth login` (see [HF hub authentication](https://huggingface.co/docs/huggingface_hub/en/quick-start#authentication))\n\n**Linux**\n\n```bash\n# (Optional) Install uv, way faster than pip, why not? :)\ncurl -LsSf https://astral.sh/uv/install.sh | sh\nsource $HOME/.local/bin/env\n\ngit clone https://github.com/atomind-ai/mlip-arena.git\ncd mlip-arena\n\n# One script uv pip installation\nbash scripts/install.sh\n```\n\n\u003e [!TIP]\n\u003e Sometimes installing all compiled models takes all the available local storage. Optional pip flag `--no-cache` could be uesed. `uv cache clean` will be helpful too.\n\n**Mac**\n\n```bash\n# (Optional) Install uv\ncurl -LsSf https://astral.sh/uv/install.sh | sh\nsource $HOME/.local/bin/env\n# One script uv pip installation\nbash scripts/install-macosx.sh\n```\n\n## ⏩ Quickstart\n\nInstruction for individual benchmark is provided in the README in each corresponding folder under [/benchmark](../benchmarks/).\n\n## ⚙️ Workflow Overview\n\n### ✅ The first Prefect task: molecular dynamics\n\nArena provides a unified interface to run all the compiled MLIPs. This can be achieved simply by looping through `MLIPEnum`:\n\n```python\nfrom mlip_arena.models import MLIPEnum\nfrom mlip_arena.tasks import MD\nfrom mlip_arena.tasks.utils import get_calculator\n\nfrom ase import units\nfrom ase.build import bulk\n\natoms = bulk(\"Cu\", \"fcc\", a=3.6) * (5, 5, 5)\n\nresults = []\n\nfor model in MLIPEnum:\n    result = MD(\n        atoms=atoms,\n        calculator=get_calculator(\n            model,\n            calculator_kwargs=dict(), # passing into calculator\n            dispersion=True,\n            dispersion_kwargs=dict(\n                damping='bj', xc='pbe', cutoff=40.0 * units.Bohr\n            ), # passing into TorchDFTD3Calculator\n        ), # compatible with custom ASE Calculator\n        ensemble=\"nve\", # nvt, nvt available\n        dynamics=\"velocityverlet\", # compatible with any ASE Dynamics objects and their class names\n        total_time=1e3, # 1 ps = 1e3 fs\n        time_step=2, # fs\n    )\n    results.append(result)\n```\n\n### 🚀 Parallelize benchmarks at scale\n\nTo run multiple benchmarks in parallel, add `.submit` before the task function and wrap all the tasks into a flow to dispatch the tasks to worker for concurrent execution. See Prefect Doc on [tasks](https://docs.prefect.io/v3/develop/write-tasks) and [flow](https://docs.prefect.io/v3/develop/write-flows) for more details.\n\n```python\n...\nfrom prefect import flow\n\n@flow\ndef run_all_tasks():\n\n    futures = []\n    for model in MLIPEnum:\n        future = MD.submit(\n            atoms=atoms,\n            ...\n        )\n        future.append(future)\n\n    return [f.result(raise_on_failure=False) for f in futures]\n```\n\nFor a more practical example using HPC resources, please now refer to [submission script](../benchmarks/submit.py) or [MD stability benchmark](../benchmarks/stability/temperature.ipynb).\n\n### List of modular tasks\n\nThe implemented tasks are available under `mlip_arena.tasks.\u003cmodule\u003e.run` or `from mlip_arena.tasks import *` for convenient imports (currently doesn't work if [phonopy](https://phonopy.github.io/phonopy/install.html) is not installed).\n\n- [OPT](../mlip_arena/tasks/optimize.py#L56): Structure optimization\n- [EOS](../mlip_arena/tasks/eos.py#L42): Equation of state (energy-volume scan)\n- [MD](../mlip_arena/tasks/md.py#L200): Molecular dynamics with flexible dynamics (NVE, NVT, NPT) and temperature/pressure scheduling (annealing, shearing, *etc*)\n- [PHONON](../mlip_arena/tasks/phonon.py#L110): Phonon calculation driven by [phonopy](https://phonopy.github.io/phonopy/install.html)\n- [NEB](../mlip_arena/tasks/neb.py#L96): Nudged elastic band\n- [NEB_FROM_ENDPOINTS](../mlip_arena/tasks/neb.py#L164): Nudge elastic band with convenient image interpolation (linear or IDPP)\n- [ELASTICITY](../mlip_arena/tasks/elasticity.py#L78): Elastic tensor calculation\n\n## Contribute and Development\n\nPRs are welcome. Please clone the repo and submit PRs with changes.\n\nTo make change to huggingface space, fetch large files from git lfs first and run streamlit:\n\n```\ngit lfs fetch --all\ngit lfs pull\nstreamlit run serve/app.py\n```\n\n### Add new MLIP models\n\nIf you have pretrained MLIP models that you would like to contribute to the MLIP Arena and show benchmark in real-time, there are two ways:\n\n#### External ASE Calculator (easy)\n\n1. Implement new ASE Calculator class in [mlip_arena/models/externals](../mlip_arena/models/externals).\n2. Name your class with awesome model name and add the same name to [registry](../mlip_arena/models/registry.yaml) with metadata.\n\n\u003e [!CAUTION]\n\u003e Remove unneccessary outputs under `results` class attributes to avoid error for MD simulations. Please refer to [CHGNet](../mlip_arena/models/externals/chgnet.py) as an example.\n\n#### Hugging Face Model (recommended, difficult)\n\n0. Inherit Hugging Face [ModelHubMixin](https://huggingface.co/docs/huggingface_hub/en/package_reference/mixins) class to your awesome model class definition. We recommend [PytorchModelHubMixin](https://huggingface.co/docs/huggingface_hub/en/package_reference/mixins#huggingface_hub.PyTorchModelHubMixin).\n1. Create a new [Hugging Face Model](https://huggingface.co/new) repository and upload the model file using [push_to_hub function](https://huggingface.co/docs/huggingface_hub/en/package_reference/mixins#huggingface_hub.ModelHubMixin.push_to_hub).\n2. Follow the template to code the I/O interface for your model [here](../mlip_arena/models/README.md).\n3. Update model [registry](../mlip_arena/models/registry.yaml) with metadata\n\n### Run benchmarks and submit model\n\nOnce your model is ready (either registered or initialized as a custom ASE Calculator), you can run the core benchmark suite on a SLURM cluster:\n\n1. Move into the `benchmarks/` directory:\n   ```bash\n   cd benchmarks\n   ```\n2. Open and modify the `submit_model.py` template script. Under the **USER CONFIGURATION** section:\n   - Provide your `MODEL` (as a registered string or custom ASE Calculator instance).\n   - Adjust the `SLURM_CONFIG` parameters for your specific HPC allocation (including any conda environments or module loads in the `job_script_prologue`).\n3. Submit the pipeline:\n   ```bash\n   python submit_model.py\n   ```\n   This will dynamically distribute and run the core benchmarks (diatomics, EOS bulk, and E-V scans) via a Dask-Jobqueue on your SLURM cluster.\n\n### Add new benchmark\n\n\u003e [!NOTE]\n\u003e Please reuse, extend, or chain the general tasks defined [above](#list-of-modular-tasks) and add new folder and script under [/benchmarks](../benchmarks/)\n\n## Citation\n\nIf you find the work useful, please consider citing the following:\n\n```bibtex\n@inproceedings{\n    chiang2025mlip,\n    title={{MLIP} Arena: Advancing Fairness and Transparency in Machine Learning Interatomic Potentials via an Open, Accessible Benchmark Platform},\n    author={Yuan Chiang and Tobias Kreiman and Christine Zhang and Matthew C. Kuner and Elizabeth Jin Weaver and Ishan Amin and Hyunsoo Park and Yunsung Lim and Jihan Kim and Daryl Chrzan and Aron Walsh and Samuel M Blau and Mark Asta and Aditi S. Krishnapriyan},\n    booktitle={The Thirty-ninth Annual Conference on Neural Information Processing Systems Datasets and Benchmarks Track},\n    year={2025},\n    url={https://openreview.net/forum?id=SAT0KPA5UO}\n}\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fatomind-ai%2Fmlip-arena","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fatomind-ai%2Fmlip-arena","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fatomind-ai%2Fmlip-arena/lists"}