{"id":51138757,"url":"https://github.com/Ahren09/AgentReview","last_synced_at":"2026-07-14T03:00:40.332Z","repository":{"id":259986818,"uuid":"864708082","full_name":"Ahren09/AgentReview","owner":"Ahren09","description":"Official Implementation for EMNLP 2024 (Main Track, Oral) \"AgentReview: Exploring Academic Peer Review with LLM Agent.\"","archived":false,"fork":false,"pushed_at":"2026-05-10T21:39:43.000Z","size":2809,"stargazers_count":112,"open_issues_count":0,"forks_count":10,"subscribers_count":1,"default_branch":"main","last_synced_at":"2026-05-10T23:30:39.166Z","etag":null,"topics":["agent","chatgpt","computational-social-science","deep-learning","emnlp","emnlp2024","gpt-4","gpt-4o","large-language-models","llm","machine-learning"],"latest_commit_sha":null,"homepage":"https://agentreview.github.io/","language":"Jupyter Notebook","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/Ahren09.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2024-09-29T00:21:20.000Z","updated_at":"2026-05-10T21:40:00.000Z","dependencies_parsed_at":"2024-11-05T05:17:20.828Z","dependency_job_id":null,"html_url":"https://github.com/Ahren09/AgentReview","commit_stats":null,"previous_names":["ahren09/agentreview"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/Ahren09/AgentReview","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Ahren09%2FAgentReview","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Ahren09%2FAgentReview/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Ahren09%2FAgentReview/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Ahren09%2FAgentReview/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/Ahren09","download_url":"https://codeload.github.com/Ahren09/AgentReview/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Ahren09%2FAgentReview/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":35443979,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-26T15:22:16.424Z","status":"online","status_checked_at":"2026-07-14T02:00:06.603Z","response_time":114,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["agent","chatgpt","computational-social-science","deep-learning","emnlp","emnlp2024","gpt-4","gpt-4o","large-language-models","llm","machine-learning"],"created_at":"2026-06-25T20:30:42.282Z","updated_at":"2026-07-14T03:00:40.326Z","avatar_url":"https://github.com/Ahren09.png","language":"Jupyter Notebook","funding_links":[],"categories":["⚙️ Systems and Methods"],"sub_categories":["🧑‍⚖️ Meta-Review and Decision Support"],"readme":"\u003ch1 align=\"center\"\u003e🎓 AgentReview\u003c/h1\u003e\n\n\u003cp align=\"center\"\u003e\u003cem\u003eThe first LLM-agent simulation of the academic peer review process.\u003c/em\u003e\u003c/p\u003e\n\n\u003cp align=\"center\"\u003e\n  \u003ca href=\"https://aclanthology.org/2024.emnlp-main.70/\"\u003e\u003cimg alt=\"EMNLP 2024 Oral\" src=\"https://img.shields.io/badge/EMNLP_2024-Oral-8B0000\"\u003e\u003c/a\u003e\n  \u003ca href=\"https://arxiv.org/abs/2406.12708\"\u003e\u003cimg alt=\"arXiv\" src=\"https://img.shields.io/badge/arXiv-2406.12708-b31b1b?logo=arxiv\u0026logoColor=white\"\u003e\u003c/a\u003e\n  \u003ca href=\"https://aclanthology.org/2024.emnlp-main.70/\"\u003e\u003cimg alt=\"ACL Anthology\" src=\"https://img.shields.io/badge/ACL_Anthology-2024.emnlp--main.70-ED1C24\"\u003e\u003c/a\u003e\n  \u003ca href=\"https://huggingface.co/spaces/Ahren09/AgentReview\"\u003e\u003cimg alt=\"HF Demo\" src=\"https://img.shields.io/badge/🤗_Demo-Hugging_Face-FFD21E\"\u003e\u003c/a\u003e\n  \u003ca href=\"https://agentreview.github.io/\"\u003e\u003cimg alt=\"Website\" src=\"https://img.shields.io/badge/Website-agentreview.github.io-4C8BF5\"\u003e\u003c/a\u003e\n  \u003ca href=\"https://github.com/Ahren09/AgentReview\"\u003e\u003cimg alt=\"Code\" src=\"https://img.shields.io/badge/Code-GitHub-181717?logo=github\u0026logoColor=white\"\u003e\u003c/a\u003e\n  \u003cimg alt=\"Python\" src=\"https://img.shields.io/badge/Python-3.10%2B-3776AB?logo=python\u0026logoColor=white\"\u003e\n  \u003cimg alt=\"Gradio\" src=\"https://img.shields.io/badge/Gradio-5.4-F97316?logo=gradio\u0026logoColor=white\"\u003e\n  \u003ca href=\"LICENSE\"\u003e\u003cimg alt=\"License\" src=\"https://img.shields.io/badge/License-Apache_2.0-blue.svg\"\u003e\u003c/a\u003e\n\u003c/p\u003e\n\n\u003cp align=\"center\"\u003e\n  \u003cimg src=\"static/img/Overview.png\" alt=\"AgentReview overview\" width=\"82%\"\u003e\n\u003c/p\u003e\n\n\u003cp align=\"center\"\u003e\n  \u003ca href=\"https://ahren09.github.io/\"\u003eYiqiao Jin\u003c/a\u003e\u003csup\u003e1*\u003c/sup\u003e \u0026middot;\n  \u003ca href=\"https://scholar.google.com/citations?user=0tiw7wgAAAAJ\"\u003eQinlin Zhao\u003c/a\u003e\u003csup\u003e2*\u003c/sup\u003e \u0026middot;\n  \u003ca href=\"https://hello-diana.github.io/\"\u003eYiyang Wang\u003c/a\u003e\u003csup\u003e1\u003c/sup\u003e \u0026middot;\n  \u003ca href=\"https://hhhhhhao.github.io/\"\u003eHao Chen\u003c/a\u003e\u003csup\u003e3\u003c/sup\u003e \u0026middot;\n  \u003ca href=\"https://kaijiezhu11.github.io/\"\u003eKaijie Zhu\u003c/a\u003e\u003csup\u003e4\u003c/sup\u003e \u0026middot;\n  \u003ca href=\"https://yijia-xiao.com/\"\u003eYijia Xiao\u003c/a\u003e\u003csup\u003e5\u003c/sup\u003e \u0026middot;\n  \u003ca href=\"https://jd92.wang/\"\u003eJindong Wang\u003c/a\u003e\u003csup\u003e6\u003c/sup\u003e\n\u003c/p\u003e\n\n\u003cp align=\"center\"\u003e\n  \u003csup\u003e1\u003c/sup\u003eGeorgia Institute of Technology \u0026nbsp;\n  \u003csup\u003e2\u003c/sup\u003eUniversity of Science and Technology of China \u0026nbsp;\n  \u003csup\u003e3\u003c/sup\u003eCarnegie Mellon University \u0026nbsp;\u003cbr\u003e\n  \u003csup\u003e4\u003c/sup\u003eUC Santa Barbara \u0026nbsp;\n  \u003csup\u003e5\u003c/sup\u003eUC Los Angeles \u0026nbsp;\n  \u003csup\u003e6\u003c/sup\u003eWilliam \u0026amp; Mary\n\u003c/p\u003e\n\n\u003cp align=\"center\"\u003e\u003csub\u003e\u003csup\u003e*\u003c/sup\u003e Equal contribution.\u003c/sub\u003e\u003c/p\u003e\n\n---\n\n## 📊 At a glance\n\n| Metric | Value |\n| --- | --- |\n| Total generated peer review documents | **53,800+** |\n| Reviews \u0026 rebuttals | **10,460** |\n| Reviewer–AC discussions | **23,535** |\n| Meta-reviews / final decisions | **9,414 / 9,414** |\n| Conferences covered | **ICLR 2020 – 2023** |\n| Submissions sampled (oral · spotlight · poster · reject) | **523** \u0026nbsp;(19 · 29 · 125 · 350) |\n| Decision variation attributable to reviewer bias | **37.1 %** |\n\n---\n\n## 📝 Abstract\n\nPeer review is fundamental to the integrity and advancement of scientific publication. Traditional methods of peer review analyses often rely on exploration and statistics of existing peer review data, which do not adequately address the multivariate nature of the process, account for the latent variables, and are further constrained by privacy concerns due to the sensitive nature of the data. We introduce **AgentReview**, the first large language model (LLM) based peer review simulation framework, which effectively disentangles the impacts of multiple latent factors and addresses the privacy issue. Our study reveals significant insights, including a notable **37.1 % variation in paper decisions due to reviewers' biases**, supported by sociological theories such as the *social influence theory*, *altruism fatigue*, and *authority bias*. We believe that this study could offer valuable insights to improve the design of peer review mechanisms.\n\n---\n\n## 🏗️ Architecture\n\n\u003cp align=\"center\"\u003e\n  \u003cimg src=\"static/img/ReviewPipeline.png\" alt=\"AgentReview 5-phase pipeline\" width=\"92%\"\u003e\n\u003c/p\u003e\n\nAgentReview models peer review as a structured **five-phase pipeline** with three role types — *reviewers*, *authors*, and *area chairs* — each instantiated as an LLM agent with configurable latent traits. By varying one trait at a time against a fixed *baseline* setting, the framework disentangles otherwise-confounded factors such as reviewer commitment, intention, knowledgeability, AC style, and author anonymity, while preserving real reviewer privacy.\n\n---\n\n\u003cdetails\u003e\n\u003csummary\u003e\u003cb\u003e📑 Table of Contents\u003c/b\u003e\u003c/summary\u003e\n\n- [Key findings](#-key-findings)\n- [Framework](#-framework)\n  - [Roles](#roles)\n  - [Five-phase pipeline](#five-phase-pipeline)\n- [Installation](#-installation)\n- [Data setup](#-data-setup)\n- [Quick start](#-quick-start)\n- [Customizing your own setting](#%EF%B8%8F-customizing-your-own-setting)\n- [Notes](#-notes)\n- [Citation](#-citation)\n- [Acknowledgments](#-acknowledgments)\n- [License](#%EF%B8%8F-license)\n\n\u003c/details\u003e\n\n---\n\n## ✨ Key findings\n\nFive sociological phenomena emerge from the simulation, each tied to a measurable shift in review outcomes:\n\n| Phenomenon | Sociological theory | Quantitative effect |\n| --- | --- | --- |\n| **Social Influence** | Conformity to perceived majority opinion | **−27.2 %** standard deviation in ratings after rebuttals |\n| **Altruism Fatigue \u0026 Peer Effects** | One free-rider triggers collective disengagement | A single under-committed reviewer drives a **−18.7 %** drop in commitment across all reviewers |\n| **Groupthink \u0026 Echo Chamber** | Amplification of negative views among biased peers | **−0.17** rating among biased reviewers, plus a **−0.25** spillover on unbiased reviewers |\n| **Authority Bias \u0026 Halo Effect** | Renowned-author identity inflates perceived quality | Revealing identity for just **10 %** of papers shifts **27.7 %** of decisions |\n| **Anchoring Bias** | Heavy reliance on initial impressions | The rebuttal phase exerts only a **minimal** effect on final outcomes |\n\n---\n\n## 🔬 Framework\n\n### Roles\n\nThree LLM-agent roles are configured along orthogonal trait axes, all set via prompts:\n\n| Role | Trait axis | Variants |\n| --- | --- | --- |\n| **Reviewer** | Commitment | responsible · irresponsible |\n| | Intention | benign · malicious |\n| | Knowledgeability | knowledgeable · unknowledgeable |\n| **Author** | Identity disclosure | anonymous · known |\n| **Area Chair** | Decision style | authoritarian · conformist · inclusive |\n\n### Five-phase pipeline\n\n| Phase | Stage | What happens |\n| :-: | --- | --- |\n| **I** | Reviewer Assessment | Three reviewers independently evaluate each manuscript |\n| **II** | Author–Reviewer Discussion | Authors submit rebuttals addressing reviewer concerns |\n| **III** | Reviewer–AC Discussion | The AC facilitates discussion; reviewers update their initial ratings |\n| **IV** | Meta-Review Compilation | The AC synthesizes all signals into a single meta-review |\n| **V** | Paper Decision | The AC makes the final accept / reject call (fixed acceptance rate of 32 %) |\n\n---\n\n## 📦 Installation\n\n| Requirement | Version |\n| --- | --- |\n| Python | 3.10+ |\n| LLM access | OpenAI **or** Azure OpenAI API key |\n| OS | Linux / macOS / WSL |\n\n```bash\ngit clone https://github.com/Ahren09/AgentReview.git\ncd AgentReview\npip install -r requirements.txt\n```\n\n\u003cdetails\u003e\n\u003csummary\u003e\u003cb\u003e🔑 Set environment variables — OpenAI\u003c/b\u003e\u003c/summary\u003e\n\n```bash\nexport OPENAI_API_KEY=sk-...\n```\n\n\u003c/details\u003e\n\n\u003cdetails\u003e\n\u003csummary\u003e\u003cb\u003e🔑 Set environment variables — Azure OpenAI\u003c/b\u003e\u003c/summary\u003e\n\n```bash\nexport AZURE_ENDPOINT=https://\u003cyour-endpoint\u003e.openai.azure.com/\nexport AZURE_DEPLOYMENT=\u003cyour-deployment-name\u003e\nexport AZURE_OPENAI_KEY=\u003cyour-key\u003e\n```\n\n\u003c/details\u003e\n\n---\n\n## 💾 Data setup\n\nTwo zip archives are hosted on [Dropbox](https://www.dropbox.com/scl/fo/etzu5h8kwrx8vrcaep9tt/ALCnxFt2cT9aF477d-h1-E8?rlkey=9r5ep9psp8u4yaxxo9caf5nnc\u0026st=aymhgu32\u0026dl=0):\n\n| Archive | Contents | Target | Required? |\n| --- | --- | --- | :-: |\n| [`AgentReview_Paper_Data.zip`](https://www.dropbox.com/scl/fi/l17brtbzsy3xwflqd58ja/AgentReview_Paper_Data.zip?rlkey=vldiexmgzi7zycmz7pumgbooc\u0026st=b6g3nkry\u0026dl=0) | PDFs of sampled ICLR papers + real ICLR 2020–2023 reviews | `data/` | ✅ |\n| [`AgentReview_LLM_Reviews.zip`](https://www.dropbox.com/scl/fi/ckr0hpxyedx8u9s6235y6/AgentReview_LLM_Reviews.zip?rlkey=cgexir5xu38tm79eiph8ulbkq\u0026st=q23x2trr\u0026dl=0) | The full LLM-generated review dataset from the paper | `outputs/` | optional |\n\n```bash\nunzip AgentReview_Paper_Data.zip   -d data/\nunzip AgentReview_LLM_Reviews.zip  -d outputs/    # optional\n```\n\n---\n\n## 🚀 Quick start\n\nRun a full simulated review pass on ICLR 2024 with a `malicious_Rx1` reviewer setting:\n\n```bash\npython run_paper_review_cli.py \\\n    --conference ICLR2024 \\\n    --openai_client_type azure_openai \\\n    --data_dir data \\\n    --experiment_name malicious_Rx1\n```\n\nOr explore interactively:\n\n- **Notebook** — [`notebooks/demo.ipynb`](notebooks/demo.ipynb)\n- **Live demo** — [Hugging Face Space](https://huggingface.co/spaces/Ahren09/AgentReview)\n- **End-to-end script** — [`run.sh`](run.sh)\n\n\u003e **Note:** all project files should be run from the `AgentReview` directory.\n\n---\n\n## 🛠️ Customizing your own setting\n\nDefine a new setting in `agentreview/experiment_config.py` and register it in `all_settings`:\n\n```python\nall_settings = {\n    \"BASELINE\":   baseline_setting,\n    \"benign_Rx1\": benign_Rx1_setting,\n    # ...\n    \"your_setting_name\": your_setting,\n}\n```\n\n---\n\n## 📌 Notes\n\n- We use a fixed acceptance rate of **32 %**, matching the actual ICLR 2020–2023 average. See [Conference Acceptance Rates](https://github.com/lixin4ever/Conference-Acceptance-Rate) for context.\n- API providers can apply strict content filtering. You may need to relax filtering on your deployment to obtain complete generations.\n\n---\n\n## 📚 Citation\n\n```bibtex\n@inproceedings{jin2024agentreview,\n  title     = {AgentReview: Exploring Peer Review Dynamics with LLM Agents},\n  author    = {Jin, Yiqiao and Zhao, Qinlin and Wang, Yiyang and Chen, Hao\n               and Zhu, Kaijie and Xiao, Yijia and Wang, Jindong},\n  booktitle = {Proceedings of the 2024 Conference on Empirical Methods in\n               Natural Language Processing (EMNLP)},\n  year      = {2024}\n}\n```\n\n---\n\n## 🤝 Acknowledgments\n\nThe implementation builds on the [chatarena](https://github.com/Farama-Foundation/chatarena) multi-agent framework, and uses the [OpenReview API](https://github.com/openreview/openreview-py) to retrieve real ICLR submission data.\n\n---\n\n## ⚖️ License\n\nReleased under the [Apache License 2.0](LICENSE).\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FAhren09%2FAgentReview","html_url":"https://awesome.ecosyste.ms/projects/github.com%2FAhren09%2FAgentReview","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FAhren09%2FAgentReview/lists"}