https://github.com/Ahren09/AgentReview

Official Implementation for EMNLP 2024 (Main Track, Oral) "AgentReview: Exploring Academic Peer Review with LLM Agent."
https://github.com/Ahren09/AgentReview

agent chatgpt computational-social-science deep-learning emnlp emnlp2024 gpt-4 gpt-4o large-language-models llm machine-learning

Last synced: about 19 hours ago
JSON representation

Official Implementation for EMNLP 2024 (Main Track, Oral) "AgentReview: Exploring Academic Peer Review with LLM Agent."

Host: GitHub
URL: https://github.com/Ahren09/AgentReview
Owner: Ahren09
License: apache-2.0
Created: 2024-09-29T00:21:20.000Z (almost 2 years ago)
Default Branch: main
Last Pushed: 2026-05-10T21:39:43.000Z (2 months ago)
Last Synced: 2026-05-10T23:30:39.166Z (2 months ago)
Topics: agent, chatgpt, computational-social-science, deep-learning, emnlp, emnlp2024, gpt-4, gpt-4o, large-language-models, llm, machine-learning
Language: Jupyter Notebook
Homepage: https://agentreview.github.io/
Size: 2.68 MB
Stars: 112
Watchers: 1
Forks: 10
Open Issues: 0
Metadata Files:
- Readme: README.md
- License: LICENSE

Awesome Lists containing this project

awesome-scientific-peer-review - AgentReview - Simulation of reviewer commitment, knowledgeability, intention, discussion, and decision drift. (⚙️ Systems and Methods / 🧑‍⚖️ Meta-Review and Decision Support)

README

          
🎓 AgentReview


The first LLM-agent simulation of the academic peer review process.




  

  

  

  

  

  

  

  

  





  





  Yiqiao Jin^1* ·

  Qinlin Zhao^2* ·

  Yiyang Wang¹ ·

  Hao Chen³ ·

  Kaijie Zhu⁴ ·

  Yijia Xiao⁵ ·

  Jindong Wang⁶





  ¹Georgia Institute of Technology  

  ²University of Science and Technology of China  

  ³Carnegie Mellon University  


  ⁴UC Santa Barbara  

  ⁵UC Los Angeles  

  ⁶William & Mary



_{^* Equal contribution.}


---

## 📊 At a glance

| Metric | Value |

| --- | --- |

| Total generated peer review documents | **53,800+** |

| Reviews & rebuttals | **10,460** |

| Reviewer–AC discussions | **23,535** |

| Meta-reviews / final decisions | **9,414 / 9,414** |

| Conferences covered | **ICLR 2020 – 2023** |

| Submissions sampled (oral · spotlight · poster · reject) | **523**  (19 · 29 · 125 · 350) |

| Decision variation attributable to reviewer bias | **37.1 %** |

---

## 📝 Abstract

Peer review is fundamental to the integrity and advancement of scientific publication. Traditional methods of peer review analyses often rely on exploration and statistics of existing peer review data, which do not adequately address the multivariate nature of the process, account for the latent variables, and are further constrained by privacy concerns due to the sensitive nature of the data. We introduce **AgentReview**, the first large language model (LLM) based peer review simulation framework, which effectively disentangles the impacts of multiple latent factors and addresses the privacy issue. Our study reveals significant insights, including a notable **37.1 % variation in paper decisions due to reviewers' biases**, supported by sociological theories such as the *social influence theory*, *altruism fatigue*, and *authority bias*. We believe that this study could offer valuable insights to improve the design of peer review mechanisms.

---

## 🏗️ Architecture



  



AgentReview models peer review as a structured **five-phase pipeline** with three role types — *reviewers*, *authors*, and *area chairs* — each instantiated as an LLM agent with configurable latent traits. By varying one trait at a time against a fixed *baseline* setting, the framework disentangles otherwise-confounded factors such as reviewer commitment, intention, knowledgeability, AC style, and author anonymity, while preserving real reviewer privacy.

---

📑 Table of Contents

- [Key findings](#-key-findings)

- [Framework](#-framework)

  - [Roles](#roles)

  - [Five-phase pipeline](#five-phase-pipeline)

- [Installation](#-installation)

- [Data setup](#-data-setup)

- [Quick start](#-quick-start)

- [Customizing your own setting](#%EF%B8%8F-customizing-your-own-setting)

- [Notes](#-notes)

- [Citation](#-citation)

- [Acknowledgments](#-acknowledgments)

- [License](#%EF%B8%8F-license)

---

## ✨ Key findings

Five sociological phenomena emerge from the simulation, each tied to a measurable shift in review outcomes:

| Phenomenon | Sociological theory | Quantitative effect |

| --- | --- | --- |

| **Social Influence** | Conformity to perceived majority opinion | **−27.2 %** standard deviation in ratings after rebuttals |

| **Altruism Fatigue & Peer Effects** | One free-rider triggers collective disengagement | A single under-committed reviewer drives a **−18.7 %** drop in commitment across all reviewers |

| **Groupthink & Echo Chamber** | Amplification of negative views among biased peers | **−0.17** rating among biased reviewers, plus a **−0.25** spillover on unbiased reviewers |

| **Authority Bias & Halo Effect** | Renowned-author identity inflates perceived quality | Revealing identity for just **10 %** of papers shifts **27.7 %** of decisions |

| **Anchoring Bias** | Heavy reliance on initial impressions | The rebuttal phase exerts only a **minimal** effect on final outcomes |

---

## 🔬 Framework

### Roles

Three LLM-agent roles are configured along orthogonal trait axes, all set via prompts:

| Role | Trait axis | Variants |

| --- | --- | --- |

| **Reviewer** | Commitment | responsible · irresponsible |

| | Intention | benign · malicious |

| | Knowledgeability | knowledgeable · unknowledgeable |

| **Author** | Identity disclosure | anonymous · known |

| **Area Chair** | Decision style | authoritarian · conformist · inclusive |

### Five-phase pipeline

| Phase | Stage | What happens |

| :-: | --- | --- |

| **I** | Reviewer Assessment | Three reviewers independently evaluate each manuscript |

| **II** | Author–Reviewer Discussion | Authors submit rebuttals addressing reviewer concerns |

| **III** | Reviewer–AC Discussion | The AC facilitates discussion; reviewers update their initial ratings |

| **IV** | Meta-Review Compilation | The AC synthesizes all signals into a single meta-review |

| **V** | Paper Decision | The AC makes the final accept / reject call (fixed acceptance rate of 32 %) |

---

## 📦 Installation

| Requirement | Version |

| --- | --- |

| Python | 3.10+ |

| LLM access | OpenAI **or** Azure OpenAI API key |

| OS | Linux / macOS / WSL |

```bash

git clone https://github.com/Ahren09/AgentReview.git

cd AgentReview

pip install -r requirements.txt

```

🔑 Set environment variables — OpenAI

```bash

export OPENAI_API_KEY=sk-...

```

🔑 Set environment variables — Azure OpenAI

```bash

export AZURE_ENDPOINT=https://.openai.azure.com/

export AZURE_DEPLOYMENT=

export AZURE_OPENAI_KEY=

```

---

## 💾 Data setup

Two zip archives are hosted on [Dropbox](https://www.dropbox.com/scl/fo/etzu5h8kwrx8vrcaep9tt/ALCnxFt2cT9aF477d-h1-E8?rlkey=9r5ep9psp8u4yaxxo9caf5nnc&st=aymhgu32&dl=0):

| Archive | Contents | Target | Required? |

| --- | --- | --- | :-: |

| [`AgentReview_Paper_Data.zip`](https://www.dropbox.com/scl/fi/l17brtbzsy3xwflqd58ja/AgentReview_Paper_Data.zip?rlkey=vldiexmgzi7zycmz7pumgbooc&st=b6g3nkry&dl=0) | PDFs of sampled ICLR papers + real ICLR 2020–2023 reviews | `data/` | ✅ |

| [`AgentReview_LLM_Reviews.zip`](https://www.dropbox.com/scl/fi/ckr0hpxyedx8u9s6235y6/AgentReview_LLM_Reviews.zip?rlkey=cgexir5xu38tm79eiph8ulbkq&st=q23x2trr&dl=0) | The full LLM-generated review dataset from the paper | `outputs/` | optional |

```bash

unzip AgentReview_Paper_Data.zip   -d data/

unzip AgentReview_LLM_Reviews.zip  -d outputs/    # optional

```

---

## 🚀 Quick start

Run a full simulated review pass on ICLR 2024 with a `malicious_Rx1` reviewer setting:

```bash

python run_paper_review_cli.py \

    --conference ICLR2024 \

    --openai_client_type azure_openai \

    --data_dir data \

    --experiment_name malicious_Rx1

```

Or explore interactively:

- **Notebook** — [`notebooks/demo.ipynb`](notebooks/demo.ipynb)

- **Live demo** — [Hugging Face Space](https://huggingface.co/spaces/Ahren09/AgentReview)

- **End-to-end script** — [`run.sh`](run.sh)

> **Note:** all project files should be run from the `AgentReview` directory.

---

## 🛠️ Customizing your own setting

Define a new setting in `agentreview/experiment_config.py` and register it in `all_settings`:

```python

all_settings = {

    "BASELINE":   baseline_setting,

    "benign_Rx1": benign_Rx1_setting,

    # ...

    "your_setting_name": your_setting,

}

```

---

## 📌 Notes

- We use a fixed acceptance rate of **32 %**, matching the actual ICLR 2020–2023 average. See [Conference Acceptance Rates](https://github.com/lixin4ever/Conference-Acceptance-Rate) for context.

- API providers can apply strict content filtering. You may need to relax filtering on your deployment to obtain complete generations.

---

## 📚 Citation

```bibtex

@inproceedings{jin2024agentreview,

  title     = {AgentReview: Exploring Peer Review Dynamics with LLM Agents},

  author    = {Jin, Yiqiao and Zhao, Qinlin and Wang, Yiyang and Chen, Hao

               and Zhu, Kaijie and Xiao, Yijia and Wang, Jindong},

  booktitle = {Proceedings of the 2024 Conference on Empirical Methods in

               Natural Language Processing (EMNLP)},

  year      = {2024}

}

```

---

## 🤝 Acknowledgments

The implementation builds on the [chatarena](https://github.com/Farama-Foundation/chatarena) multi-agent framework, and uses the [OpenReview API](https://github.com/openreview/openreview-py) to retrieve real ICLR submission data.

---

## ⚖️ License

Released under the [Apache License 2.0](LICENSE).

ecosyste.ms

Data

Tools

Indexes

Applications

Experiments

Awesome

https://github.com/Ahren09/AgentReview

Awesome Lists containing this project

README

🎓 AgentReview