{"id":51138712,"url":"https://github.com/jd-coderepos/awesome-scientific-peer-review","last_synced_at":"2026-06-25T20:30:42.075Z","repository":{"id":365656502,"uuid":"1273134981","full_name":"jd-coderepos/awesome-scientific-peer-review","owner":"jd-coderepos","description":"A curated list of datasets, tools, papers, models, benchmarks, and workflows for AI-assisted scientific peer review.","archived":false,"fork":false,"pushed_at":"2026-06-18T09:08:36.000Z","size":50,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2026-06-18T10:24:35.558Z","etag":null,"topics":["awesome","awesome-list","large-language-models","natural-language-processing","peer-review","scientific-publishing"],"latest_commit_sha":null,"homepage":null,"language":null,"has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"cc-by-4.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/jd-coderepos.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":"CODE_OF_CONDUCT.md","threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2026-06-18T08:36:40.000Z","updated_at":"2026-06-18T09:08:54.000Z","dependencies_parsed_at":null,"dependency_job_id":null,"html_url":"https://github.com/jd-coderepos/awesome-scientific-peer-review","commit_stats":null,"previous_names":["jd-coderepos/awesome-scientific-peer-review"],"tags_count":null,"template":false,"template_full_name":null,"purl":"pkg:github/jd-coderepos/awesome-scientific-peer-review","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jd-coderepos%2Fawesome-scientific-peer-review","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jd-coderepos%2Fawesome-scientific-peer-review/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jd-coderepos%2Fawesome-scientific-peer-review/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jd-coderepos%2Fawesome-scientific-peer-review/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/jd-coderepos","download_url":"https://codeload.github.com/jd-coderepos/awesome-scientific-peer-review/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jd-coderepos%2Fawesome-scientific-peer-review/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":34792208,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-26T15:22:16.424Z","status":"online","status_checked_at":"2026-06-25T02:00:05.521Z","response_time":101,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["awesome","awesome-list","large-language-models","natural-language-processing","peer-review","scientific-publishing"],"created_at":"2026-06-25T20:30:41.312Z","updated_at":"2026-06-25T20:30:42.062Z","avatar_url":"https://github.com/jd-coderepos.png","language":null,"funding_links":[],"categories":["Other Lists"],"sub_categories":["Vue Lists"],"readme":"\u003cp align=\"center\"\u003e\n  \u003cimg src=\"assets/banner.png\" alt=\"Awesome Scientific Peer Review\"\u003e\n\u003c/p\u003e\n\n\u003c!--lint disable double-link--\u003e\n\n\u003cp align=\"center\"\u003e\n  \u003ca href=\"https://awesome.re\"\u003e\u003cimg src=\"https://awesome.re/badge-flat.svg\" alt=\"Awesome\"\u003e\u003c/a\u003e\n  \u003ca href=\"https://creativecommons.org/licenses/by/4.0/\"\u003e\u003cimg src=\"https://img.shields.io/badge/license-CC%20BY%204.0-blue.svg\" alt=\"License: CC BY 4.0\"\u003e\u003c/a\u003e\n  \u003ca href=\"#contributing\"\u003e\u003cimg src=\"https://img.shields.io/badge/contributions-welcome-brightgreen.svg\" alt=\"Contributions Welcome\"\u003e\u003c/a\u003e\n\u003c/p\u003e\n\nA curated, taxonomy-aware catalog of datasets, systems, tools, and evaluation resources for AI-assisted scientific peer review.\n\n\u003e [!IMPORTANT]\n\u003e Scientific peer review requires expert judgment, confidentiality, fairness, and accountability. These resources should support human reviewers and editors, not replace responsible human decision-making.\n\n\u003c!--lint disable table-pipe-alignment--\u003e\n\n## Contents\n\n\u003c!--lint disable awesome-toc--\u003e\n\n- [📖 How to Use This Catalog](#-how-to-use-this-catalog)\n- [🧭 Browse by Task](#-browse-by-task)\n- [📐 Peer Review Evaluation Taxonomy](#-peer-review-evaluation-taxonomy)\n- [🗂️ Datasets and Benchmarks](#️-datasets-and-benchmarks)\n- [⚙️ Systems and Methods](#️-systems-and-methods)\n- [🧱 Infrastructure and Building Blocks](#-infrastructure-and-building-blocks)\n- [🏛️ Policies, Ethics, and Governance](#️-policies-ethics-and-governance)\n- [📚 Surveys and Field Studies](#-surveys-and-field-studies)\n- [✅ Evaluation Checklist](#-evaluation-checklist)\n- [🔭 Open Gaps](#-open-gaps)\n- [🔗 Related Awesome Lists](#-related-awesome-lists)\n- [🤝 Contributing](#-contributing)\n- [✨ Acknowledgements](#-acknowledgements)\n\n\u003c!--lint enable awesome-toc--\u003e\n\n## 📖 How to Use This Catalog\n\nEvery named resource is linked to its primary paper, repository, dataset, API, or official product page. Entries appear once in the canonical catalogs; the task and taxonomy tables are linked indexes into those resources.\n\n### Visual Legend\n\n\u003c!--lint disable awesome-list-item--\u003e\n\n| Dimension | Symbols |\n| --- | --- |\n| Resource | 🟦 dataset/benchmark · 🟩 code/tool · 🟪 system/model · 🟨 paper/survey · 🟥 policy/commercial |\n| Access | ✅ open artifact · 📦 open data · 📄 paper only · 🌐 hosted service/API · 🔒 proprietary · ⚠️ access unclear |\n| Workflow | 📥 ingest · 👥 match · ✍️ review · 🔍 assess · 🧑‍⚖️ meta-review · ↔️ rebuttal · 📝 revision · 🛡️ audit |\n| Taxonomy facet | 🧩 argument · 🎯 content aspect · 🧱 structure · 💬 sentiment · ⚖️ polarity |\n\n\u003c!--lint enable awesome-list-item--\u003e\n\n## 🧭 Browse by Task\n\n| Goal | Start with | Baseline or comparison |\n| --- | --- | --- |\n| Generate a scientific review | [NLPeer](https://github.com/UKPLab/nlpeer), [PeerRead](https://github.com/allenai/PeerRead) | Structured LLM prompt; compare with [ReviewAdvisor](https://github.com/neulab/ReviewAdvisor), [ReviewRobot](https://github.com/EagleW/ReviewRobot), or [TreeReview](https://github.com/YuanChang98/tree-review). |\n| Evaluate review usefulness | [RevUtil](https://github.com/bodasadallah/RevUtil), [ReAct](https://github.com/gtmdotme/ReAct) | Classify actionability/helpfulness; add grounding checks from [SubstanReview](https://github.com/YanzhuGuo/SubstanReview). |\n| Detect weak or deficient critiques | [ReviewCritique](https://github.com/jiangshdd/ReviewCritique), [LazyReview](https://github.com/UKPLab/acl2025-lazy-review) | Rubric-based classifier or LLM judge with human validation. |\n| Measure aspect coverage and blind spots | [PeerRead](https://aclanthology.org/N18-1145/), [Peer Review Analyze](https://doi.org/10.1371/journal.pone.0259238) | Sentence-level aspect classifier; compare focus allocation with [Mind the Blind Spots](https://arxiv.org/abs/2502.17086). |\n| Mine arguments in reviews | [AMPERE](https://xinyuhua.github.io/Resources/naacl19/), [Argument Mining Driven Analysis](https://arxiv.org/abs/2012.07743) | Proposition classifier for evaluation, request, fact, reference, and quote. |\n| Link reviews and rebuttals | [DISAPERE](https://github.com/nnkennard/DISAPERE), [APE](https://github.com/LiyingCheng95/ArgumentPairExtraction) | Sentence-pair similarity or NLI; compare with [MLMC](https://github.com/TianyuTerry/MLMC). |\n| Generate or evaluate meta-reviews | [MReD](https://arxiv.org/abs/2110.07474), [ReviewAgents](https://arxiv.org/abs/2503.08506) | Structured synthesis that preserves disagreement and uncertainty. |\n| Align reviews with manuscript changes | [ARIES](https://arxiv.org/abs/2306.12587), [CASIMIR](https://arxiv.org/abs/2403.00241), [F1000RD](https://github.com/UKPLab/f1000rd) | Sentence alignment between review comments and edited spans. |\n| Match reviewers to papers | [OpenReview Matcher](https://github.com/openreview/openreview-matcher) | Abstract/publication similarity using [Sentence Transformers](https://github.com/UKPLab/sentence-transformers), then add conflicts and workload constraints. |\n| Build a literature-grounded assistant | [GROBID](https://github.com/kermitt2/grobid), [Semantic Scholar API](https://api.semanticscholar.org/api-docs/graph), [OpenAlex](https://docs.openalex.org/) | Parse → retrieve → critique; evaluate retrieval and faithfulness with [Ragas](https://github.com/explodinggradients/ragas) or [TruLens](https://github.com/truera/trulens). |\n| Evaluate an LLM reviewer | [ReviewCritique](https://github.com/jiangshdd/ReviewCritique), [RevUtil](https://github.com/bodasadallah/RevUtil), [YESciEval](https://arxiv.org/abs/2505.14279) | Multidimensional rubric plus qualified human review; do not rely on one scalar score. |\n| Audit robustness or gaming | [Breaking the Reviewer](https://github.com/Lin-TzuLing/Breaking-the-Reviewer), [Are We There Yet?](https://arxiv.org/abs/2412.01708) | Repeated runs, paper rewrites, adversarial phrasing, and prompt-injection tests. |\n\n## 📐 Peer Review Evaluation Taxonomy\n\nThe catalog incorporates the two-level taxonomy from the attached research paper. **Paper review evaluation facets** describe what a review evaluates about the manuscript. **Peer review evaluation facets** describe how well the review itself performs that evaluation.\n\nThe five facet families below are cross-cutting. They are useful both for choosing datasets and for designing evaluation modules.\n\n| Facet family | Paper review evaluation | Peer review evaluation | Representative resources |\n| --- | --- | --- | --- |\n| 🧩 **Argument** | Evaluations, requests, facts, references, claims, evidence, and review-rebuttal links. | Actionability, substantiation, argument completeness, balance, and deficiency types. | [AMPERE](https://xinyuhua.github.io/Resources/naacl19/), [APE](https://github.com/LiyingCheng95/ArgumentPairExtraction), [DISAPERE](https://github.com/nnkennard/DISAPERE), [ReAct](https://github.com/gtmdotme/ReAct), [SubstanReview](https://github.com/YanzhuGuo/SubstanReview), [RevUtil](https://github.com/bodasadallah/RevUtil), [ReviewCritique](https://github.com/jiangshdd/ReviewCritique). |\n| 🎯 **Content aspect** | Novelty, originality, soundness, correctness, clarity, substance, impact, comparison, appropriateness, and replicability. | Coverage, balance, attention skew, and omission of important manuscript weaknesses. | [PeerRead](https://aclanthology.org/N18-1145/), [Peer Review Analyze](https://doi.org/10.1371/journal.pone.0259238), [ReviewAdvisor](https://github.com/neulab/ReviewAdvisor), [Identifying Aspects in Peer Reviews](https://github.com/UKPLab/emnlp2025-aspects-in-reviews), [Mind the Blind Spots](https://arxiv.org/abs/2502.17086). |\n| 🧱 **Structure** | Summary-strengths-weaknesses-conclusion organization, question trees, causal workflows, and links to manuscript sections. | Pragmatic composition, review-rebuttal interaction, cross-round evolution, and revision structure. | [F1000RD](https://github.com/UKPLab/f1000rd), [NLPeer](https://github.com/UKPLab/nlpeer), [MReD](https://arxiv.org/abs/2110.07474), [DISAPERE](https://github.com/nnkennard/DISAPERE), [TreeReview](https://github.com/YuanChang98/tree-review), [STRICTA](https://github.com/UKPLab/acl2025-stricta). |\n| 💬 **Sentiment** | Aspect-conditioned positive or negative sentiment about manuscript properties. | Politeness, harshness, helpfulness, and interpersonal tone. | [Aspect-Based Sentiment Analysis](https://arxiv.org/abs/2006.03257), [Peer Review Analyze](https://doi.org/10.1371/journal.pone.0259238), [RevUtil](https://github.com/bodasadallah/RevUtil), [Peer Reviews of Peer Reviews](https://doi.org/10.1371/journal.pone.0320444). |\n| ⚖️ **Polarity** | Strength versus weakness, accept-supporting versus reject-supporting arguments, disagreement, and contradiction. | Positivity bias, contradiction, confidence, uncertainty, hedging, and calibration. | [Argument Mining Driven Analysis](https://arxiv.org/abs/2012.07743), [When Reviewers Lock Horns](https://github.com/sandeep82945/Contradiction-in-Peer-Review), [HedgePeer](https://github.com/Tirthankar-Ghosal/HedgePeer-Dataset), [Breaking the Reviewer](https://github.com/Lin-TzuLing/Breaking-the-Reviewer). |\n\n### Taxonomy-Aligned Evaluation Blueprint\n\nA practical reviewer-assistance evaluation should combine:\n\n1. 🎯 **Content-aspect coverage** - Did the review inspect the criteria that matter for this paper and venue?\n2. 🧩 **Argument and grounding** - Are criticisms specific, justified, and linked to manuscript evidence?\n3. 🧱 **Structure and interaction** - Is feedback organized, traceable, and responsive across reviews, rebuttals, and revisions?\n4. 💬 **Utility and tone** - Is the review professional and useful to authors?\n5. ⚖️ **Polarity and calibration** - Are strengths, weaknesses, disagreement, and uncertainty represented faithfully?\n6. 🛡️ **Robustness and integrity** - Is the system stable under manipulation, repetition, and changing model conditions?\n\n## 🗂️ Datasets and Benchmarks\n\nThis is the canonical dataset catalog. Access symbols describe the linked artifact, not every upstream source from which the dataset was derived.\n\n| Resource | Best for | Contains | Facets | Access |\n| --- | --- | --- | --- | --- |\n| [PeerRead](https://github.com/allenai/PeerRead) | Review generation, score prediction, aspect modeling | Papers, reviews, scores, decisions | 🎯 ⚖️ | 🟦 ✅ |\n| [NLPeer](https://github.com/UKPLab/nlpeer) | Cross-domain review modeling and guided reading | Papers, reviews, decisions, revisions | 🎯 🧱 | 🟦 ✅ |\n| [MOPRD](https://arxiv.org/abs/2212.04972) | Multidisciplinary, multi-round workflows | Manuscript versions, reviews, rebuttals, meta-reviews, decisions | 🧩 🧱 | 🟦 📄 |\n| [F1000RD](https://github.com/UKPLab/f1000rd) | Pragmatic roles, intertextual links, revision histories | Open-review articles, reviews, versions, links | 🧩 🧱 | 🟦 ✅ |\n| [AMPERE](https://xinyuhua.github.io/Resources/naacl19/) | Review argument mining | Reviews, proposition spans, argument roles | 🧩 | 🟦 📦 |\n| [DISAPERE](https://github.com/nnkennard/DISAPERE) | Review-rebuttal discourse and linking | Reviews, rebuttals, discourse/aspect/polarity labels, links | 🧩 🧱 ⚖️ | 🟦 ✅ |\n| [MReD](https://arxiv.org/abs/2110.07474) | Meta-review generation | Reviews, meta-reviews, structural labels | 🧱 | 🟦 📄 |\n| [Peer Review Analyze](https://doi.org/10.1371/journal.pone.0259238) | Purpose, section, aspect, and sentiment analysis | Reviews with multilayer sentence labels | 🎯 🧱 💬 | 🟦 ⚠️ |\n| [ReAct](https://github.com/gtmdotme/ReAct) | Actionability and comment function | Review comments, actionability/function labels | 🧩 | 🟦 ✅ |\n| [RevUtil](https://github.com/bodasadallah/RevUtil) | Review utility | Comments, actionability, grounding, verifiability, helpfulness | 🧩 💬 | 🟦 ✅ |\n| [ReviewCritique](https://github.com/jiangshdd/ReviewCritique) | Deficiency detection | Papers, reviews, deficiency labels, explanations | 🧩 | 🟦 ✅ |\n| [LazyReview](https://github.com/UKPLab/acl2025-lazy-review) | Lazy-thinking and critique failure modes | Review sentences, cognitive-deficiency categories | 🧩 ⚖️ | 🟦 ✅ |\n| [SubstanReview](https://github.com/YanzhuGuo/SubstanReview) | Claim-evidence grounding | Review claims and supporting evidence links | 🧩 | 🟦 ✅ |\n| [ARIES](https://arxiv.org/abs/2306.12587) | Review-to-edit alignment | Review comments and manuscript edits | 🧩 🧱 | 🟦 📄 |\n| [CASIMIR](https://arxiv.org/abs/2403.00241) | Revision intent and version alignment | Paper versions, aligned sentences, revision intents, reviews | 🧱 | 🟦 📄 |\n| [HedgePeer](https://github.com/Tirthankar-Ghosal/HedgePeer-Dataset) | Uncertainty and calibration language | Reviews, hedge cues, hedge spans | ⚖️ | 🟦 ✅ |\n| [ASAP-Review](https://aclanthology.org/2023.ijcnlp-srw.6/) | Aspect-informed score prediction | Review comments and reported aspect/sentiment annotations | 🎯 💬 | 🟦 ⚠️ |\n| [CiteTracked](https://pure.itu.dk/portal/da/publications/0270dbc3-5fa8-4b2c-8e65-3a867f5f8cb4) | Longitudinal review-impact analysis | ML papers, review text, citation counts | 🎯 | 🟦 📄 |\n| [GenReview](https://anonymous.4open.science/r/gen_review) | Human versus controlled LLM-review comparison | Human and positive/neutral/negative generated reviews | 💬 ⚖️ | 🟦 📦 |\n| [arXivEdits](https://arxiv.org/abs/2210.15067) | General scientific revision modeling | Aligned arXiv versions, edits, revision intents | 🧱 | 🟦 📄 |\n\n## ⚙️ Systems and Methods\n\n### ✍️ Review Generation and Critique\n\n- 🟪 ✅ [ReviewAdvisor](https://github.com/neulab/ReviewAdvisor) - Aspect-aware extract-then-generate baseline for summary, originality, soundness, comparison, and replicability.\n- 🟪 ✅ [ReviewRobot](https://github.com/EagleW/ReviewRobot) - Explainable review generation using paper, related-work, and background knowledge graphs.\n- 🟪 📄 [Automated Focused Feedback Generation](https://arxiv.org/abs/2405.20477) - Planner-investigator-reviewer-controller workflow for targeted manuscript weaknesses.\n- 🟪 📄 [MARG](https://arxiv.org/abs/2401.04259) - Leader-worker and aspect-specialist agents for long-paper review generation.\n- 🟪 ✅ [OpenReviewer](https://arxiv.org/abs/2412.11948) - Open scientific-review model for structured critique of machine-learning papers.\n- 🟪 📄 [ReviewAgents](https://arxiv.org/abs/2503.08506) - Multiple reviewer agents plus an area-chair agent for review and meta-review synthesis.\n- 🟪 ✅ [TreeReview](https://github.com/YuanChang98/tree-review) - Dynamic question-tree decomposition with answer sufficiency checks.\n- 🟪 ✅ [STRICTA](https://github.com/UKPLab/acl2025-stricta) - Human-correctable READ-EXTRACT-INFER workflow for critical assessment.\n- 🟪 🌐 [DeepReview](https://ai-researcher.net/) - Staged novelty verification, retrieval-grounded reviewing, and reliability checking.\n\n### 🔍 Review Quality and Facet Evaluation\n\n- 🟪 ✅ [Identifying Aspects in Peer Reviews](https://github.com/UKPLab/emnlp2025-aspects-in-reviews) - Paper Aspect Prediction and Review Aspect Prediction for expected versus observed review coverage.\n- 🟨 📄 [Mind the Blind Spots](https://arxiv.org/abs/2502.17086) - Focus-level comparison of review targets and evaluation aspects.\n- 🟨 📄 [Aspect-Based Sentiment Analysis of Scientific Reviews](https://arxiv.org/abs/2006.03257) - Aspect-conditioned sentiment and reviewer-disagreement modeling.\n- 🟨 📄 [Peer Reviews of Peer Reviews](https://doi.org/10.1371/journal.pone.0320444) - Randomized experiments on feedback about reviewing behavior and quality.\n- 🟨 📄 [Can Large Language Models Provide Useful Feedback on Research Papers?](https://arxiv.org/abs/2310.01783) - Large-scale human/LLM feedback comparison. ([Project](https://github.com/Weixin-Liang/LLM-scientific-feedback))\n- 🟨 📄 [Is Your Paper Being Reviewed by an LLM?](https://arxiv.org/abs/2502.19614) - Benchmark for detecting AI-generated peer reviews.\n\n### 🧩 Argument, Rebuttal, and Interaction\n\n- 🟪 ✅ [APE](https://github.com/LiyingCheng95/ArgumentPairExtraction) - Joint argument-span extraction and review-rebuttal pairing.\n- 🟪 ✅ [MLMC](https://github.com/TianyuTerry/MLMC) - Attention-guided cross-sequence model for argument-pair extraction.\n- 🟨 📄 [Argument Mining Driven Analysis of Peer Reviews](https://arxiv.org/abs/2012.07743) - PRO/CON/NON modeling relative to paper acceptability.\n- 🟩 ✅ [When Reviewers Lock Horns](https://github.com/sandeep82945/Contradiction-in-Peer-Review) - Contradiction and disagreement detection across reviews.\n- 🟪 📄 [DEFEND](https://arxiv.org/abs/2603.27360) - Human-in-the-loop, evidence-grounded rebuttal reasoning and drafting.\n\n### 🧑‍⚖️ Meta-Review and Decision Support\n\n- 🟪 ✅ [AgentReview](https://github.com/Ahren09/AgentReview) - Simulation of reviewer commitment, knowledgeability, intention, discussion, and decision drift.\n- 🟪 ✅ [Auto-PRE](https://github.com/cjj826/Auto-PRE) - Configurable reviewer-agent framework for evaluating language-generation systems.\n- 🟪 ✅ [ReviewGraph](https://github.com/relic-yuexi/ReViewGraph) - Heterogeneous graph reasoning over simulated reviewer-author debates.\n\n\u003e Decision-support systems should expose evidence, uncertainty, and reviewer disagreement while leaving final judgments to accountable human chairs and editors.\n\n### 👥 Reviewer Matching and Editorial Workflow\n\n- 🟩 🌐 [OpenReview](https://openreview.net/) - Peer-review platform with public APIs, assignments, discussions, rebuttals, and review data. ([API Documentation](https://docs.openreview.net/))\n- 🟩 ✅ [OpenReview Matcher](https://github.com/openreview/openreview-matcher) - Minimum-cost-flow assignment under affinities and constraints.\n- 🟥 🔒 [Prophy Referee Finder](https://www.prophy.ai/referee-finder) - Commercial reviewer discovery using publication records, semantic matching, and conflict signals.\n\n### 🛡️ Robustness and Integrity\n\n- 🟨 📄 [Are We There Yet?](https://arxiv.org/abs/2412.01708) - Tests manipulation, incompleteness, prestige bias, and other risks of LLM reviewing.\n- 🟪 ✅ [Breaking the Reviewer](https://github.com/Lin-TzuLing/Breaking-the-Reviewer) - Adversarial attacks against automated paper-reviewing systems.\n- 🟨 📄 [Stop Automating Peer Review Without Rigorous Evaluation](https://arxiv.org/abs/2605.03202) - Studies review homogenization and score gaming through paper rewriting.\n\n## 🧱 Infrastructure and Building Blocks\n\n### 📥 Scientific Document Parsing and Scholarly Retrieval\n\n- 🟩 ✅ [GROBID](https://github.com/kermitt2/grobid) - Converts scholarly PDFs into structured TEI XML.\n- 🟩 ✅ [CERMINE](https://github.com/CeON/CERMINE) - Extracts metadata, references, and text from scholarly PDFs.\n- 🟩 ✅ [Nougat](https://github.com/facebookresearch/nougat) - Neural PDF-to-markup model for academic documents.\n- 🟩 ✅ [Science Parse](https://github.com/allenai/science-parse) - Metadata and bibliography parser retained as a reference implementation.\n- 🟩 🌐 [Semantic Scholar Academic Graph API](https://api.semanticscholar.org/api-docs/graph) - Paper, author, citation, recommendation, and embedding endpoints.\n- 🟩 🌐 [OpenAlex](https://docs.openalex.org/) - Open scholarly graph for works, authors, institutions, concepts, and citations.\n- 🟩 🌐 [Crossref REST API](https://www.crossref.org/documentation/retrieve-metadata/rest-api/) - DOI metadata and bibliographic normalization.\n- 🟩 🌐 [OpenCitations](https://opencitations.net/) - Open citation data and APIs.\n- 🟩 🌐 [arXiv API](https://info.arxiv.org/help/api/) - Preprint metadata and search.\n- 🟩 🌐 [ACL Anthology](https://aclanthology.org/) - Open computational-linguistics papers, metadata, and PDFs.\n- 🟩 🌐 [Open Research Knowledge Graph](https://orkg.org/) - Structured comparison of research contributions. ([API](https://orkg.org/about/19/API))\n- 🟦 📦 [S2ORC](https://arxiv.org/abs/1911.02782) - Structured scholarly text, citations, figures, tables, and metadata.\n- 🟩 🌐 [Unpaywall](https://unpaywall.org/products/api) - Legally available open-access locations for scholarly works.\n- 🟩 ✅ [Sentence Transformers](https://github.com/UKPLab/sentence-transformers) - Embeddings and reranking for reviewer matching and literature retrieval.\n\n### 🔍 Evaluation Frameworks and Judge Models\n\n- 🟪 ✅ [YESciEval](https://arxiv.org/abs/2505.14279) - Configurable scientific-domain LLM-as-a-judge framework with evaluator qualification.\n- 🟪 ✅ [Prometheus](https://github.com/prometheus-eval/prometheus-eval) - Open rubric-conditioned evaluator models.\n- 🟨 📄 [G-Eval](https://arxiv.org/abs/2303.16634) - Rubric and chain-of-thought-based LLM evaluation method.\n- 🟩 ✅ [DeepEval](https://github.com/confident-ai/deepeval) - Unit-testing framework for LLM applications.\n- 🟩 ✅ [Ragas](https://github.com/explodinggradients/ragas) - Retrieval relevance, grounding, and faithfulness evaluation.\n- 🟩 ✅ [TruLens](https://github.com/truera/trulens) - Tracing and feedback functions for grounded LLM applications.\n- 🟩 ✅ [Phoenix](https://github.com/Arize-ai/phoenix) - Tracing and evaluation for retrieval, agents, and judges.\n\n### General RAG and Agent Frameworks\n\nThese are reusable building blocks, not peer-review-specific systems.\n\n- 🟩 ✅ [LlamaIndex](https://github.com/run-llama/llama_index) - Data and retrieval framework for literature-grounded assistants.\n- 🟩 ✅ [Haystack](https://github.com/deepset-ai/haystack) - Retrieval, document processing, and production RAG pipelines.\n- 🟩 ✅ [LangChain](https://github.com/langchain-ai/langchain) - Retrieval, tools, structured output, and agent orchestration.\n- 🟩 ✅ [DSPy](https://github.com/stanfordnlp/dspy) - Optimization of modular language-model programs.\n- 🟩 ✅ [AutoGen](https://github.com/microsoft/autogen) - Multi-agent application framework.\n- 🟩 ✅ [CrewAI](https://github.com/crewAIInc/crewAI) - Role-based agent orchestration.\n- 🟩 ✅ [Semantic Kernel](https://github.com/microsoft/semantic-kernel) - SDK for model calls, tools, memory, and agents.\n\n## 🏛️ Policies, Ethics, and Governance\n\n- 🟥 📄 [COPE Ethical Guidelines for Peer Reviewers](https://publicationethics.org/resources/guidelines-new/cope-ethical-guidelines-peer-reviewers) - Confidentiality, conflicts, objectivity, timeliness, and accountability.\n- 🟥 📄 [Nature Portfolio Artificial Intelligence Policy](https://www.nature.com/nature-portfolio/editorial-policies/ai) - AI authorship, generated content, and reviewer confidentiality.\n- 🟥 📄 [ACM Policy on Authorship](https://www.acm.org/publications/policies/new-acm-policy-on-authorship) - Generative AI, authorship responsibility, disclosure, and publication integrity.\n- 🟥 📄 [ICLR Code of Ethics](https://iclr.cc/public/CodeOfEthics) - Ethics requirements relevant to confidential and responsible reviewing.\n- 🟥 📄 [NeurIPS Code of Ethics](https://neurips.cc/public/EthicsGuidelines) - Ethics guidance for authors, reviewers, and organizers.\n\n## 📚 Surveys and Field Studies\n\n- 🟨 📄 [Can AI Be a Good Peer Reviewer?](https://arxiv.org/abs/2604.27924) - Survey of review generation, rebuttal, meta-review, revision, evaluation, and governance.\n- 🟨 📄 [What Can Natural Language Processing Do for Peer Review?](https://arxiv.org/abs/2405.06563) - Task and dataset survey spanning the review workflow. ([Companion Resources](https://github.com/OAfzal/nlp-for-peer-review))\n- 🟨 📄 [Can We Automate Scientific Reviewing?](https://arxiv.org/abs/2102.00176) - Foundational analysis of aspect-aware review generation and evaluation.\n- 🟨 📄 [Challenges, Experiments, and Computational Solutions in Peer Review](https://doi.org/10.1145/3528086) - Reviewer assignment, review quality, incentives, bias, and conference experiments.\n- 🟨 📄 [Reviewer Assignment Problem: A Scoping Review](https://arxiv.org/abs/2305.07887) - Reviewer-matching methods, criteria, representations, and evaluation.\n\n\u003c!--lint disable awesome-list-item--\u003e\n\n## ✅ Evaluation Checklist\n\n- **Coverage:** Does the review inspect the manuscript aspects that matter for its type and venue?\n- **Grounding:** Are critiques linked to manuscript passages, experiments, references, or retrieved evidence?\n- **Utility:** Are comments actionable, specific, verifiable, and helpful?\n- **Structure:** Are summary, strengths, weaknesses, requests, and conclusions organized and traceable?\n- **Tone:** Is feedback professional without hiding legitimate criticism?\n- **Calibration:** Are uncertainty, confidence, and decision relevance represented appropriately?\n- **Human agreement:** Has the system been validated by qualified reviewers rather than only model judges?\n- **Robustness:** Is it stable under repeated runs, rewrites, adversarial text, and prompt injection?\n- **Governance:** Are confidentiality, provenance, licensing, disclosure, and accountability explicit?\n- **Reproducibility:** Are models, prompts, retrieval sources, rubrics, and versions documented?\n\n## 🔭 Open Gaps\n\n- Multidisciplinary, multilingual, and non-conference review corpora remain limited.\n- Few datasets connect reviews, rebuttals, revisions, editorial discussion, and decisions end to end.\n- Novelty, significance, and methodological validity still require strong domain expertise.\n- Review-quality benchmarks need better grounding, calibration, fairness, and long-document tests.\n- Multi-agent systems need reproducibility checks across runs and model versions.\n- Reviewer matching needs transparent conflict, workload, diversity, seniority, and recency evaluation.\n- More datasets need clear licenses, ethical release documentation, and machine-readable provenance.\n\n\u003c!--lint enable awesome-list-item--\u003e\n\n## 🔗 Related Awesome Lists\n\n- [Awesome Peer Review](https://github.com/formula12/Awesome-Peer-Review#readme) - Broader bibliography of computational peer-review research.\n- [Awesome Scientific Language Models](https://github.com/yuzhimanhua/Awesome-Scientific-Language-Models#readme) - Scientific language models and datasets.\n- [Awesome LLM Evaluation](https://github.com/CSHaitao/Awesome-LLMs-as-Judges#readme) - LLM judges, evaluator bias, robustness, and benchmarking.\n- [Awesome RAG](https://github.com/hymie122/RAG-Survey#readme) - Retrieval-augmented generation methods and benchmarks.\n- [Awesome AI Agents](https://github.com/e2b-dev/awesome-ai-agents#readme) - Agent frameworks and applications.\n- [Awesome LLM](https://github.com/Hannibal046/Awesome-LLM#readme) - General large-language-model research and engineering.\n\n## 🤝 Contributing\n\nContributions are welcome. Please read [CONTRIBUTING.md](CONTRIBUTING.md). Add each resource to one canonical catalog location and use the visual symbols from the legend.\n\n## ✨ Acknowledgements\n\nThis catalog was developed from original literature review and taxonomy research, including the two-level, five-family evaluation taxonomy summarized above.\n\nThe initial repository structure and editorial work were assisted by [OpenAI Codex](https://openai.com/codex/). Resource selection, interpretation, and maintenance remain human-led.\n\n\u003c!--lint enable double-link table-pipe-alignment--\u003e\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjd-coderepos%2Fawesome-scientific-peer-review","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fjd-coderepos%2Fawesome-scientific-peer-review","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjd-coderepos%2Fawesome-scientific-peer-review/lists"}