{"id":48647239,"url":"https://github.com/PouriaRouzrokh/LatteReview","last_synced_at":"2026-04-26T01:01:04.965Z","repository":{"id":267501566,"uuid":"901454241","full_name":"PouriaRouzrokh/LatteReview","owner":"PouriaRouzrokh","description":"This is the repository for the LatteReview python package. LatteReview is a low-code Python package designed to automate systematic literature review processes through AI-powered agents.","archived":false,"fork":false,"pushed_at":"2026-03-18T22:14:38.000Z","size":17592,"stargazers_count":94,"open_issues_count":4,"forks_count":8,"subscribers_count":5,"default_branch":"main","last_synced_at":"2026-03-19T04:56:02.315Z","etag":null,"topics":["ai","deep-learning","large-language-models","multi-agent","multi-agent-systems","multiagent","review","review-tools"],"latest_commit_sha":null,"homepage":"https://pouriarouzrokh.github.io/LatteReview","language":"Jupyter Notebook","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"other","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/PouriaRouzrokh.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":null,"funding":".github/FUNDING.yml","license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null},"funding":{"github":"PouriaRouzrokh","ko_fi":"pouriarouzrokh"}},"created_at":"2024-12-10T17:28:50.000Z","updated_at":"2026-03-19T02:58:41.000Z","dependencies_parsed_at":"2024-12-21T03:33:45.949Z","dependency_job_id":null,"html_url":"https://github.com/PouriaRouzrokh/LatteReview","commit_stats":null,"previous_names":["pouriarouzrokh/lattereviewers","pouriarouzrokh/lattereview"],"tags_count":1,"template":false,"template_full_name":null,"purl":"pkg:github/PouriaRouzrokh/LatteReview","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/PouriaRouzrokh%2FLatteReview","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/PouriaRouzrokh%2FLatteReview/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/PouriaRouzrokh%2FLatteReview/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/PouriaRouzrokh%2FLatteReview/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/PouriaRouzrokh","download_url":"https://codeload.github.com/PouriaRouzrokh/LatteReview/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/PouriaRouzrokh%2FLatteReview/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":32282187,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-04-25T18:29:39.964Z","status":"ssl_error","status_checked_at":"2026-04-25T18:29:32.149Z","response_time":59,"last_error":"SSL_read: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["ai","deep-learning","large-language-models","multi-agent","multi-agent-systems","multiagent","review","review-tools"],"created_at":"2026-04-10T06:00:29.406Z","updated_at":"2026-04-26T01:01:04.949Z","avatar_url":"https://github.com/PouriaRouzrokh.png","language":"Jupyter Notebook","funding_links":["https://github.com/sponsors/PouriaRouzrokh","https://ko-fi.com/pouriarouzrokh","http://ko-fi.com/pouriarouzrokh"],"categories":["🔬 Autonomous Research \u0026 Self-Improving Agents","🔬 Research-agent systems"],"sub_categories":["Research-Agent Systems"],"readme":"# LatteReview 🤖☕\n\n[![PyPI version](https://badge.fury.io/py/lattereview.svg)](https://badge.fury.io/py/lattereview)\n[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT)\n[![Python 3.9+](https://img.shields.io/badge/python-3.9+-blue.svg)](https://www.python.org/downloads/)\n[![Code style: black](https://img.shields.io/badge/code%20style-black-000000.svg)](https://github.com/psf/black)\n[![Maintained: yes](https://img.shields.io/badge/Maintained%3F-yes-green.svg)](https://github.com/prouzrokh/lattereview)\n[![View on arXiv](https://img.shields.io/badge/arXiv-View%20Paper-orange)](https://arxiv.org/abs/2501.05468)\n[![Sponsor me on GitHub](https://img.shields.io/badge/Sponsor%20me-GitHub%20Sponsors-pink.svg)](https://github.com/sponsors/PouriaRouzrokh)\n[![Support me on Ko-fi](https://img.shields.io/badge/Support%20me-Ko--fi-orange.svg?logo=ko-fi\u0026logoColor=white)](http://ko-fi.com/pouriarouzrokh)\n\n\u003cp\u003e\u003cimg src=\"docs/images/robot.png\" width=\"400\"\u003e\u003c/p\u003e\n\n---\n\n🚨 **NEW**: Now supports the Gemini 2.5 family of models using a new GoogleProvider class.\n\n---\n\nLatteReview is a powerful Python package designed to automate academic literature review processes through AI-powered agents. Just like enjoying a cup of latte ☕, reviewing numerous research articles should be a pleasant, efficient experience that doesn't consume your entire day!\n\n## 🎯 Key Features\n\n- Multi-agent review system with customizable roles and expertise levels for each reviewer\n- Support for multiple review rounds with hierarchical decision-making workflows\n- Review diverse content types including article titles, abstracts, custom texts, and even **images** using LLM-powered reviewer agents\n- Define reviewer agents with specialized backgrounds and distinct evaluation capabilities (e.g., scoring or concept abstraction or custom reviewers of your own preferance)\n- Create flexible review workflows where multiple agents operate in parallel or sequential arrangements\n- Enable reviewer agents to analyze peer feedback, cast votes, and propose corrections to other reviewers' assessments\n- Enhance reviews with item-specific context integration, supporting use cases like **Retrieval Augmented Generation (RAG)**\n- Broad compatibility with LLM providers through LiteLLM, including OpenAI and Ollama\n- Model-agnostic integration supporting OpenAI, Gemini, Claude, Groq, and local models via Ollama\n- High-performance asynchronous processing for efficient batch reviews\n- Standardized output format featuring detailed scoring metrics and reasoning transparency\n- Robust cost tracking and memory management systems\n- Extensible architecture supporting custom review workflow implementation\n- **NEW**: Support for RIS (Research Information Systems) file format for academic literature review\n\n## 💾Installation\n\n```bash\npip install lattereview\n```\n\nPlease refer to the [installation guide](./docs/installation.md) for detailed instructions.\n\n## 🚀 Quick Start and Documentation\n\nLatteReview enables you to create custom literature review workflows with multiple AI reviewers. Each reviewer can use different models and providers based on your needs. Below is a working example of how you can use LatteReview for doing a quick title/abstract review with two junior and one senior reviewers (all AI agents)! And this is just the beginning! Beyond study screening, LatteReview can handle data abstraction, customized pipelines, image analysis, and much more. Explore the [Tutorials](#-tutorials) for more examples!\n\nPlease refer to the [Quick Start](./docs/quickstart.md) page and [Documentation](https://pouriarouzrokh.github.io/LatteReview/) page for detailed instructions.\n\n```python\nfrom lattereview.providers import LiteLLMProvider\nfrom lattereview.agents import TitleAbstractReviewer\nfrom lattereview.workflows import ReviewWorkflow\nimport pandas as pd\nimport asyncio\nfrom dotenv import load_dotenv\n\n# Load environment variables from the .env file in the root directory of your project\nload_dotenv()\n\n# First Reviewer: Conservative approach\nreviewer1 = TitleAbstractReviewer(\n    provider=LiteLLMProvider(model=\"gpt-4o-mini\"),\n    name=\"Alice\",\n    backstory=\"a radiologist with expertise in systematic reviews\",\n    inclusion_criteria=\"The study must focus on applications of artificial intelligence in radiology.\",\n    exclusion_criteria=\"Exclude studies that are not peer-reviewed or not written in English.\",\n    model_args={\"temperature\": 0.2},\n)\n\n# Second Reviewer: More exploratory approach\nreviewer2 = TitleAbstractReviewer(\n    provider=LiteLLMProvider(model=\"gemini/gemini-1.5-flash\"),\n    name=\"Bob\",\n    backstory=\"a computer scientist specializing in medical AI\",\n    inclusion_criteria=\"The study must focus on applications of artificial intelligence in radiology.\",\n    exclusion_criteria=\"Exclude studies that are not peer-reviewed or not written in English.\",\n    model_args={\"temperature\": 0.2},\n)\n\n# Expert Reviewer: Resolves disagreements\nexpert = TitleAbstractReviewer(\n    provider=LiteLLMProvider(model=\"o3-mini\"),\n    name=\"Carol\",\n    backstory=\"a professor of AI in medical imaging\",\n    inclusion_criteria=\"The study must focus on applications of artificial intelligence in radiology.\",\n    exclusion_criteria=\"Exclude studies that are not peer-reviewed or not written in English.\",\n    model_args={\"reasoning_effort\": \"high\"},\n    additional_context=\"Alice and Bob disagree with each other on whether or not to include this article. You can find their reasonings above.\",\n)\n\n# Define workflow\nworkflow = ReviewWorkflow(\n    workflow_schema=[\n        {\n            \"round\": 'A',  # First round: Initial review by both reviewers\n            \"reviewers\": [reviewer1, reviewer2],\n            \"text_inputs\": [\"title\", \"abstract\"]\n        },\n        {\n            \"round\": 'B',  # Second round: Expert reviews only disagreements\n            \"reviewers\": [expert],\n            \"text_inputs\": [\"title\", \"abstract\", \"round-A_Alice_output\", \"round-A_Bob_output\"],\n            \"filter\": lambda row: row[\"round-A_Alice_evaluation\"] != row[\"round-A_Bob_evaluation\"]\n        }\n    ]\n)\n\n# Load and process your data\ndata = pd.read_excel(\"articles.xlsx\")  # Must have 'title' and 'abstract' columns\nresults = asyncio.run(workflow(data))  # Returns a pandas DataFrame with all original and output columns\n\n# Save results\nresults.to_csv(\"review_results.csv\", index=False)\n```\n\n## 🔌 Model Support\n\nLatteReview offers flexible model integration through multiple providers:\n\n- **LiteLLMProvider** (Recommended): Supports OpenAI, Anthropic (Claude), Gemini, Groq, and more\n- **OpenAIProvider**: Direct integration with OpenAI and Gemini APIs\n- **OllamaProvider**: Optimized for local models via Ollama\n\nNote: Models should support async operations and structured JSON outputs for optimal performance.\n\n## 📖 Documentation\n\nFull documentation and API reference are available at: [https://pouriarouzrokh.github.io/LatteReview](https://pouriarouzrokh.github.io/LatteReview)\n\n## 🎓 Tutorials\n\n✅ TitleAbstractReviewer: \n    🔸[1.](https://github.com/PouriaRouzrokh/LatteReview/blob/main/tutorials/title_abstract_review/title_abstract_review.ipynb) A simple task of abstract screening based on 1-5 scoring + inclusion and exclusion criteria\n✅ AbstractionReviewer:\n    🔸[1.](https://github.com/PouriaRouzrokh/LatteReview/blob/main/tutorials/abstraction_review_simple/abstraction_review_sample.ipynb) Data abstraction from abstracts/manuscripts\n✅ ScoringReviewer: \n    🔸[1.](https://github.com/PouriaRouzrokh/LatteReview/blob/main/tutorials/scoring_review_simple/scoring_review_simple.ipynb) A simple task of abstract screening based on custom scoring by multiple agents\n    🔸[2.](https://github.com/PouriaRouzrokh/LatteReview/blob/main/tutorials/scoring_review_rag/scoring_review_rag.ipynb) Question answering with RAG (Retrieval Augmented Generation)\n    🔸[3.](https://github.com/PouriaRouzrokh/LatteReview/blob/main/tutorials/scoring_review_image/scoring_review_image.ipynb) Image analysis by LatteReview  \n✅ Custom Reviewer:\n    🔸[1.](https://github.com/PouriaRouzrokh/LatteReview/blob/main/tutorials/custom_reviewer/abstraction_review_literature_analysis.ipynb) How to Customize the AbstractReviewer Agent for Your Needs\n    🔸[2.](https://github.com/PouriaRouzrokh/LatteReview/blob/main/tutorials/base_functionalities/base_functionalities.ipynb) Chat with the agents and other base functionalities\n    🔸[3.](https://github.com/PouriaRouzrokh/LatteReview/blob/main/tutorials/abstraction_review_literature_analysis/abstraction_review_literature_analysis.ipynb): Combination of differnet agents for a comprehensive literature review\n\n## 🛣️ Roadmap for Future Features\n\n- [x] Implementing LiteLLM to add support for additional model providers\n- [x] Draft the package full documentation\n- [x] Enable agents to return a percentage of certainty\n- [x] Enable agents to be grounded in static references (text provided by the user)\n- [x] Enable agents to be grounded in dynamic references (i.e., recieve a function that outputs a text based on the input text. This function could, e.g., be a RAG function.)\n- [x] Support for image-based inputs and multimodal analysis\n- [x] Development of `AbstractionReviewer` class for automated paper summarization\n- [x] Showcase how `AbstractionReviewer` class could be used to analyse the literature around a certain topic.\n- [x] Adding a tutorial example and also a section to the docs on how to create custom reviewer agents.\n- [x] Adding a `TitleAbstractReviewer` agent and adding a tutorial for it.\n- [x] Evaluating LatteReview.\n- [x] Writing the white paper for the package and public launch\n- [x] Addign support for `RIS` files.\n- [ ] Adding support for Deepseek R1 models (and models w/o structured output capablity in general).\n- [ ] Development of a no-code web application\n- [ ] (for v\u003e) Adding conformal prediction tool for calibrating agents on their certainty scores\n- [ ] (for v\u003e2.0.0) Adding a dialogue tool for enabling agents to seek external help (from helper agents or parallel reviewer agents) during review.\n- [ ] (for v\u003e2.0.0) Adding a memory component to the agents for saving their own insights or insightful feedback they receive from the helper agents.\n\n## 👨‍💻 Author\n\n\u003ctable border=\"0\"\u003e\n\u003ctr\u003e\n\u003ctd style=\"width: 80px;\"\u003e\n\u003cimg src=\"https://github.com/PouriaRouzrokh.png?size=80\" alt=\"Pouria Rouzrokh\" style=\"border-radius: 50%;\" /\u003e\n\u003c/td\u003e\n\u003ctd\u003e\n\u003cstrong\u003ePouria Rouzrokh, MD, MPH, MHPE\u003c/strong\u003e\u003cbr\u003e\nMedical Practitioner and Machine Learning Engineer\u003cbr\u003e\nIncoming Radiology Resident @Yale University\u003cbr\u003e\nFormer Data Scientist @Mayo Clinic AI Lab\u003cbr\u003e\n\u003ca href=\"https://x.com/prouzrokh\"\u003e\n  \u003cimg src=\"https://img.shields.io/twitter/follow/prouzrokh?style=social\" alt=\"Twitter Follow\" /\u003e\n\u003c/a\u003e\n\u003ca href=\"https://linkedin.com/in/pouria-rouzrokh\"\u003e\n  \u003cimg src=\"https://img.shields.io/badge/LinkedIn-Connect-blue\" alt=\"LinkedIn\" /\u003e\n\u003c/a\u003e\n\u003ca href=\"https://scholar.google.com/citations?user=Ksv9I0sAAAAJ\u0026hl=en\"\u003e\n  \u003cimg src=\"https://img.shields.io/badge/Google%20Scholar-Profile-green\" alt=\"Google Scholar\" /\u003e\n\u003c/a\u003e\n\u003ca href=\"https://github.com/PouriaRouzrokh\"\u003e\n  \u003cimg src=\"https://img.shields.io/badge/GitHub-Profile-black\" alt=\"GitHub\" /\u003e\n\u003c/a\u003e\n\u003ca href=\"mailto:po.rouzrokh@gmail.com\"\u003e\n  \u003cimg src=\"https://img.shields.io/badge/Email-Contact-red\" alt=\"Email\" /\u003e\n\u003c/a\u003e\n\u003c/td\u003e\n\u003c/tr\u003e\n\u003c/table\u003e\n\n## ❤️ Support LatteReview\n\nIf you find LatteReview helpful in your research or work, consider supporting its continued development. Since we're already sharing a virtual coffee break while reviewing papers, maybe you'd like to treat me to a real one? ☕ 😊\n\n### Ways to Support:\n\n- [Become my sponsor](https://github.com/sponsors/PouriaRouzrokh) on GitHub\n- [Treat me to a cup of coffee](http://ko-fi.com/pouriarouzrokh) on Ko-fi ☕\n- [Star the repository](https://github.com/PouriaRouzrokh/LatteReview) to help others discover the project\n- Submit bug reports, feature requests, or contribute code\n- Share your experience using LatteReview in your research\n\n## 📜 License\n\nThis work is licensed under a Creative Commons Attribution-NonCommercial 4.0 International License.  \nTo view a copy of this license, visit [LICENSE](http://creativecommons.org/licenses/by-nc/4.0/).\n\n## 🤝 Contributing\n\nI welcome contributions! Please feel free to submit a Pull Request.\n\n## Acknowledgement\n\nI would like to express my heartfelt gratitude to [Moein Shariatnia](https://github.com/moein-shariatnia) for his invaluable support and contributions to this project.\n\n## 📚 Citation\n\nIf you use LatteReview in your research, please cite our paper:\n\n```bibtex\n@misc{rouzrokh2025lattereview,\n    title={LatteReview: A Multi-Agent Framework for Systematic Review Automation Using Large Language Models},\n    author={Pouria Rouzrokh and Moein Shariatnia},\n    year={2025},\n    eprint={2501.05468},\n    archivePrefix={arXiv},\n    primaryClass={cs.CL}\n}\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FPouriaRouzrokh%2FLatteReview","html_url":"https://awesome.ecosyste.ms/projects/github.com%2FPouriaRouzrokh%2FLatteReview","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FPouriaRouzrokh%2FLatteReview/lists"}