{"id":27984502,"url":"https://github.com/automix-llm/automix","last_synced_at":"2025-05-08T05:02:03.099Z","repository":{"id":202085871,"uuid":"702704921","full_name":"automix-llm/automix","owner":"automix-llm","description":"Mixing Language Models with Self-Verification and Meta-Verification","archived":false,"fork":false,"pushed_at":"2024-12-12T22:03:03.000Z","size":2368,"stargazers_count":97,"open_issues_count":0,"forks_count":8,"subscribers_count":3,"default_branch":"main","last_synced_at":"2024-12-12T23:18:01.768Z","etag":null,"topics":["few-shot-learning","large-language-models","model-selection","prompting","question-answering"],"latest_commit_sha":null,"homepage":"","language":"Jupyter Notebook","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/automix-llm.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2023-10-09T20:53:55.000Z","updated_at":"2024-12-12T22:03:07.000Z","dependencies_parsed_at":"2023-10-22T12:28:35.102Z","dependency_job_id":"70a90e7a-9878-4617-8d92-fd89c46b248b","html_url":"https://github.com/automix-llm/automix","commit_stats":null,"previous_names":["automix-llm/automix"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/automix-llm%2Fautomix","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/automix-llm%2Fautomix/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/automix-llm%2Fautomix/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/automix-llm%2Fautomix/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/automix-llm","download_url":"https://codeload.github.com/automix-llm/automix/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":253002857,"owners_count":21838640,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["few-shot-learning","large-language-models","model-selection","prompting","question-answering"],"created_at":"2025-05-08T05:02:00.359Z","updated_at":"2025-05-08T05:02:03.084Z","avatar_url":"https://github.com/automix-llm.png","language":"Jupyter Notebook","funding_links":[],"categories":["2.3 Ensemble After Inference","Jupyter Notebook"],"sub_categories":["2.3.2 (c,2) Cascade"],"readme":"\u003cdiv align=\"center\"\u003e\n    \u003ch1\u003e AutoMix: Automatically Mixing Language Models\u003c/h1\u003e\n    \u003c!-- \u003cp\u003e\u003ci\u003eAuto Solve Google's Recaptcha Image Challenge with near human performance!\u003c/i\u003e\u003c/p\u003e --\u003e\n    \u003ca href=\"https://automix-llm.github.io/automix/\"\u003e\u003cimg src=\"https://img.shields.io/website?down_message=down\u0026style=for-the-badge\u0026up_message=up\u0026url=https%3A%2F%2Fautomix-llm.github.io/automix/\"\u003e\u003c/a\u003e\n\u003ca href=\"https://arxiv.org/abs/2310.12963\"\u003e\u003cimg src=\"https://img.shields.io/badge/arXiv-2310.12963-red.svg?style=for-the-badge\"\u003e\u003c/a\u003e\n    \u003ca href=\"https://pypi.org/project/automix-llm/\"\u003e\u003cimg src=\"https://img.shields.io/pypi/v/automix-llm?style=for-the-badge\"\u003e\u003c/a\u003e\n\t\u003cbr\u003e\n\t\u003cbr\u003e\n\u003c/div\u003e\n\n\u003c!-- # AutoMix: Automatically Mixing Language Models --\u003e\n\n\u003c!-- [Arxiv Paper](https://arxiv.org/abs/2310.12963) --\u003e\n\n\u003cdiv align=\"center\"\u003e\n    \u003cimg src=\"https://github.com/automix-llm/automix/assets/1304693/a81ba101-247d-4fa7-8dc3-989dd5884483\" width=\"500\"\u003e\n\u003c/div\u003e\n\n\n## What is AutoMix?\n\nThe idea behind AutoMix is simple: \n\n1. Send a query to small language model (SLM), gets a noisy label on its correctness using **few-shot self-verification** done with the same model (SLM).\n\n2. Use a meta-verifier to _double check_ verifier's output, and route the query to a larger language model (LLM) if needed.\n\n\n\n\u003cdiv align=\"center\"\u003e\n    \u003cimg src=\"https://github.com/automix-llm/automix/assets/1304693/3043c5b1-08b0-42dd-9180-ca13f45a83eb\" width=\"850\"\u003e\n\u003c/div\u003e\n\n\n\n## Self-Verification and Meta-verification\n\n\nAt the center of automix is the idea of context-grounded self-verification:\n\n\n\u003cdiv align=\"center\"\u003e\n    \u003cimg src=\"https://github.com/automix-llm/automix/assets/1304693/ff187307-9366-471f-ab59-9204c177f711\" width=\"850\"\u003e\n\u003c/div\u003e\n\n- However, such verification can often be noisy, so we introduce an additional layer of meta-verification using [POMDPs](https://www.pomdp.org/) or thresholding.\n\n\n## Notebooks\n\n### Running inference\n\n- [**Step1 Run inference to solve tasks**](https://github.com/automix-llm/automix/blob/main/colabs/Step1_SolveQueries.ipynb) - Task prompts, code to run inference from different language models.\n   [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/automix-llm/automix/blob/main/colabs/Step1_SolveQueries.ipynb)\n\n### Few-shot self-verification\n\n- [**Step2 Self Verify**](https://github.com/automix-llm/automix/blob/main/colabs/Step2_SelfVerify.ipynb) - Verification prompts, code to run verification on the outputs produced in step 1.\n   [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/automix-llm/automix/blob/main/colabs/Step2_SelfVerify.ipynb)\n\n\n\n### Meta-verification\n\n- [**Step3 Meta Verify**](https://github.com/automix-llm/automix/blob/main/colabs/Step3_MetaVerify.ipynb) - Run meta-verification using different AutoMix methods on outputs produced from Step 2. \n   [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/automix-llm/automix/blob/main/colabs/Step3_MetaVerify.ipynb)\n\n- You can run `pip install automix-llm' to use the meta-verifier system wide.\n\n\n\n### Replicating the results\n\n- To replicate the results in the paper, please run `python scripts paper_results.py`\n\n\n## Data and Outputs\n\n\n- We experiment with 5 datasets: CNLI, CoQA, NarrativeQA, QASPER, and Quality.\n\n\n- Note: The dataset are sourced from [scrolls](https://www.scrolls-benchmark.com/). Please cite scrolls and the appropriate sources if you use these datasets. We are making them available in a sinlge jsonl file for ease of use and reproducibility. For details on how CoQa was prepared, please see [**Preparing COQA**](https://github.com/automix-llm/automix/blob/main/colabs/Preparing_COQA.ipynb).\n   \n\n- **Inputs:** All input data for the AutoMix project is provided in `automix_inputs.jsonl`. You can access and download it directly from [Google Drive](https://drive.google.com/file/d/1dhyt7UuYumk9Gae9eJ_mpTVrLeSTuRht/view?usp=sharing).\n\n- **Outputs from LLAMA2:** The outputs generated using the LLAMA2 model are stored in `automix_llama2_outputs.jsonl`, available alongside the input file in the linked Google Drive.\n\n```\nid: A unique identifier for each question and answer pair.\npid: An additional identifier potentially mapping to specific instances or model variants.\nbase_ctx: The context.\nquestion: Input question or query.\noutput: Ground truth.\ndataset: .\nllama13b_pred_ans: The answer generated by the llama13b model.\nllama70b_pred_ans: The answer generated by the llama70b model.\nllama13b_ver: Verification outputs of the llama13b model’s answers.\n```\n\n### Stats\n\n```txt\n--------------------------------\n| Dataset      | Split | Count |\n|--------------|-------|-------|\n| cnli         | train | 7191  |\n|              | val   | 1037  |\n| coqa         | train | 3941  |\n|              | val   | 3908  |\n| narrative_qa | train | 9946  |\n|              | val   | 5826  |\n| qasper       | train | 2556  |\n|              | val   | 1715  |\n| quality      | train | 2515  |\n|              | val   | 2085  |\n--------------------------------\nName: split, dtype: int64\n```\n\n\n\n\n## Citation\n\n```\n@misc{madaan2023automix,\n      title={AutoMix: Automatically Mixing Language Models}, \n      author={Aman Madaan and Pranjal Aggarwal and Ankit Anand and Srividya Pranavi Potharaju and Swaroop Mishra and Pei Zhou and Aditya Gupta and Dheeraj Rajagopal and Karthik Kappaganthu and Yiming Yang and Shyam Upadhyay and Mausam and Manaal Faruqui},\n      year={2023},\n      eprint={2310.12963},\n      archivePrefix={arXiv},\n      primaryClass={cs.CL}\n}\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fautomix-llm%2Fautomix","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fautomix-llm%2Fautomix","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fautomix-llm%2Fautomix/lists"}