{"id":18816096,"url":"https://github.com/orionw/lm-expansions","last_synced_at":"2025-09-02T19:30:54.795Z","repository":{"id":246714111,"uuid":"751084197","full_name":"orionw/LM-expansions","owner":"orionw","description":"When do Generative Query and Document Expansions Fail? A Comprehensive Study Across Methods, Retrievers, and Datasets","archived":false,"fork":false,"pushed_at":"2024-06-29T22:12:36.000Z","size":69,"stargazers_count":4,"open_issues_count":0,"forks_count":0,"subscribers_count":2,"default_branch":"master","last_synced_at":"2024-11-07T23:53:05.502Z","etag":null,"topics":["information-retrieval","ir","llm","machine-learning","nlp","prompting"],"latest_commit_sha":null,"homepage":"https://arxiv.org/abs/2309.08541","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/orionw.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-01-31T22:29:16.000Z","updated_at":"2024-10-15T00:10:07.000Z","dependencies_parsed_at":null,"dependency_job_id":"dc63e0e9-4f80-4d81-9795-e080a08fffe9","html_url":"https://github.com/orionw/LM-expansions","commit_stats":null,"previous_names":["orionw/lm-expansions"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/orionw%2FLM-expansions","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/orionw%2FLM-expansions/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/orionw%2FLM-expansions/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/orionw%2FLM-expansions/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/orionw","download_url":"https://codeload.github.com/orionw/LM-expansions/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":231804139,"owners_count":18429028,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["information-retrieval","ir","llm","machine-learning","nlp","prompting"],"created_at":"2024-11-07T23:52:26.982Z","updated_at":"2024-12-30T01:36:32.583Z","avatar_url":"https://github.com/orionw.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# When do Generative Query and Document Expansions Fail? A Comprehensive Study Across Methods, Retrievers, and Datasets\n\n[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT)\n\nOfficial repository for the paper [When do Generative Query and Document Expansions Fail? A Comprehensive Study Across Methods, Retrievers, and Datasets](https://aclanthology.org/2024.findings-eacl.134/) including code to reproduce and links to the data generated by the models.\n\n## Table of Contents\n\n- [Overview](#overview)\n- [Data](#data)\n- [Requirements](#requirements)\n- [Setup](#setup)\n- [Reproduce](#reproduce)\n  - [Reproduce Expansions Data](#reproduce-expansions-data)\n  - [Reproduce Model Results Using Expansions](#reproduce-model-results-using-expansions)\n- [License](#license)\n- [Citing](#citing)\n\n## Overview\n\nThis project presents a comprehensive study on generative query and document expansions across various methods, retrievers, and datasets. It aims to identify when these expansions fail and provide insights into improving information retrieval systems.\n\n## Data\n\nThe generations from the models can be found at [orionweller/llm-based-expansions-generations](https://huggingface.co/datasets/orionweller/llm-based-expansions-generations), organized by dataset and expansion type.\n\n## Requirements\n\n- Python 3.10\n- conda\n- OpenAI API key (for using OpenAI models)\n- Together.ai or Anthropic API keys (if using their services)\n- GPU (if using Llama for generation)\n- pyserini (for BM25 results reproduction)\n\n## Setup\n\n1. Clone the repository:\n   ```\n   git clone https://github.com/orionw/LM-expansions.git\n   cd LM-expansions\n   ```\n\n2. Install the correct Python environment:\n   ```\n   conda env create --file=environment.yaml -y \u0026\u0026 conda activate expansions\n   ```\n\n3. Download the local data:\n   ```\n   git clone https://huggingface.co/datasets/orionweller/llm-based-expansions-eval-datasets\n   ```\n\n   This dataset contains local data not available on Huggingface, such as `scifact-refute` and other datasets formatted in a common format. To reproduce the creation of `scifact-refute`, check out `scripts/make_scifact_refute.py`.\n\n## Reproduce\n\n### Reproduce Expansions Data\n\n1. Set up your environment variables (e.g., `OPENAI_API_KEY`) if using OpenAI models.\n\n2. Create or modify a prompt config. Examples are in `prompt_configs/*`. For instance:\n   ```\n   bash generate_expansions.sh scifact_refute prompt_configs/chatgpt_doc2query.jsonl\n   ```\n\n3. Adjust parameters as needed:\n   - `num_examples`: maximum number of instances to predict\n   - `temperature`: controls the randomness of predictions\n\n   Note: If using Together.ai or Anthropic API keys, define them accordingly. For Llama generation, ensure you're using a GPU.\n\n### Reproduce Model Results Using Expansions\n\n1. Run the model using the following command structure:\n   ```\n   bash rerank.sh \u003cdataset name\u003e \u003cname of run\u003e \u003cshard id\u003e \u003cnum shards\u003e \u003cquery expansion path or \"none\"\u003e \u003c\"none\" if not using document expansions otherwise \"replace\" or \"append\" the query with the expansion\u003e \u003cdocument expansion path or \"none\"\u003e \u003c\"none\" if not using query expansions otherwise \"replace\" or \"append\" the query with the expansion\u003e \u003cmodel name\u003e \u003cnumber of queries to run\u003e \u003cnumber of docs to run\u003e\n   ```\n\n   Example:\n   ```\n   bash rerank.sh \"scifact_refute\" \"testing\" 0 1 \"none\" \"none\" \"llm-based-expansions-generations/scifact_refute/expansion_hyde_chatgpt64.jsonl\" \"replace\" \"contriever_msmarco\" 10 100\n   ```\n\n2. Results will be written to `results/\u003cdataset name\u003e/\u003cname of run\u003e/\u003cdataset name\u003e-\u003cname of run\u003e-run.txt`.\n\n3. Evaluate the results:\n   ```\n   bash evaluate.sh scifact_refute testing\n   ```\n\nTo reproduce the top 1000 BM25 results:\n\n1. Install `pyserini` following their [installation docs](https://github.com/castorini/pyserini/blob/master/docs/installation.md).\n\n2. Run the BM25 retrieval:\n   ```\n   bash make_bm25_run.sh \u003cyour folder\u003e \u003cyour dataset name\u003e \u003cdocument id field\u003e \u003cdocument text fields\u003e \u003cquery id field\u003e \u003cquery text fields\u003e\n   ```\n\n   Example:\n   ```\n   bash make_bm25_run.sh bm25 scifact_refute doc_id \"title,text\" query_id text\n   ```\n\n## License\n\nThis project is licensed under the MIT License - see the [LICENSE](LICENSE) file for details.\n\n## Citing\n\nIf you found the code, data or paper useful, please cite:\n\n```bibtex\n@inproceedings{weller-etal-2024-generative,\n    title = \"When do Generative Query and Document Expansions Fail? A Comprehensive Study Across Methods, Retrievers, and Datasets\",\n    author = \"Weller, Orion  and\n      Lo, Kyle  and\n      Wadden, David  and\n      Lawrie, Dawn  and\n      Van Durme, Benjamin  and\n      Cohan, Arman  and\n      Soldaini, Luca\",\n    booktitle = \"Findings of the Association for Computational Linguistics: EACL 2024\",\n    month = mar,\n    year = \"2024\",\n    address = \"St. Julian{'}s, Malta\",\n    publisher = \"Association for Computational Linguistics\",\n    url = \"https://aclanthology.org/2024.findings-eacl.134\",\n    pages = \"1987--2003\",\n}\n```\n\nThis project also built off of many others (see the paper for a full list of references), including code from [TART](https://github.com/facebookresearch/tart/tree/main) and [InPars](https://github.com/zetaalphavector/InPars), please check them and the others out!","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Forionw%2Flm-expansions","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Forionw%2Flm-expansions","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Forionw%2Flm-expansions/lists"}