{"id":21392244,"url":"https://github.com/akariasai/learning_to_retrieve_reasoning_paths","last_synced_at":"2025-10-11T23:40:45.718Z","repository":{"id":40947489,"uuid":"238331979","full_name":"AkariAsai/learning_to_retrieve_reasoning_paths","owner":"AkariAsai","description":"The official implementation of ICLR 2020, \"Learning to Retrieve Reasoning Paths over Wikipedia Graph for Question Answering\".","archived":false,"fork":false,"pushed_at":"2024-07-25T10:57:30.000Z","size":345,"stargazers_count":432,"open_issues_count":5,"forks_count":63,"subscribers_count":18,"default_branch":"master","last_synced_at":"2025-03-29T01:15:13.796Z","etag":null,"topics":["hotpotqa","multi-hop-reasoning","natural-questions","open-domain-qa","reading-comprehension","retrieval","squad"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/AkariAsai.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2020-02-04T23:58:00.000Z","updated_at":"2025-03-27T22:33:41.000Z","dependencies_parsed_at":"2024-12-13T11:11:02.412Z","dependency_job_id":"3962f67e-4235-4197-942a-f1839252735e","html_url":"https://github.com/AkariAsai/learning_to_retrieve_reasoning_paths","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/AkariAsai%2Flearning_to_retrieve_reasoning_paths","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/AkariAsai%2Flearning_to_retrieve_reasoning_paths/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/AkariAsai%2Flearning_to_retrieve_reasoning_paths/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/AkariAsai%2Flearning_to_retrieve_reasoning_paths/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/AkariAsai","download_url":"https://codeload.github.com/AkariAsai/learning_to_retrieve_reasoning_paths/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":247276189,"owners_count":20912288,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["hotpotqa","multi-hop-reasoning","natural-questions","open-domain-qa","reading-comprehension","retrieval","squad"],"created_at":"2024-11-22T13:39:42.321Z","updated_at":"2025-10-11T23:40:40.698Z","avatar_url":"https://github.com/AkariAsai.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Learning to Retrieve Reasoning Paths over Wikipedia Graph for Question Answering\n\u003cp align=\"center\"\u003e\u003cimg width=\"80%\" src=\"img/odqa_overview-1.png\" /\u003e\u003c/p\u003e\n\nThis is the official implementation of the following paper:    \nAkari Asai, Kazuma Hashimoto, Hannaneh Hajishirzi, Richard Socher, Caiming Xiong. [Learning to Retrieve Reasoning Paths over Wikipedia Graph for Question Answering](https://arxiv.org/abs/1911.10470). In: Proceedings of ICLR. 2020\n\nIn the paper, we introduce a graph-based retriever-reader framework that learns to retrieve reasoning paths (a reasoning path = a chain of multiple paragraphs to answer multi-hop questions) from English Wikipedia using its graphical structure, and further verify and extract answers from the selected reasoning paths. Our experimental results show state-of-the-art results across three diverse open-domain QA datasets: [HotpotQA (full wiki)](https://hotpotqa.github.io/), [Natural Questions](https://ai.google.com/research/NaturalQuestions/) Open, [SQuAD](https://rajpurkar.github.io/SQuAD-explorer/) Open. \n\n*Acknowledgements*: To implement our BERT-based modules, we used the [huggingface's transformers](https://huggingface.co/transformers/) library. The implementation of TF-IDF based document ranker and splitter started from the [DrQA](https://github.com/facebookresearch/DrQA) and [document-qa](https://github.com/allenai/document-qa) repositories. Huge thanks to the contributors of those amazing repositories!\n\n\n## Quick Links\n0. [Quick Run on HotpotQA](#0-quick-run-on-hotpotqa)\n1. [Installation](#1-installation)\n2. [Train](#2-train)\n3. [Evaluation](#3-evaluation)\n4. [Interactive Demo](#4-interactive-demo)\n5. [Others](#5-others)\n6. [Citation and Contact](#citation-and-contact)\n\n## 0. Quick Run on HotpotQA\nWe provide [quick_start_hotpot.sh](quick_start_hotpot.sh), with which you can easily set up and run evaluation on HotpotQA full wiki (on the first 100 questions).\n\nThe script will\n1. download our trained models and evaluation data (See [Installation](#1-installation) for the details), \n2. run the whole pipeline on the evaluation data (See [Evaluation](#3-evaluation)), and \n3. calculate the QA scores and supporting facts scores. \n\nThe evaluation will give us the following results:\n```\n{'em': 0.6, 'f1': 0.7468968253968253, 'prec': 0.754030303030303, 'recall': 0.7651666666666667, 'sp_em': 0.49, 'sp_f1': 0.7769365079365077, 'sp_prec': 0.8275, 'sp_recall': 0.7488333333333332, 'joint_em': 0.33, 'joint_f1': 0.6249458756180065, 'joint_prec': 0.6706212121212122, 'joint_recall': 0.6154999999999999}\n```\n\nWanna try your own open-domain question? See [Interactive Demo](#4-interactive-demo)! Once you run the [quick_start_hotpot.sh](quick_start_hotpot.sh), you can easily switch to the demo mode by changing some options in the command. \n\n## 1. Installation \n### Requirements\n\nOur framework requires Python 3.5 or higher. We do not support Python 2.X.     \n\nIt also requires installing [pytorch-pretrained-bert version (version 0.6.2)](https://github.com/huggingface/transformers/tree/v0.6.2) and [PyTorch](http://pytorch.org/) version 1.0 or higher. The other dependencies are listed in [requirements.txt](requirements.txt).   \nWe are planning to [migrate from pytorch-pretrained-bert](https://huggingface.co/transformers/migration.html) to transformers soon. \n\n### Set up \nRun the following commands to clone the repository and install our framework:\n\n```bash\ngit clone https://github.com/AkariAsai/learning_to_retrieve_reasoning_paths.git\ncd learning_to_retrieve_reasoning_paths\npip install -r requirements.txt\n```\n\n### Downloading trained models\nAll the trained models used in our paper for the three datasets are available in google drive: \n - HotpotQA full wiki: [hotpot_models.zip](https://drive.google.com/open?id=1ra37xtEXSROG_f90XxR4kgElGJWUHQyM)\n - Natural Questions Open: [nq_models.zip](https://drive.google.com/open?id=120JNI49nK-W014cjneuXJuUC09To3KeQ)\n - SQuAD Open: [squad_models.zip](https://drive.google.com/open?id=1_z54KceuYXnA0fJvdASDubti9Zb8aAR4): for SQuAD Open, please download `.db` and `.npz` files following [DrQA](https://github.com/facebookresearch/DrQA/blob/master/download.sh) repository.\n\nAlternatively, you can download a zip file containing all models by using [gdown](https://pypi.org/project/gdown/).\n\ne.g., download HotpotQA models\n```bash\nmkdir models\ncd models\ngdown https://drive.google.com/uc?id=1ra37xtEXSROG_f90XxR4kgElGJWUHQyM\nunzip hotpot_models.zip\nrm hotpot_models.zip\ncd ..\n```\n**Note: the size of the zip file is about 4GB for HotpotQA models, and once it is extracted, the total size of the models is more than 8GB (including the introductory paragraph only Wikipedia database). The `nq_models.zip` include full Wikipedia database, which is around 30GB once extracted.**\n\n\n### Downloading data\n#### for training\n- You can download all of the training datasets from [here (google drive)](https://drive.google.com/drive/folders/1nYQOtoxJiiL5XK6PHOeluTWgh3PTOCVD?usp=sharing). \n- We create (1) data to train graph-based retriever, and (2) data to train reader by augmenting the publicly available machine reading comprehension datasets (HotpotQA, SQuAD and Natural Questions). \nSee the details of the process in Section 3.1.2 and Section 3.2 in [our paper](https://arxiv.org/pdf/1911.10470.pdf). \n\n#### for evaluation \n- Following previous work such as [DrQA](https://github.com/shmsw25/qa-hard-em) or [qa-hard-em](https://github.com/shmsw25/qa-hard-em), we convert the original machine reading comprehension datasets to sets of question and answer pairs. You can download our preprocessed data from [here](https://drive.google.com/open?id=1na7vxYWadK2kS2aqg88RDMFir8PP-2lS).\n\n- For HotpotQA, we only use question-answer pairs as input, but we need to use the original HotpotQA development set (either fullwiki or distractor) to evaluate supporting fact evaluations from [HotpotQA's website](https://hotpotqa.github.io/). \n\n```bash\nmkdir data\ncd data\nmkdir hotpot\ncd hotpot\ngdown https://drive.google.com/uc?id=1m_7ZJtWQsZ8qDqtItDTWYlsEHDeVHbPt # download preprocessed full wiki data\nwget http://curtis.ml.cmu.edu/datasets/hotpot/hotpot_dev_fullwiki_v1.json # download the original full wiki data for sp evaluation. \ncd ../..\n```\n\n## 2. Train\nIn this work, we use a two-stage training approach, which lets you train the reader and retriever independently and easily switch to new reader models. \nThe details of the training process can be seen in the README files in [graph_retriever](graph_retriever), [reader](reader) and [sequence_sentence_selector](sequential_sentence_selector).\n\nYou can download our pre-trained models from the link mentioned above. \n\n## 3. Evaluation\nAfter downloading a TF-IDF retriever, training a graph-retriever and reader models, you can test the performance of our entire system.\n\n\n#### HotpotQA\nIf you set up using `quick_start_hotpot.sh`, you can run full evaluation by setting the `--eval_file_path` option to `data/hotpot/hotpot_fullwiki_first_100.jsonl` . \n\n```bash\npython eval_main.py \\\n--eval_file_path data/hotpot/hotpot_fullwiki_data.jsonl \\\n--eval_file_path_sp data/hotpot/hotpot_dev_distractor_v1.json \\\n--graph_retriever_path models/hotpot_models/graph_retriever_path/pytorch_model.bin \\\n--reader_path models/hotpot_models/reader \\\n--sequential_sentence_selector_path models/hotpot_models/sequential_sentence_selector/pytorch_model.bin \\\n--tfidf_path models/hotpot_models/tfidf_retriever/wiki_open_full_new_db_intro_only-tfidf-ngram=2-hash=16777216-tokenizer=simple.npz \\\n--db_path models/hotpot_models/wiki_db/wiki_abst_only_hotpotqa_w_original_title.db \\\n--bert_model_sequential_sentence_selector bert-large-uncased --do_lower_case \\\n--tfidf_limit 500 --eval_batch_size 4 --pruning_by_links --beam_graph_retriever 8 \\\n--beam_sequential_sentence_selector 8 --max_para_num 2000 --sp_eval\n```\n\nThe evaluation will give us the following results (equivalent to our reported results):\n```\n{'em': 0.6049966239027684, 'f1': 0.7330873757783022, 'prec': 0.7613180885780131, 'recall': 0.7421444532461545, 'sp_em': 0.49169480081026334, 'sp_f1': 0.7605390258327606, 'sp_prec': 0.8103758721584524, 'sp_recall': 0.7325846435805953, 'joint_em': 0.35827143821742063, 'joint_f1': 0.6143774960171196, 'joint_prec': 0.679462464277477, 'joint_recall': 0.5987834193329556}\n```\n\n#### SQuAD Open\n\n```bash\npython eval_main.py \\\n--eval_file_path data/squad/squad_open_domain_data.jsonl \\\n--graph_retriever_path models/squad_models/selector/pytorch_model.bin \\\n--reader_path models/squad_models/reader \\\n--tfidf_path DrQA/data/wikipedia/docs-tfidf-ngram=2-hash=16777216-tokenizer=simple.npz \\\n--db_path DrQA/data/wikipedia/docs.db \\\n--bert_model bert-base-uncased --do_lower_case \\\n--tfidf_limit 50 --eval_batch_size 4 \\\n--beam_graph_retriever 8 --max_para_num 2000 --use_full_article \n```\n\n#### Natural Questions\n\n```\npython eval_main.py \\\n--eval_file_path data/nq_open_domain_data.jsonl \\\n--graph_retriever_path models/nq/selector/pytorch_model.bin --reader_path models/nq/reader/ \\\n--tfidf_path models/nq_models/tfidf_retriever/wiki_20181220_nq_hyper_linked-tfidf-ngram=2-hash=16777216-tokenizer=simple.npz \\\n--db_path models/nq_models/wiki_db/wiki_20181220_nq_hyper_linked.db \\\n--bert_model bert-base-uncased --do_lower_case --tfidf_limit 20 --eval_batch_size 4 --pruning_by_links \\\n--beam_graph_retriever 8 --max_para_num 2000 --use_full_article \n```\n\n#### (optional) Using TagMe for initial retrieval\nAs mentioned in Appendix B.7 in our paper, you can optionally use an entity linking system ([TagMe](https://sobigdata.d4science.org/web/tagme/tagme-help)) for the initial retrieval.   \n\nTo uee TagMe, \n1. [register](https://services.d4science.org/group/d4science-services-gateway/explore?siteId=22994) to get API key, and \n3. set the API key via `--tagme_api_key` option, and set `--tagme` option true.  \n\n```\npython eval_main.py \\\n--eval_file_path data/nq_open_domain_data.jsonl \\\n--graph_retriever_path models/nq/selector/pytorch_model.bin --reader_path models/nq/reader/ \\\n--tfidf_path models/nq_models/tfidf_retriever/wiki_20181220_nq_hyper_linked-tfidf-ngram=2-hash=16777216-tokenizer=simple.npz \\\n--db_path models/nq_models/wiki_db/wiki_20181220_nq_hyper_linked.db \\\n--bert_model bert-base-uncased --do_lower_case --tfidf_limit 20 --eval_batch_size 4 --pruning_by_links --beam 8 --max_para_num 2000 --use_full_article --tagme --tagme_api_key YOUR_API_KEY\n```\n\n*The implementation of the two-step TF-IDF retrieval module (article retrieval --\u003e paragraph-level re-ranking) for Natural Questions is currently in progress, which might give slightly lower scores than the reported results in our paper. We'll fix the issue soon.*\n\n## 4. Interactive demo\nYou could run interactive demo and ask open-domain questions. Our model answers the question with supporting facts. \n\nIf you set up using `quick_start.sh` script, you can run full evaluation by changing the script name to from `eval_main.py` to `demo.py`, and removing `--eval_file_path` and `--eval_file_path_sp` options. \n\ne.g.,\n```bash\npython demo.py \\\n--graph_retriever_path models/hotpot_models/graph_retriever_path/pytorch_model.bin \\\n--reader_path models/hotpot_models/reader \\\n--sequential_sentence_selector_path models/hotpot_models/sequential_sentence_selector/pytorch_model.bin \\\n--tfidf_path models/hotpot_models/tfidf_retriever/wiki_open_full_new_db_intro_only-tfidf-ngram=2-hash=16777216-tokenizer=simple.npz \\\n--db_path models/hotpot_models/wiki_db/wiki_abst_only_hotpotqa_w_original_title.db \\\n--do_lower_case --beam 4 --quiet --max_para_num 200 \\\n--tfidf_limit 20 --pruning_by_links \\\n```\n\nAn output example is as follows:\n```\n#### Reader results ####\n[    {\n        \"q_id\": \"DEMO_0\",\n        \"question\": \"Bordan Tkachuk was the CEO of a company that provides what sort of products?\",\n        \"answer\": \"IT products and services\",\n        \"context\": [\n            \"Cintas_0\",\n            \"Bordan Tkachuk_0\",\n            \"Viglen_0\"\n        ]\n    }\n]\n\n#### Supporting facts ####\n[\n    {\n        \"q_id\": \"DEMO_0\",\n        \"supporting facts\": {\n            \"Viglen_0\": [\n                [0, \"Viglen Ltd provides IT products and services, including storage systems, servers, workstations and data/voice communications equipment and services.\"\n                ]\n            ],\n            \"Bordan Tkachuk_0\": [\n                [0, \"Bordan Tkachuk ( ) is a British business executive, the former CEO of Viglen, also known from his appearances on the BBC-produced British version of \\\"The Apprentice,\\\" interviewing for his boss Lord Sugar.\"\n                ]\n            ]\n        }\n    }\n]\n\n```\n\n\n## 5. Others\n### Distant supervision \u0026 negative examples data generation\nIn this work, we augment the original MRC data with negative and distant supervision examples to make our retriever and reader robust to inference time noise. Our experimental results show these training strategy gives significant performance improvements.\n\nAll of the training data is available [here (google drive)](https://drive.google.com/drive/u/1/folders/1nYQOtoxJiiL5XK6PHOeluTWgh3PTOCVD).\n\n*We are planning to release our codes to augment training data with negative examples and distant examples to guide future research in open-domain QA fields. Please stay tuned!*\n\n### Dataset format \nFor quick experiments and detailed human analysis, we save intermediate results for each step: original Q-A pair (format A), TF-IDF retrieval (format B), our graph-based (format C) retriever. \n\n#### Format A (eval data, the input of TF-IDF retriever)\nFor the evaluation pipeline, our initial input is a simple `jsonlines` format where each line contains one example with `id = [str]`, `question = [str]` and `answer = List[str]` (or `answers = List[str]` for datasets where multiple answers are annotated for each question) information. \n\nFor SQuAD Open and HotpotQA fullwiki, you can download the preprocessed format A files from [here](https://drive.google.com/file/d/1f3YtvImDxB9h6GuVGFelzxvgwEcqxuHn/view?usp=sharing).\n\ne.g., HotpotQA fullwiki dev\n```\n{\n\"id\": \"5ab3b0bf5542992ade7c6e39\", \n\"question\": \"What year did Guns N Roses perform a promo for a movie starring Arnold Schwarzenegger \nas a former New York Police detective?\", \n\"answer\": [\"1999\"]\n}\n```\n\ne.g., SQuAD Open dev\n```py\n{\n\"id\": \"56beace93aeaaa14008c91e0\", \n\"question\": \"What venue did Super Bowl 50 take place in?\", \n\"answers\": [\"Levi's Stadium\", \"Levi's Stadium\", \n\"Levi's Stadium in the San Francisco Bay Area at Santa Clara\"]\n}\n```\n\n#### Format B (TF-IDF retriever output)\nFor TF-IDF results, we store the data as a list of `JSON`, and each data point contains several information.\n\n- `q_id = [str]`\n- `question = [str]` \n- `answer = List[str]`\n- `context = Dict[str, str]`: Top $N$ paragraphs which are ranked high by our TF-IDF retriever. \n- `all_linked_para_title_dic = Dict[str, List[str]]`: Hyper-linked paragraphs' titles from paragraphs in `context`. \n- `all_linked_paras_dic = Dict[str, str]`: the paragraphs of the hyper-linked paragraphs. \n\nFor training data, we have additional items that are used as ground-truth reasoning paths.\n- `short_gold = List[str]`\n- `redundant_gold = List[str]`\n- `all_redundant_gold = List[List[str]]`\n\ne.g., HotpotQA fullwiki dev \n\n```py\n{\n\"question\": 'Were Scott Derrickson and Ed Wood of the same nationality?'.\n\"q_id\": \"5ab3b0bf5542992ade7c6e39\", \n\"context\": \n    {\"Scott Derrickson_0\": \"Scott Derrickson (born July 16, 1966) is an American director,....\", \n      \"Ed Wood'_0\": \"...\", ....}, \n'all_linked_para_title_dic':\n    {\"Scott Derrickson_0\": ['Los Angeles_0', 'California_0', 'Horror film_0', ...]},\n'all_linked_paras_dic': \n    {\"Los Angeles_0\": \"Los Angeles, officially the City of Los Angeles and often known by its initials L.A., is ...\", ...}, \n'short_gold':[], \n'redundant_gold': [],\n'all_redundant_gold': []\n}\n```\n\n#### Format C (Graph-based retriever output)\nThe graph-based retriever's output is a list of `JSON` objects as follows:\n\n- `q_id = [str]`\n- `titles = [str]`: a sequence of titles (the top one reasoning path) \n- `topk_titles = List[List[str]]`: k sequences of titles (the top k reasoning paths).\n- `context = Dict[str, str]`: the paragraphs which are included in top reasoning paths. \n\n```py\n{\n\"q_id\": \"5a713ea95542994082a3e6e4\",\n\"titles\": [\"Alvaro Mexia_0\", \"Boruca_0\"],\n\"topk_titles\": [\n    [\"Alvaro Mexia_0\", \"Boruca_0\"], \n    [\"Alvaro Mexia_0\", \"Indigenous peoples of Florida_0\"], \n    [\"Alvaro Mexia_0\"], \n    [\"List of Ambassadors of Spain to the United States_0\", \"Boruca_0\"], \n    [\"Alvaro Mexia_0\", \"St. Augustine, Florida_0\"], \n    [\"Alvaro Mexia_0\", \"Cape Canaveral, Florida_0\"], \n    [\"Alvaro Mexia_0\", \"Florida_0\"], \n    [\"Parque de la Bombilla (Mexico City)_0\", \"Alvaro Mexia_0\", \"Boruca_0\"]],\n\"context\": {\n    \"Alvaro Mexia_0\": \"Alvaro Mexia was a 17th-century Spanish explorer and cartographer of the east coast of Florida....\",       \"Boruca_0\": \"The Boruca (also known as the Brunca or the Brunka) are an indigenous people living in Costa Rica\"}\n}\n```\n\n\n## Citation and Contact\nIf you find this codebase is useful or use in your work, please cite our paper. \n```\n@inproceedings{\nasai2020learning,\ntitle={Learning to Retrieve Reasoning Paths over Wikipedia Graph for  Question Answering},\nauthor={Akari Asai and Kazuma Hashimoto and Hannaneh Hajishirzi and Richard Socher and Caiming Xiong},\nbooktitle={International Conference on Learning Representations},\nyear={2020}\n}\n```\n\nPlease contact Akari Asai ([@AkariAsai](https://twitter.com/AkariAsai?s=20), akari[at]cs.washington.edu) for questions and suggestions.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fakariasai%2Flearning_to_retrieve_reasoning_paths","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fakariasai%2Flearning_to_retrieve_reasoning_paths","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fakariasai%2Flearning_to_retrieve_reasoning_paths/lists"}