{"id":13754271,"url":"https://github.com/shijx12/TransferNet","last_synced_at":"2025-05-09T22:31:40.352Z","repository":{"id":42054292,"uuid":"358166636","full_name":"shijx12/TransferNet","owner":"shijx12","description":"Pytorch implementation of EMNLP 2021 paper \"TransferNet: An Effective and Transparent Framework for Multi-hop Question Answering over Relation Graph \"","archived":false,"fork":false,"pushed_at":"2023-06-12T07:21:56.000Z","size":359,"stargazers_count":63,"open_issues_count":15,"forks_count":18,"subscribers_count":2,"default_branch":"master","last_synced_at":"2024-11-16T07:33:17.436Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/shijx12.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2021-04-15T07:26:18.000Z","updated_at":"2024-11-10T14:18:39.000Z","dependencies_parsed_at":"2024-08-03T09:17:18.269Z","dependency_job_id":null,"html_url":"https://github.com/shijx12/TransferNet","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/shijx12%2FTransferNet","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/shijx12%2FTransferNet/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/shijx12%2FTransferNet/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/shijx12%2FTransferNet/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/shijx12","download_url":"https://codeload.github.com/shijx12/TransferNet/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":253335720,"owners_count":21892718,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-08-03T09:01:52.771Z","updated_at":"2025-05-09T22:31:37.297Z","avatar_url":"https://github.com/shijx12.png","language":"Python","funding_links":[],"categories":["知识图谱问答KBQA、多跳推理"],"sub_categories":["其他_文本生成、文本对话"],"readme":"# TransferNet\n\nPytorch implementation of EMNLP 2021 paper \n\n**[TransferNet: An Effective and Transparent Framework for Multi-hop Question Answering over Relation Graph](https://arxiv.org/abs/2104.07302)**\n\u003cbr\u003e\n[Jiaxin Shi](https://shijx12.github.io), Shulin Cao, Lei Hou, [Juanzi Li](http://keg.cs.tsinghua.edu.cn/persons/ljz/), [Hanwang Zhang](http://www.ntu.edu.sg/home/hanwangzhang/#aboutme)\n\nWe perform transparent multi-hop reasoning over relation graphs of label form (i.e., knowledge graph) and text form. This is an example:\n\n\u003cdiv align=\"center\"\u003e\n    \u003cimg src=\"example.png\" width=\"60%\"\u003e\n\u003c/div\u003e\u003cbr/\u003e\n\nIf you find this code useful in your research, please cite\n``` tex\n@inproceedings{shi2021transfernet,\n  title={TransferNet: An Effective and Transparent Framework for Multi-hop Question Answering over Relation Graph},\n  author={Jiaxin Shi, Shulin Cao, Lei Hou, Juanzi Li, Hanwang Zhang},\n  booktitle={EMNLP},\n  year={2021}\n}\n```\n\n## dependencies\n- pytorch\u003e=1.2.0\n- [transformers](https://github.com/huggingface/transformers)\n- tqdm\n- nltk\n- shutil\n\n## Prepare Datasets\n- [MetaQA](https://goo.gl/f3AmcY), we only use its vanilla version.\n- [MovieQA](http://www.thespermwhale.com/jaseweston/babi/movieqa.tar.gz), we need its `knowledge_source/wiki.txt` as the text corpus for our MetaQA-Text experiments. Copy the file into the folder of MetaQA, and put it together with `kb.txt`. The files of MetaQA should be something like\n```shell\nMetaQA\n+-- kb\n|   +-- kb.txt\n|   +-- wiki.txt\n+-- 1-hop\n|   +-- vanilla\n|   |   +-- qa_train.txt\n|   |   +-- qa_dev.txt\n|   |   +-- qa_test.txt\n+-- 2-hop\n+-- 3-hop\n```\n- [WebQSP](https://drive.google.com/drive/folders/1RlqGBMo45lTmWz9MUPTq-0KcjSd3ujxc?usp=sharing), which has been processed by [EmbedKGQA](https://github.com/malllabiisc/EmbedKGQA).\n- [ComplexWebQuestions](https://drive.google.com/file/d/1ua7h88kJ6dECih6uumLeOIV9a3QNdP-g/view?usp=sharing), which has been processed by [NSM](https://github.com/RichardHGL/WSDM2021_NSM).\n- [GloVe 300d pretrained vector](http://nlp.stanford.edu/data/glove.840B.300d.zip), which is used in the BiGRU model. After unzipping it, you need to convert the txt file to pickle file by \n``` shell\npython pickle_glove.py --txt \u003c/path/to/840B.300d.txt\u003e --pt \u003c/output/file/name\u003e\n```\n\n\n## Experiments\n\n### MetaQA-KB\n\n1. Preprocess\n```shell\npython -m MetaQA-KB.preprocess --input_dir \u003cPATH/TO/METAQA\u003e --output_dir \u003cPATH/TO/PROCESSED/FILES\u003e\n```\n\n2. Train\n```shell\npython -m MetaQA-KB.train --glove_pt \u003cPATH/TO/GLOVE/PICKLE\u003e --input_dir \u003cPATH/TO/PROCESSED/FILES\u003e --save_dir \u003cPATH/TO/CHECKPOINT\u003e\n```\n\n3. Predict on the test set\n```shell\npython -m MetaQA-KB.predict --input_dir \u003cPATH/TO/PROCESSED/FILES\u003e --ckpt \u003cPATH/TO/CHECKPOINT\u003e --mode test\n```\n\n4. Visualize the reasoning process. It will enter an IPython environment after showing the information of each sample. You can print more variables that you are insterested in. To stop the process, you need to quit the IPython by `Ctrl+D` and then kill the loop by `Ctrl+C` immediately.\n```shell\npython -m MetaQA-KB.predict --input_dir \u003cPATH/TO/PROCESSED/FILES\u003e --ckpt \u003cPATH/TO/CHECKPOINT\u003e --mode vis\n```\n\n### MetaQA-Text\n\n1. Preprocess\n```shell\npython -m MetaQA-Text.preprocess --input_dir \u003cPATH/TO/METAQA\u003e --output_dir \u003cPATH/TO/PROCESSED/FILES\u003e\n```\n\n2. Train\n```shell\npython -m MetaQA-Text.train --glove_pt \u003cPATH/TO/GLOVE/PICKLE\u003e --input_dir \u003cPATH/TO/PROCESSED/FILES\u003e --save_dir \u003cPATH/TO/CHECKPOINT\u003e\n```\n\nThe scripts for inference and visualization are the same as **MetaQA-KB**. Just change the python module to `MetaQA-Text.predict`.\n\n\n### MetaQA-Text + 50% KB\n\n1. Preprocess\n```shell\npython -m MetaQA-Text.preprocess --input_dir \u003cPATH/TO/METAQA\u003e --output_dir \u003cPATH/TO/PROCESSED/FILES\u003e --kb_ratio 0.5\n```\n\n2. Train, it needs more active paths than MetaQA-Text\n```shell\npython -m MetaQA-Text.train --input_dir \u003cPATH/TO/PROCESSED/FILES\u003e --save_dir \u003cPATH/TO/CHECKPOINT\u003e --max_active 800 --batch_size 32\n```\n\nThe scripts for inference and visualization are the same as **MetaQA-Text**.\n\n\n### WebQSP\nWebQSP does not need preprocess. We can directly start the training:\n\n```shell\npython -m WebQSP.train --input_dir \u003cPATH/TO/UNZIPPED/DATA\u003e --save_dir \u003cPATH/TO/CHECKPOINT\u003e\n```\n\n\n### ComplexWebQuestions\nSimilar to WebQSP, CWQ does not need preprocess. We can directly start the training:\n\n```shell\npython -m CompWebQ.train --input_dir \u003cPATH/TO/UNZIPPED/DATA\u003e --save_dir \u003cPATH/TO/CHECKPOINT\u003e\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fshijx12%2FTransferNet","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fshijx12%2FTransferNet","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fshijx12%2FTransferNet/lists"}