{"id":13678962,"url":"https://github.com/facebookresearch/KILT","last_synced_at":"2025-04-29T16:30:33.138Z","repository":{"id":42572279,"uuid":"271075510","full_name":"facebookresearch/KILT","owner":"facebookresearch","description":"Library for Knowledge Intensive Language Tasks","archived":true,"fork":false,"pushed_at":"2022-03-31T11:01:39.000Z","size":919,"stargazers_count":920,"open_issues_count":13,"forks_count":91,"subscribers_count":25,"default_branch":"main","last_synced_at":"2024-12-17T01:37:37.630Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/facebookresearch.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":"CODE_OF_CONDUCT.md","threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2020-06-09T18:07:11.000Z","updated_at":"2024-12-15T04:56:01.000Z","dependencies_parsed_at":"2022-08-26T01:41:18.333Z","dependency_job_id":null,"html_url":"https://github.com/facebookresearch/KILT","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/facebookresearch%2FKILT","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/facebookresearch%2FKILT/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/facebookresearch%2FKILT/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/facebookresearch%2FKILT/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/facebookresearch","download_url":"https://codeload.github.com/facebookresearch/KILT/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":251540013,"owners_count":21605830,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-08-02T13:01:00.342Z","updated_at":"2025-04-29T16:30:32.342Z","avatar_url":"https://github.com/facebookresearch.png","language":"Python","funding_links":[],"categories":["Python","Uncategorized"],"sub_categories":["Uncategorized"],"readme":"![KILT logo](./img/KILT_logo.png)\n--------------------------------------------------------------------------------\n\n# A Benchmark for Knowledge Intensive Language Tasks\n\n[http://kiltbenchmark.com/](http://kiltbenchmark.com)\n\n\nThe KILT benchmark is described in the following paper:\n\n```bibtex\n@inproceedings{petroni-etal-2021-kilt,\n    title = \"{KILT}: a Benchmark for Knowledge Intensive Language Tasks\",\n    author = {Petroni, Fabio  and Piktus, Aleksandra  and\n      Fan, Angela  and Lewis, Patrick  and\n      Yazdani, Majid  and De Cao, Nicola  and\n      Thorne, James  and Jernite, Yacine  and\n      Karpukhin, Vladimir  and Maillard, Jean  and\n      Plachouras, Vassilis  and Rockt{\\\"a}schel, Tim  and\n      Riedel, Sebastian},\n    booktitle = \"Proceedings of the 2021 Conference of the North American Chapter of the Association \n                 for Computational Linguistics: Human Language Technologies\",\n    month = jun,\n    year = \"2021\",\n    address = \"Online\",\n    publisher = \"Association for Computational Linguistics\",\n    url = \"https://aclanthology.org/2021.naacl-main.200\",\n    doi = \"10.18653/v1/2021.naacl-main.200\",\n    pages = \"2523--2544\",\n}\n```\n\n[https://arxiv.org/abs/2009.02252](https://arxiv.org/abs/2009.02252)\n\n\n## Setup the env\n\n```bash\nconda create -n kilt37 -y python=3.7 \u0026\u0026 conda activate kilt37\npip install -e .\n```\n\n## KILT knowledge source\n\nThe KILT knowledge source can be downloaded here: [kilt_knowledgesource.json](http://dl.fbaipublicfiles.com/KILT/kilt_knowledgesource.json) (34.76GiB).\u003cbr\u003e\nIt is based on the [2019/08/01 Wikipedia dump](http://dl.fbaipublicfiles.com/BLINK/enwiki-pages-articles.xml.bz2).\u003cbr\u003e\nWe use [mongoDB](https://www.mongodb.com) to index the knowledge base (but you can use any json-based db).\u003cbr\u003e\nTo import the knowledge source in mongoDB run:\n\n```bash\nwget http://dl.fbaipublicfiles.com/KILT/kilt_knowledgesource.json\nmongoimport --db kilt --collection knowledgesource --file kilt_knowledgesource.json\n```\n\n\n### Structure of each record\n\n```python\n{\n 'wikipedia_title': 'Email marketing',\n 'wikipedia_id': 1101759, \n 'text': ['p1', 'p2',...., 'pn'], # list of paragraph text\n 'anchors': [{\"text\":,\"href\":,\"paragraph_id\":,\"start\":,\"end\":} ]  , \n 'categories': 'comma separated list of categories'\n 'history': # some info from wikipedia, including original url\n 'wikidata_info': # wikidata info\n }\n```\n\n### Query the knowledge source\n\n```python\nfrom kilt.knowledge_source import KnowledgeSource\n\n# get the knowledge souce\nks = KnowledgeSource()\n\n# count entries - 5903530\nks.get_num_pages()\n\n# get page by id\npage = ks.get_page_by_id(27097632)\n\n# get pages by title\npage = ks.get_page_by_title(\"Michael Jordan\")\n```\n\n\n## KILT data\n\nExamples:\n![KILT example](./img/infographic_e.jpg)\n\n### download the data\n\n```bash\nmkdir data\npython scripts/download_all_kilt_data.py\npython scripts/get_triviaqa_input.py\n```\n\nYou can also download and use the KILT data through [the HuggingFace's nlp library](https://huggingface.co/datasets?search=kilt).\n\nNote that we release only the input for the test sets, without answers.\nTest answers are used for [the KILT challenge on EvalAI](https://evalai.cloudcv.org/web/challenges/challenge-page/689/overview) where participants can upload their models’ predictions and be listed on the public leaderboard (there are strict submission limits to discourage overfitting on test data).\n\n### KILT data format\n\n```python\n{'id': # original data point id if available otherwise unique id\n 'input': # question / claim / sentence / etc\n 'output': [ # each element might contain an answer, a provenance or both\n    {\n    'answer': # answer in textual form\n    'provenance': [\n        # evidence set for the answer from the KILT ks\n        {\n            'wikipedia_id':  # *mandatory* \n            'title': \n            'section': \n            'start_paragraph_id': \n            'start_character': \n            'end_paragraph_id':\n            'end_character': \n            'bleu_score': # wrt original evidence\n            'meta': # dataset/task specific\n        }\n        ] \n      }\n    ]\n 'meta': # dataset/task specific\n }\n```\n\n### KILT data catalogue\n\n| dataset | task | train | dev | test |\n| ------------- | ------------- | ------------- | ------------- | ------------- | \n| [FEVER](https://fever.ai) | Fact Checking | [fever-train-kilt.jsonl](http://dl.fbaipublicfiles.com/KILT/fever-train-kilt.jsonl)\u003cbr\u003e(104,966 lines, 38.9MiB)  | [fever-dev-kilt.jsonl](http://dl.fbaipublicfiles.com/KILT/fever-dev-kilt.jsonl)\u003cbr\u003e(10,444 lines, 6.17MiB) | [fever-test_without_answers-kilt.jsonl](http://dl.fbaipublicfiles.com/KILT/fever-test_without_answers-kilt.jsonl)\u003cbr\u003e(10,100 lines, 839kiB) | \n| [AIDA CoNLL-YAGO](https://www.mpi-inf.mpg.de/departments/databases-and-information-systems/research/ambiverse-nlu/aida/downloads) | Entity Linking | [aidayago2-train-kilt.jsonl](http://dl.fbaipublicfiles.com/KILT/aidayago2-train-kilt.jsonl)\u003cbr\u003e(18,395 lines, 70.1MiB) | [aidayago2-dev-kilt.jsonl]( http://dl.fbaipublicfiles.com/KILT/aidayago2-dev-kilt.jsonl)\u003cbr\u003e(4,784 lines, 21.1MiB) | [aidayago2-test_without_answers-kilt.jsonl](http://dl.fbaipublicfiles.com/KILT/aidayago2-test_without_answers-kilt.jsonl)\u003cbr\u003e(4,463 lines, 14.4MiB) | \n| [WNED-WIKI](https://github.com/U-Alberta/wned) | Entity Linking | - | [wned-dev-kilt.jsonl](http://dl.fbaipublicfiles.com/KILT/wned-dev-kilt.jsonl)\u003cbr\u003e(3,396 lines, 12.9MiB) | [wned-test_without_answers-kilt.jsonl](http://dl.fbaipublicfiles.com/KILT/wned-test_without_answers-kilt.jsonl)\u003cbr\u003e(3,376 lines, 13.3MiB) | \n| [WNED-CWEB](https://github.com/U-Alberta/wned) | Entity Linking | - | [cweb-dev-kilt.jsonl](http://dl.fbaipublicfiles.com/KILT/cweb-dev-kilt.jsonl)\u003cbr\u003e(5,599 lines, 90.2MiB)  | [cweb-test_without_answers-kilt.jsonl](http://dl.fbaipublicfiles.com/KILT/cweb-test_without_answers-kilt.jsonl)\u003cbr\u003e(5,543 lines, 100MiB) | \n| [T-REx](https://hadyelsahar.github.io/t-rex) | Slot Filling | [trex-train-kilt.jsonl](http://dl.fbaipublicfiles.com/KILT/trex-train-kilt.jsonl)\u003cbr\u003e(2,284,168 lines, 1.75GiB) | [trex-dev-kilt.jsonl](http://dl.fbaipublicfiles.com/KILT/trex-dev-kilt.jsonl)\u003cbr\u003e(5,000 lines, 3.80MiB) | [trex-test_without_answers-kilt.jsonl](http://dl.fbaipublicfiles.com/KILT/trex-test_without_answers-kilt.jsonl)\u003cbr\u003e(5,000 lines, 896kiB) | \n| [Zero-Shot RE](http://nlp.cs.washington.edu/zeroshot) | Slot Filling | [structured_zeroshot-train-kilt.jsonl](http://dl.fbaipublicfiles.com/KILT/structured_zeroshot-train-kilt.jsonl)\u003cbr\u003e(147,909 lines, 71.4MiB) | [structured_zeroshot-dev-kilt.jsonl](http://dl.fbaipublicfiles.com/KILT/structured_zeroshot-dev-kilt.jsonl)\u003cbr\u003e(3,724 lines, 2.27MiB) | [structured_zeroshot-test_without_answers-kilt.jsonl](http://dl.fbaipublicfiles.com/KILT/structured_zeroshot-test_without_answers-kilt.jsonl)\u003cbr\u003e(4,966 lines, 1.22MiB) |\n| [Natural Questions](https://ai.google.com/research/NaturalQuestions) | Open Domain QA | [nq-train-kilt.jsonl](http://dl.fbaipublicfiles.com/KILT/nq-train-kilt.jsonl)\u003cbr\u003e(87,372 lines, 51.9MiB) | [nq-dev-kilt.jsonl](http://dl.fbaipublicfiles.com/KILT/nq-dev-kilt.jsonl)\u003cbr\u003e(2,837 lines, 7.94MiB) | [nq-test_without_answers-kilt.jsonl](http://dl.fbaipublicfiles.com/KILT/nq-test_without_answers-kilt.jsonl)\u003cbr\u003e(1,444 lines, 334kiB) | \n| [HotpotQA](https://hotpotqa.github.io) | Open Domain QA | [hotpotqa-train-kilt.jsonl](http://dl.fbaipublicfiles.com/KILT/hotpotqa-train-kilt.jsonl)\u003cbr\u003e(88,869 lines, 52.8MiB) | [hotpotqa-dev-kilt.jsonl](http://dl.fbaipublicfiles.com/KILT/hotpotqa-dev-kilt.jsonl)\u003cbr\u003e(5,600 lines, 3.97MiB) | [hotpotqa-test_without_answers-kilt.jsonl](http://dl.fbaipublicfiles.com/KILT/hotpotqa-test_without_answers-kilt.jsonl)\u003cbr\u003e(5,569 lines, 778kiB) |\n| [TriviaQA](http://nlp.cs.washington.edu/triviaqa) | Open Domain QA | [triviaqa-train_id-kilt.jsonl](http://dl.fbaipublicfiles.com/KILT/triviaqa-train_id-kilt.jsonl)\u003csup\u003e*\u003c/sup\u003e\u003cbr\u003e(61,844 lines, 102MiB) | [triviaqa-dev_id-kilt.jsonl](http://dl.fbaipublicfiles.com/KILT/triviaqa-dev_id-kilt.jsonl)\u003csup\u003e*\u003c/sup\u003e\u003cbr\u003e(5,359 lines, 9.81MiB) | [triviaqa-test_id_without_answers-kilt.jsonl](http://dl.fbaipublicfiles.com/KILT/triviaqa-test_id_without_answers-kilt.jsonl)\u003csup\u003e*\u003c/sup\u003e\u003cbr\u003e(6,586 lines, 123kiB) |\n| [ELI5](https://facebookresearch.github.io/ELI5/explore.html) | Open Domain QA | [eli5-train-kilt.jsonl](http://dl.fbaipublicfiles.com/KILT/eli5-train-kilt.jsonl)\u003cbr\u003e(272,634 lines, 548MiB) | [eli5-dev-kilt.jsonl](http://dl.fbaipublicfiles.com/KILT/eli5-dev-kilt.jsonl)\u003cbr\u003e(1,507 lines, 14.1MiB) | [eli5-test_without_answers-kilt.jsonl](http://dl.fbaipublicfiles.com/KILT/eli5-test_without_answers-kilt.jsonl)\u003cbr\u003e(600 lines, 99kiB) | \n| [Wizard of Wikipedia](https://parl.ai/projects/wizard_of_wikipedia) | Dialogue | [wow-train-kilt.jsonl](http://dl.fbaipublicfiles.com/KILT/wow-train-kilt.jsonl)\u003cbr\u003e(63,734 lines, 48.9MiB) | [wow-dev-kilt.jsonl](http://dl.fbaipublicfiles.com/KILT/wow-dev-kilt.jsonl)\u003cbr\u003e(3,054 lines, 2.42MiB) | [wow-test_without_answers-kilt.jsonl](http://dl.fbaipublicfiles.com/KILT/wow-test_without_answers-kilt.jsonl)\u003cbr\u003e(2,944 lines, 1.29MiB)|\n \n\u003csup\u003e*\u003c/sup\u003e run `python scripts/get_triviaqa_input.py` to get the question associated with each id\n\n### Additional data\n\nFor Entity Linking, in addition to the AIDA CoNLL-YAGO train set, the whole knowledge source can be used as training data by exploiting hyperlinks. To facilitate experimentation, we release such data in KILT format following the splits of [BLINK](https://github.com/facebookresearch/BLINK):\n- [blink-train-kilt.jsonl](http://dl.fbaipublicfiles.com/KILT/blink-train-kilt.jsonl) (9M lines)\n- [blink-dev-kilt.jsonl](http://dl.fbaipublicfiles.com/KILT/blink-dev-kilt.jsonl) (10,000 lines)\n\nWe also provide a [script](scripts/map_TAC-KBP2010_to_KILT.py) to map the TAC-KBP 2010 dataset to the knowledge source and format of KILT.\n\n## Run the retrieval evaluation\n\nPlease follow [this README](kilt/retrievers/README.md).\n\n\n## Mapping scripts\n\nMapping scripts are located in `kilt/datasets/`.\nSee `scripts/map_datasets.py` for an example.\n\n\n## Troubleshooting\n\nIf the module cannot be found, preface the python command with `PYTHONPATH=.`\n\nIf the experiments fail on GPU memory allocation, try reducing batch size.\n\n\n## License\nKILT is MIT licensed. See the [LICENSE](LICENSE) file for details.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ffacebookresearch%2FKILT","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Ffacebookresearch%2FKILT","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ffacebookresearch%2FKILT/lists"}