{"id":13937157,"url":"https://github.com/google-deepmind/rc-data","last_synced_at":"2025-09-27T08:30:48.870Z","repository":{"id":51360337,"uuid":"43331133","full_name":"google-deepmind/rc-data","owner":"google-deepmind","description":"Question answering dataset featured in \"Teaching Machines to Read and Comprehend","archived":true,"fork":false,"pushed_at":"2017-04-26T15:05:19.000Z","size":1326,"stargazers_count":1286,"open_issues_count":3,"forks_count":247,"subscribers_count":104,"default_branch":"master","last_synced_at":"2024-04-16T04:53:40.651Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/google-deepmind.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2015-09-28T22:42:19.000Z","updated_at":"2024-04-12T13:21:26.000Z","dependencies_parsed_at":"2022-08-25T13:01:51.645Z","dependency_job_id":null,"html_url":"https://github.com/google-deepmind/rc-data","commit_stats":null,"previous_names":["google-deepmind/rc-data","deepmind/rc-data"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/google-deepmind%2Frc-data","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/google-deepmind%2Frc-data/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/google-deepmind%2Frc-data/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/google-deepmind%2Frc-data/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/google-deepmind","download_url":"https://codeload.github.com/google-deepmind/rc-data/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":234410617,"owners_count":18828254,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-08-07T23:03:20.404Z","updated_at":"2025-09-27T08:30:48.061Z","avatar_url":"https://github.com/google-deepmind.png","language":"Python","funding_links":[],"categories":["Python"],"sub_categories":[],"readme":"# Question Answering Corpus\n\nThis repository contains a script to generate question/answer pairs using\nCNN and Daily Mail articles downloaded from the Wayback Machine.\n\nFor a detailed description of this corpus please read:\n[Teaching Machines to Read and Comprehend][arxiv], Hermann et al., NIPS 2015.\nPlease cite the paper if you use this corpus in your work.\n\n### Bibtex\n\n```\n@inproceedings{nips15_hermann,\nauthor = {Karl Moritz Hermann and Tom\\'a\\v{s} Ko\\v{c}isk\\'y and Edward Grefenstette and Lasse Espeholt and Will Kay and Mustafa Suleyman and Phil Blunsom},\ntitle = {Teaching Machines to Read and Comprehend},\nurl = {http://arxiv.org/abs/1506.03340},\nbooktitle = \"Advances in Neural Information Processing Systems (NIPS)\",\nyear = \"2015\",\n}\n```\n\n## Download Processed Version\n\nIn case the script does not work you can also download the processed data sets from [http://cs.nyu.edu/~kcho/DMQA/]. This should help in situations where the underlying data is not accessible (Wayback Machine partially down).\n\n## Running the Script\n\n### Prerequisites\n\nPython 2.7, `wget`, `libxml2`, `libxslt`, `python-dev` and `virtualenv`. `libxml2` must be version 2.9.1. \nYou can install `libxslt` from here: [http://xmlsoft.org/libxslt/downloads.html](http://xmlsoft.org/libxslt/downloads.html)\n\n```\nsudo pip install virtualenv\nsudo apt-get install python-dev\n```\n\n### Download Script\n\n```\nmkdir rc-data\ncd rc-data\nwget https://github.com/deepmind/rc-data/raw/master/generate_questions.py\n```\n\n### Download and Extract Metadata\n\n```\nwget https://storage.googleapis.com/deepmind-data/20150824/data.tar.gz -O - | tar -xz --strip-components=1\n```\n\nThe news article metadata is ~1 GB.\n\n### Enter Virtual Environment and Install Packages\n\n```\nvirtualenv venv\nsource venv/bin/activate\nwget https://github.com/deepmind/rc-data/raw/master/requirements.txt\npip install -r requirements.txt\n```\n\nYou may need to install `libxml2` development packages to install `lxml`:\n\n```\nsudo apt-get install libxml2-dev libxslt-dev\n```\n\n### Download URLs\n\n```\npython generate_questions.py --corpus=[cnn/dailymail] --mode=download\n```\n\nThis will download news articles from the Wayback Machine. Some URLs may be\nunavailable. The script can be run again and will cache\nURLs that already have been downloaded. Generation of questions can run\nwithout all URLs downloaded successfully.\n\n### Generate Questions\n\n```\npython generate_questions.py --corpus=[cnn/dailymail] --mode=generate\n```\n\nNote, this will generate ~1,000,000 small files for the Daily Mail so an SSD is\npreferred.\n\nQuestions are stored in [cnn/dailymail]/questions/ in the following format:\n\n```\n[URL]\n\n[Context]\n\n[Question]\n\n[Answer]\n\n[Entity mapping]\n```\n\n### Deactivate Virtual Environment\n\n```\ndeactivate\n```\n\n### Verifying Test Sets\n\n```\nwget https://github.com/deepmind/rc-data/raw/master/expected_[cnn/dailymail]_test.txt\ncomm -3 \u003c(cat expected_[cnn/dailymail]_test.txt) \u003c(ls [cnn/dailymail]/questions/test/)\n```\n\nThe filenames of the questions in the first column are missing generated questions. No output means everything is downloaded and generated correctly.\n\n[arxiv]: http://arxiv.org/abs/1506.03340\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fgoogle-deepmind%2Frc-data","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fgoogle-deepmind%2Frc-data","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fgoogle-deepmind%2Frc-data/lists"}