{"id":22035871,"url":"https://github.com/habout632/dureader","last_synced_at":"2026-04-30T00:37:27.146Z","repository":{"id":95795479,"uuid":"125481753","full_name":"habout632/dureader","owner":"habout632","description":"DuReader Dataset Preprocessing a fork of official dataset and eval script","archived":false,"fork":false,"pushed_at":"2018-04-07T08:34:17.000Z","size":13517,"stargazers_count":1,"open_issues_count":0,"forks_count":0,"subscribers_count":3,"default_branch":"master","last_synced_at":"2025-01-28T18:49:16.798Z","etag":null,"topics":["a"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/habout632.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2018-03-16T07:46:26.000Z","updated_at":"2018-04-07T08:34:18.000Z","dependencies_parsed_at":"2023-05-06T14:56:46.013Z","dependency_job_id":null,"html_url":"https://github.com/habout632/dureader","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/habout632%2Fdureader","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/habout632%2Fdureader/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/habout632%2Fdureader/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/habout632%2Fdureader/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/habout632","download_url":"https://codeload.github.com/habout632/dureader/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":245104462,"owners_count":20561377,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["a"],"created_at":"2024-11-30T10:28:14.012Z","updated_at":"2026-04-30T00:37:27.117Z","avatar_url":"https://github.com/habout632.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# DuReader Dataset\nDuReader is a new large-scale real-world and human sourced MRC dataset in Chinese. DuReader focuses on real-world open-domain question answering. The advantages of DuReader over existing datasets are concluded as follows:\n - Real question\n - Real article\n - Real answer\n - Real application scenario\n - Rich annotation\n\n# DuReader Baseline Systems\nDuReader system implements 2 classic reading comprehension models([BiDAF](https://arxiv.org/abs/1611.01603) and [Match-LSTM](https://arxiv.org/abs/1608.07905)) on [DuReader dataset](https://ai.baidu.com//broad/subordinate?dataset=dureader). The system is implemented with 2 frameworks: [PaddlePaddle](http://paddlepaddle.org) and [TensorFlow](https://www.tensorflow.org).\n\n## How to Run\n### Download the Dataset\nTo Download DuReader dataset:\n```\ncd data \u0026\u0026 bash download.sh\n```\nFor more details about DuReader dataset please refer to [DuReader Homepage](https://ai.baidu.com//broad/subordinate?dataset=dureader).\n\n### Download Thirdparty Dependencies\nWe use Bleu and Rouge as evaluation metrics, the calculation of these metrics relies on the scoring scripts under \"https://github.com/tylin/coco-caption\", to download them, run:\n\n```\ncd utils \u0026\u0026 bash download_thirdparty.sh\n```\n\n### Preprocess the Data\nAfter the dataset is downloaded, there is still some work to do to run the baseline systems. DuReader dataset offers rich amount of documents for every user question, the documents are too long for popular RC models to cope with. In our baseline models, we preprocess the train set and development set data by selecting the paragraph that is most related to the answer string, while for inferring(no available golden answer), we select the paragraph that is most related to the question string. The preprocessing strategy is implemented in `utils/preprocess.py`. To preprocess the raw data, you should first segment 'question', 'title', 'paragraphs' and then store the segemented result into 'segmented_question', 'segmented_title', 'segmented_paragraphs' like the downloaded preprocessed data, then run:\n```\ncat data/raw/trainset/search.train.json | python utils/preprocess.py \u003e data/preprocessed/trainset/search.train.json\n```\nThe preprocessed data can be automatically downloaded by `data/download.sh`, and is stored in `data/preprocessed`, the raw data before preprocessing is under `data/raw`.\n\n### Run PaddlePaddle\n\n#### Get the Vocab File\n\nOnce the preprocessed data is ready, you can run `utils/get_vocab.py` to generate the vocabulary file, for example, if you want to train model with Baidu Search data:\n```\npython utils/get_vocab.py --files data/preprocessed/trainset/search.train.json data/preprocessed/devset/search.dev.json  --vocab data/vocab.search\n```\n\nIf you want to use the demo data, run:\n```\npython utils/get_vocab.py --files data/demo/trainset/search.train.json data/demo/devset/search.dev.json  --vocab data/demo/vocab.search\n```\n\n#### Environment Requirements\nFor now we've only tested on PaddlePaddle v0.10.5, to install PaddlePaddle and for more details about PaddlePaddle, see [PaddlePaddle Homepage](http://paddlepaddle.org).\n\n#### Training\nWe implement 3 models with PaddlePaddle: Match-LSTM, BiDAF, and a classification model for data with `query_type='YES_NO'`, the model simply replaces the Pointer-Net on top of Match-LSTM model with a one-layered classifier. The 3 implemented models can all be trained and inferred by run `run.py`, to specify the model to train or to infer, use `--algo [mlstm|bidaf|yesno]`, for complete usage run `python run.py -h`.\n\nThe basic training and inference process has been wrapped in `run.sh`,  the basic usage is:\n```\nbash run.sh EXPERIMENT_NAME ALGO_NAME TASK_NAME\n```\n`EXPERIMENT_NAME` can be any legal folder name,  `ALGO_NAME` should be `bidaf`, `mlstm` or `yesno` for the 3 models have been implemented.\nFor example, to train a model with BiDAF, run:\n```\nbash run.sh test_bidaf bidaf train\n```\n`run.sh` creates a folder named `models`, and for every experiment a folder named `EXPERIMENT_NAME` is created under models, the basic experiment folder layout should be like:\n```\nmodels\n└── test_bidaf\n    ├── env\n    ├── infer\n    ├── log\n    └── models\n```\nFor training, all scripts the experiment uses will first be copied to `env`, and then run from there, and inference process is also run from `env`. `infer` folder keeps the result file created by inference, `log` folder keeps training and inference logs, and `models` folder keeps the models saved during training.\n\n*Because our datatset and the model capacity is very large, if it's out of your device's capacity to successfully run the whole process, you can try with the shipped demo data, just use `run_demo.sh` for training and inferring,the usage is the same as `run.sh`*\n\n#### Inference\nTo infer a trained model, run the same command as training and change `train` to `infer`,  and add `--testset \u003cpath_to_testset\u003e` argument. for example, suppose the 'test_bidaf' experiment is successfully trained,  to infer the saved models, run:\n```\nbash run.sh test_bidaf bidaf infer --testset ../data/preprocessed/testset/search.test.json\n```\nThe results corresponding to each model saved is under `infer` folder, and the evaluation metrics is logged into the infer log files under `log`.\n\nNote if you want to infer a 'yesno' model, please sepecify an inferred result of a RC model, i.e. 'bidaf' or 'mlstm', under `models/SOME_RC_MODEL/infer/`, to `--testset`, because the 'yesno' model need the result answer of a RC model as its input.\n\n#### Test result submission\nYou can infer and evaluate your models on development data set locally by following the above steps, once you've developed a model that works to your expectation on the dev set, we highly recommend you to submit your inference result on the released test set to us to evaluate. To get inference file on test set:\n\n1. make sure the training is over.\n2. infer your models on dev set and pick the best model.\n3. keep only the best model under `models/\u003cEXPERIMENT_NAME\u003e/models`.\n4. infer again with test set.\n5. [submit the infer result file](http://ai.baidu.com/broad/submission?dataset=dureader).\n\n### Run Tensorflow\n\nWe also implements the BIDAF and Match-LSTM models based on Tensorflow 1.0. You can refer to the [official guide](https://www.tensorflow.org/versions/r1.0/install/) for the installation of Tensorflow. The complete options for running our Tensorflow program can be accessed by using `python run.py -h`. Here we demonstrate a typical workflow as follows: \n\n#### Preparation\nBefore training the model, we have to make sure that the data is ready. For preparation, we will check the data files, make directories and extract a vocabulary for later use. You can run the following command to do this with a specified task name:\n\n```\npython run.py --prepare\n```\nYou can specify the files for train/dev/test by setting the `train_files`/`dev_files`/`test_files`. By default, we use the data in `data/demo/`\n\n#### Training\nTo train the reading comprehension model, you can specify the model type by using `--algo [BIDAF|MLSTM]` and you can also set the hyper-parameters such as the learning rate by using `--learning_rate NUM`. For example, to train a BIDAF model for 10 epochs, you can run:\n\n```\npython run.py --train --algo BIDAF --epochs 10\n```\n\nThe training process includes an evaluation on the dev set after each training epoch. By default, the model with the least Bleu-4 score on the dev set will be saved.\n\n#### Evaluation\nTo conduct a single evaluation on the dev set with the the model already trained, you can run the following command:\n\n```\npython run.py --evaluate --algo BIDAF\n```\n\n#### Prediction\nYou can also predict answers for the samples in some files using the following command:\n\n```\npython run.py --predict --algo BIDAF --test_files ../data/demo/search.dev.json\n```\n\nBy default, the results are saved at `../data/results/` folder. You can change this by specifying `--result_dir DIR_PATH`.\n\n## Copyright and License\nCopyright 2017 Baidu.com, Inc. All Rights Reserved\n\nLicensed under the Apache License, Version 2.0 (the \"License\");\nyou may not use this file except in compliance with the License.\nYou may obtain a copy of the License at\n\n    http://www.apache.org/licenses/LICENSE-2.0\n\nUnless required by applicable law or agreed to in writing, software\ndistributed under the License is distributed on an \"AS IS\" BASIS,\nWITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\nSee the License for the specific language governing permissions and\nlimitations under the License.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fhabout632%2Fdureader","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fhabout632%2Fdureader","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fhabout632%2Fdureader/lists"}