{"id":17909491,"url":"https://github.com/ymcui/mrc-model-analysis","last_synced_at":"2025-09-28T20:30:51.469Z","repository":{"id":38601088,"uuid":"463023018","full_name":"ymcui/mrc-model-analysis","owner":"ymcui","description":"Multilingual Multi-Aspect Explainability Analyses on Machine Reading Comprehension Models (iScience)","archived":false,"fork":false,"pushed_at":"2022-06-21T03:26:25.000Z","size":1998,"stargazers_count":7,"open_issues_count":1,"forks_count":2,"subscribers_count":2,"default_branch":"main","last_synced_at":"2025-01-18T16:11:35.248Z","etag":null,"topics":["bert","mrc","nlp","tensorflow"],"latest_commit_sha":null,"homepage":"https://www.cell.com/iscience/fulltext/S2589-0042(22)00446-1","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/ymcui.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2022-02-24T05:39:56.000Z","updated_at":"2022-11-23T11:56:29.000Z","dependencies_parsed_at":"2022-07-30T06:37:56.867Z","dependency_job_id":null,"html_url":"https://github.com/ymcui/mrc-model-analysis","commit_stats":null,"previous_names":[],"tags_count":1,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ymcui%2Fmrc-model-analysis","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ymcui%2Fmrc-model-analysis/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ymcui%2Fmrc-model-analysis/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ymcui%2Fmrc-model-analysis/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/ymcui","download_url":"https://codeload.github.com/ymcui/mrc-model-analysis/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":234558286,"owners_count":18852274,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["bert","mrc","nlp","tensorflow"],"created_at":"2024-10-28T19:25:49.224Z","updated_at":"2025-09-28T20:30:50.392Z","avatar_url":"https://github.com/ymcui.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# MRC-Model-Analysis\nThis repository contains source code for our paper \"Multilingual Multi-Aspect Explainability Analyses on Machine Reading Comprehension Models\" (formerly \"Understanding Attention in Machine Reading Comprehension. https://arxiv.org/abs/2108.11574v1\").\n\n- [**Multilingual Multi-Aspect Explainability Analyses on Machine Reading Comprehension Models**](https://www.cell.com/iscience/fulltext/S2589-0042(22)00446-1)\n  - Yiming Cui, Wei-Nan Zhang, Wanxiang Che, Ting Liu, Zhigang Chen, Shijin Wang\n  - Published in *iScience*, Cell Press \n\n**(new)** If you would like to know how to visualize attention zones, we have a step-by-step protocol for your perusal.\n\n- [**Visualizing attention zones in machine reading comprehension models**](https://star-protocols.cell.com/protocols/1736)\n  - Yiming Cui, Wei-Nan Zhang, Ting Liu\n  - Published in *STAR Protocols*, Cell Press\n\n\u003cp align=\"center\"\u003e\n    \u003cbr\u003e\n    \u003cimg src=\"./pics/graph_abs.png\" width=\"90%\"/\u003e\n    \u003cbr\u003e\n\u003c/p\u003e\n\n## Requirements\n```\nPython 3.7.3\nTensorFlow 1.15.3\n```\n\nNote:\n\n1. All experiments are carried out using TPU. If you are using other training devices, please adjust these scripts accordingly. \n2. The script might also work under TensorFlow 1.13 ~ 1.15.\n\n### How to run SQuAD baseline (TPU)\nRun the following script (`train_squad.sh`) with proper replacement of a few pre-defined variables.\n```bash\nGS_BUCKET=gs://your-bucket\nTPU_NAME=your-tpu-name\nTPU_ZONE=your-tpu-zone\nMODEL_OUTPUT_DIR=$GS_BUCKET/path-to-output-dir\npython -u run_squad.py \\\n  --vocab_file=$GS_BUCKET/bert/cased_L-12_H-768_A-12/vocab.txt \\\n  --bert_config_file=$GS_BUCKET/bert/cased_L-12_H-768_A-12/bert_config.json \\\n  --init_checkpoint=$GS_BUCKET/bert/cased_L-12_H-768_A-12/bert_model.ckpt \\\n  --do_train=True \\\n  --train_file=./squad/train-v1.1.json \\\n  --do_predict=True \\\n  --predict_file=./squad/dev-v1.1.json \\\n  --train_batch_size=64 \\\n  --predict_batch_size=32 \\\n  --num_train_epochs=3.0 \\\n  --max_seq_length=512 \\\n  --doc_stride=128 \\\n  --learning_rate=3e-5 \\\n  --version_2_with_negative=False \\\n  --output_dir=$MODEL_OUTPUT_DIR \\\n  --do_lower_case=False \\\n  --use_tpu=True \\\n  --tpu_name=$TPU_NAME \\\n  --tpu_zone=$TPU_ZONE\n```\n\n1. Put pre-trained BERT checkpoint in `bert` directory. Note that TPU requires a Google Cloud Storage bucket to save/load model files, which is different from local file system. If you are using GPU/CPU, please ignore $GS_BUCKET variable and just point your local file path.\n   - File download: https://storage.googleapis.com/bert_models/2018_10_18/cased_L-12_H-768_A-12.zip\n   - **WARNING**: A Google Cloud Storage bucket (path start with `gs://`) is **mandatory** when using TPU.\n   - `$TPU_NAME` and `$TPU_ZONE`are the TPU information (created by using `ctpu` or `gcloud compute` commands).\n2. Download SQuAD (v1.1) train/dev files: \n\n   - Train: https://rajpurkar.github.io/SQuAD-explorer/dataset/train-v1.1.json\n\n   - dev: https://rajpurkar.github.io/SQuAD-explorer/dataset/dev-v1.1.json\n3. Put SQuAD train/dev files in `squad` directory:\n   - `mkdir squad \u0026\u0026 mv train-v1.1.json dev-v1.1.json squad`\n\n\n### How to run CMRC 2018 baseline (TPU)\nRun the following script (`train_cmrc2018.sh`) with proper replacement of a few pre-defined variables.\n```bash\nGS_BUCKET=gs://your-bucket\nTPU_NAME=your-tpu-name\nTPU_ZONE=your-tpu-zone\nMODEL_OUTPUT_DIR=$GS_BUCKET/path-to-output-dir\npython -u run_cmrc2018.py \\\n  --vocab_file=$GS_BUCKET/bert/chinese_L-12_H-768_A-12/vocab.txt \\\n  --bert_config_file=$GS_BUCKET/bert/chinese_L-12_H-768_A-12/bert_config.json \\\n  --init_checkpoint=$GS_BUCKET/bert/chinese_L-12_H-768_A-12/bert_model.ckpt \\\n  --do_train=True \\\n  --train_file=./cmrc2018/cmrc2018_train.json \\\n  --do_predict=True \\\n  --predict_file=./cmrc2018/cmrc2018_dev.json \\\n  --train_batch_size=64 \\\n  --predict_batch_size=32 \\\n  --num_train_epochs=2 \\\n  --max_seq_length=512 \\\n  --doc_stride=128 \\\n  --learning_rate=3e-5 \\\n  --do_lower_case=True \\\n  --output_dir=$MODEL_OUTPUT_DIR \\\n  --use_tpu=True \\\n  --tpu_name=$TPU_NAME \\\n  --tpu_zone=$TPU_ZONE\n```\n\n1. Put pre-trained BERT checkpoint in `bert` directory. Note that TPU requires a Google Cloud Storage bucket to save/load model files, which is different from local file system. If you are using GPU/CPU, please ignore $GS_BUCKET variable and just point your local file path.\n\n   - File download: https://storage.googleapis.com/bert_models/2018_11_03/chinese_L-12_H-768_A-12.zip\n\n   - **WARNING**: A Google Cloud Storage bucket (path start with `gs://`) is **mandatory** when using TPU.\n   - `$TPU_NAME` and `$TPU_ZONE`are the TPU information (created by using `ctpu` or `gcloud compute` commands).\n\n2. Download CMRC 2018 train/dev file:\n   - https://github.com/ymcui/cmrc2018/tree/master/squad-style-data\n\n3. Put CMRC 2018 train/dev files in `cmrc2018` directory\n   - `mkdir cmrc2018 \u0026\u0026 mv train-v1.1.json dev-v1.1.json cmrc2018`\n\n\n### How to mask attention zones \n\nSimply pass an additional argument `--mask_zone` and `--mask_layer` to `run_squad.py` or `run_cmrc2018.py` script when decoding.\n\n#### `--mask_zone` argument\n\nThe followings are valid values for `--mask_zone`:\n\n- \"q2\": masking Q2 zone\n- \"q2p\": masking Q2P zone\n- \"p2q\": masking P2Q zone\n- \"p2\": masking P2 zone\n- \"all\": masking all zones\n\n#### `--mask_layer` argument\n\nSpecify which layer should be masked. The starting index is 0. For example, in BERT-base, the index for the first transformer layer is `0` and the last is `11`.\n\n\n### Visualization (new)\n\nPlease consult our detailed protocol that was published in STAR Protocols. The presented protocol provides a step-by-step guideline on how to visualize attention zones in machine reading comprehension models.\n\nSee: [**Visualizing attention zones in machine reading comprehension models**](https://star-protocols.cell.com/protocols/1736)\n\n\u003cp align=\"center\"\u003e\n    \u003cbr\u003e\n    \u003cimg src=\"./pics/star_protocol.png\" width=\"60%\"/\u003e\n    \u003cbr\u003e\n\u003c/p\u003e\n\n## Citation\n\nIf your find our work helpful, please consider cite our paper.\n\n```bibtex\n@article{cui-etal-2022-mrc,\n      title = {Multilingual Multi-Aspect Explainability Analyses on Machine Reading Comprehension Models},\n      author = {Cui, Yiming and Zhang, Wei-Nan and Che, Wanxiang and Liu, Ting and Chen, Zhigang and Wang, Shijin},\n      journal = {iScience},\n      volume = {25},\n      number = {5},\n      pages = {104176},\n      year = {2022},\n      issn = {2589-0042},\n      doi = {https://doi.org/10.1016/j.isci.2022.104176},\n      url = {https://www.sciencedirect.com/science/article/pii/S2589004222004461},\n}\n```\n\nAlso, if you find our step-by-step protocol useful, please cite the following paper.\n```\n@article{cui-2022-mrc-protocol,\ntitle = {Visualizing attention zones in machine reading comprehension models},\njournal = {STAR Protocols},\nvolume = {3},\nnumber = {3},\npages = {101481},\nyear = {2022},\nissn = {2666-1667},\ndoi = {https://doi.org/10.1016/j.xpro.2022.101481},\nauthor = {Yiming Cui and Wei-Nan Zhang and Ting Liu},\n}\n```\n\n## Zenodo Archive\n[![DOI](https://zenodo.org/badge/463023018.svg)](https://zenodo.org/badge/latestdoi/463023018)\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fymcui%2Fmrc-model-analysis","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fymcui%2Fmrc-model-analysis","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fymcui%2Fmrc-model-analysis/lists"}