{"id":13595123,"url":"https://github.com/StonyBrookNLP/deformer","last_synced_at":"2025-04-09T10:32:53.809Z","repository":{"id":37641562,"uuid":"260568081","full_name":"StonyBrookNLP/deformer","owner":"StonyBrookNLP","description":"[ACL 2020] DeFormer: Decomposing Pre-trained Transformers for Faster Question Answering","archived":false,"fork":false,"pushed_at":"2023-05-22T22:45:57.000Z","size":440,"stargazers_count":120,"open_issues_count":9,"forks_count":16,"subscribers_count":23,"default_branch":"master","last_synced_at":"2024-11-06T17:45:44.766Z","etag":null,"topics":["acl2020","decomposition","deformer","question-answering","transformer"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/StonyBrookNLP.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null}},"created_at":"2020-05-01T22:10:27.000Z","updated_at":"2024-10-09T04:34:58.000Z","dependencies_parsed_at":"2024-01-16T22:32:04.469Z","dependency_job_id":null,"html_url":"https://github.com/StonyBrookNLP/deformer","commit_stats":null,"previous_names":[],"tags_count":1,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/StonyBrookNLP%2Fdeformer","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/StonyBrookNLP%2Fdeformer/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/StonyBrookNLP%2Fdeformer/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/StonyBrookNLP%2Fdeformer/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/StonyBrookNLP","download_url":"https://codeload.github.com/StonyBrookNLP/deformer/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248020593,"owners_count":21034459,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["acl2020","decomposition","deformer","question-answering","transformer"],"created_at":"2024-08-01T16:01:44.371Z","updated_at":"2025-04-09T10:32:48.799Z","avatar_url":"https://github.com/StonyBrookNLP.png","language":"Python","funding_links":[],"categories":["Python"],"sub_categories":[],"readme":"# **DeFormer**: **De**composing Pre-trained Trans**former**s for Faster Question Answering\n\nThis repo is the code for the [DeFormer paper](https://awk.ai/assets/deformer.pdf)  (Accepted to ACL 2020).\n\n\u003cimg style=\"margin:auto;width:50%\" src=\"https://awk.ai/assets/deformer-sketch.png\" alt=\"deformer\"/\u003e\n\n\u003c!--ts--\u003e\n   * [Installation](#installation)\n   * [Usage](#usage)\n      * [Data Processing](#dataset-processing)\n        * [download dataset](#dataset-processing)\n        * [convert dataset](#dataset-processing)\n        * [generate examples](#dataset-processing)\n      * [Training and Evaluation](#training-and-evaluation)\n      * [Experimenting](#experimenting)\n        * [tune ebert](#tune-ebert)\n        * [tune sbert](#tune-sbert)\n      * [Profiling](#profiling)\n      * [Demo](#demo)\n      * [Tools](#tools)\n      * [Handy Commands](#handy-commands)\n   * [FAQ](#faq)\n   * [Citation](#citation)\n\u003c!--te--\u003e\n\n## Installation\n\nTested on Ubuntu 16.04, 18.04 and macOS. (Windows should also work, but not tested)\n\nYou can create a separate python environment, e.g. `virtualenv -p python3.7 .env`\nand activate it by `source .env/bin/activate`\n\n1. Requirements: Python\u003e=3.5 and TensorFlow \u003e=1.14.0,\u003c2.0\n\n2. `pip install \"tensorflow\u003e=1.14.0,\u003c2.0\"` or `pip install tensorflow-gpu==1.15.3` (for GPU)\n\n3. `pip install -r requirements.txt`\n\n**NOTE**: we call `ebert` for DeFormer BERT version, and `sbert` for applying KD \u0026 LRS in the paper.\n\nFor XLNet, you can check [my fork](https://github.com/csarron/xlnet) for a reference implementation.\n\n## Usage\n\n### Dataset Processing\n\n#### downloading datasets to `data/datasets`\n  - GLUE: [link](https://gist.githubusercontent.com/csarron/2a7f5da27f45e7e0795c9946f7c95f76/raw/17b8dd0d724281ed7c3b2aeeda662b92809aadd5/download_glue_data.py)\n  - SQuAD v1.1: [train-v1.1.json](https://rajpurkar.github.io/SQuAD-explorer/dataset/train-v1.1.json) and \n  [dev-v1.1.json](https://rajpurkar.github.io/SQuAD-explorer/dataset/dev-v1.1.json)\n  - [RACE dataset](https://www.cs.cmu.edu/~glai1/data/race/)\n\n\u003c!--\n  - squad v2.0: [train-v2.0.json](https://rajpurkar.github.io/SQuAD-explorer/dataset/train-v2.0.json) and \n  [dev-v2.0.json](https://rajpurkar.github.io/SQuAD-explorer/dataset/dev-v2.0.json)\n  - hotpot qa: [train_v1.1.json](http://curtis.ml.cmu.edu/datasets/hotpot/hotpot_train_v1.1.json) and \n  [dev_distractor_v1.json](http://curtis.ml.cmu.edu/datasets/hotpot/hotpot_dev_distractor_v1.json)\n--\u003e\n\n   the dataset dir should look like below (use `tree -L 2 data/datasets`):\n````log\ndata/datasets\n├── BoolQ\n│   ├── test.jsonl\n│   ├── train.jsonl\n│   └── val.jsonl\n├── mnli\n│   ├── dev_mismatched.tsv\n│   └── train.tsv\n├── qqp\n│   ├── dev.tsv\n│   ├── test.tsv\n│   └── train.tsv\n├── RACE\n│   ├── dev\n│   ├── test\n│   └── train\n└── squad_v1.1\n    ├── dev-v1.1.json\n    └── train-v1.1.json\n````\n\n#### convert to DeFormer format\n\nconvert:\n\n````bash\ndeformer_dir=data/datasets/deformer\nmkdir -p ${deformer_dir}\n\n# squad v1.1\nfor version in 1.1; do\n    data_dir=data/datasets/squad_v${version}\n    for split in dev train; do\n        python tools/convert_squad.py ${data_dir}/${split}-v${version}.json \\\n        ${deformer_dir}/squad_v${version}-${split}.jsonl\n    done\ndone\n\n# mnli\ndata_dir=data/datasets/mnli\npython tools/convert_pair_dataset.py ${data_dir}/train.tsv ${deformer_dir}/mnli-train.jsonl -t mnli\npython tools/convert_pair_dataset.py ${data_dir}/dev_matched.tsv ${deformer_dir}/mnli-dev.jsonl  -t mnli\n\n# qqp\ndata_dir=data/datasets/qqp\npython tools/convert_pair_dataset.py ${data_dir}/train.tsv ${deformer_dir}/qqp-train.jsonl -t qqp\npython tools/convert_pair_dataset.py ${data_dir}/dev.tsv ${deformer_dir}/qqp-dev.jsonl -t qqp\n\n# boolq\ndata_dir=data/datasets/BoolQ\npython tools/convert_pair_dataset.py ${data_dir}/train.jsonl ${deformer_dir}/boolq-train.jsonl -t boolq\npython tools/convert_pair_dataset.py ${data_dir}/val.jsonl ${deformer_dir}/boolq-dev.jsonl -t boolq\n\n# race\ndata_dir=data/datasets/RACE\npython tools/convert_race.py ${data_dir}/train ${deformer_dir}/race-train.jsonl\npython tools/convert_race.py ${data_dir}/dev ${deformer_dir}/race-dev.jsonl\n\n````\n\nsplit 10% of train for tuning hyper-parameters:\n  \n````bash\ncd ${deformer_dir}\n\ncat squad_v1.1-train.jsonl | shuf \u003e squad_v1.1-train-shuf.jsonl\nhead -n8760 squad_v1.1-train-shuf.jsonl \u003e squad_v1.1-tune.jsonl\ntail -n78839 squad_v1.1-train-shuf.jsonl \u003e squad_v1.1-train.jsonl\n\ncat boolq-train.jsonl | shuf \u003e boolq-train-shuf.jsonl\nhead -n943 boolq-train-shuf.jsonl \u003e boolq-tune.jsonl\ntail -n8484 boolq-train-shuf.jsonl \u003e boolq-train.jsonl\n\ncat race-train.jsonl | shuf \u003e race-train-shuf.jsonl\nhead -n8786 race-train-shuf.jsonl \u003e race-tune.jsonl\ntail -n79080 race-train-shuf.jsonl \u003e race-train.jsonl\n\ncat qqp-train.jsonl | shuf \u003e qqp-train-shuf.jsonl\nhead -n36385 qqp-train-shuf.jsonl \u003e qqp-tune.jsonl\ntail -n327464 qqp-train-shuf.jsonl \u003e qqp-train.jsonl\n\ncat mnli-train.jsonl | shuf \u003e mnli-train-shuf.jsonl\nhead -n39270 mnli-train-shuf.jsonl \u003e mnli-tune.jsonl\ntail -n353432 mnli-train-shuf.jsonl \u003e mnli-train.jsonl\n\n```` \n\n#### download BERT vocab\n\ndownload [bert.vocab](https://github.com/StonyBrookNLP/deformer/releases/download/v1.0/bert.vocab) to `data/res`\n\n#### generating training and evaluation examples:\n\n  usage: `python prepare.py -h`\n  \n  - e.g., convert `squad_v1.1` for `bert`: \n    ````bash\n    python prepare.py -m bert -t squad_v1.1 -s dev\n    python prepare.py -m bert -t squad_v1.1 -s tune\n    python prepare.py -m bert -t squad_v1.1 -s train -sm tf\n    ````\n\n  - e.g., convert `squad_v1.1` for `xlnet`: \n    ````bash\n    model=xlnet\n    task=squad_v1.1\n    python prepare.py -m ${model} -t ${task} -s dev\n    python prepare.py -m ${model} -t ${task} -s train -sm tf\n    ````\n\n  - convert all available tasks and all models:\n    ````bash\n    for model in bert ebert; do\n      for task in squad_v1.1 mnli qqp boolq race; do\n        python prepare.py -m ${model} -t ${task} -s dev\n        python prepare.py -m ${model} -t ${task} -s tune\n        python prepare.py -m ${model} -t ${task} -s train -sm tf\n      done\n    done\n    ````\n\n\n### Training and Evaluation\n\n#### SQuAD 1.1 Quickstart\n\ndownload original fine-tuned BERT-base checkpoints from [bert-base-squad_v1.1.tgz](https://github.com/StonyBrookNLP/deformer/releases/download/v1.0/bert-base-squad_v1.1.tgz)\nand DeFormer fine-tuned version from [ebert-base-s9-squad_v1.1.tgz](https://github.com/StonyBrookNLP/deformer/releases/download/v1.0/ebert-base-s9-squad_v1.1.tgz)\n\n`python eval.py -m bert -t squad_v1.1 2\u003e\u00261 | tee data/bert-base-eval.log`\nexample output:\n````log\nINFO:2020-07-01_15:36:30.339:eval.py:65: model.ckpt-8299, em=80.91769157994324, f1=88.33819502660548, metric=88.33819502660548\n````\n\n`python eval.py -m ebert -t squad_v1.1 2\u003e\u00261 | tee data/ebert-base-s9-eval.log`\n\nexample output:\n````log\nINFO:2020-07-01_15:39:15.418:eval.py:65: model.ckpt-8321, em=79.12961210974456, f1=86.99636369864814, metric=86.99636369864814\n````\n\n#### Train and Eval\n\nSee `config/*.ini` for customizing training and evaluation script\n\n- train: `python train.py` specify model by `-m`(`--model`), task by `-t`(`--task`), eval is similar.\nsee below example commands for `boolq`: \n\n    ````bash\n    # for running on tpu, should specify gcs bucket data_dir, and set use_tpu to yes\n    # also need to set tpu_name=\u003csome_ip_or_just_name\u003e if not exported to environment\n    base_dir=\u003cyour google cloud storage bucket\u003e\n    data_dir=${base_dir} use_tpu=yes \\\n    python train.py -m bert -t boolq 2\u003e\u00261 | tee data/boolq-bert-train.log\n    \n    data_dir=${base_dir} use_tpu=yes \\\n    python eval.py -m bert -t boolq 2\u003e\u00261 | tee data/boolq-bert-eval.log\n  \n    # for list of models and list of tasks\n    for task in boolq mnli qqp squad_v1.1; do\n      for model in bert ebert; do\n        data_dir=${base_dir} use_tpu=yes \\\n        python train.py -m ${model} -t ${task} 2\u003e\u00261 | tee data/${task}-${model}-train.log\n        \n        data_dir=${base_dir} use_tpu=yes \\\n        python eval.py -m ${model} -t ${task} 2\u003e\u00261 | tee data/${task}-${model}-eval.log\n      done\n    done\n    ````\n\n- BERT wwm large:\n\n    ````bash\n    base_dir=\u003cyour google cloud storage bucket\u003e\n    for t in boolq qqp squad_v1.1 mnli; do\n      use_tpu=yes data_dir=${base_dir} \\\n      learning_rate=1e-5 epochs=2 keep_checkpoint_max=1 \\\n      init_checkpoint=${base_dir}/ckpt/init/wwm_uncased_large/bert_model.ckpt \\\n      checkpoint_dir=${base_dir}/ckpt/bert_large/${t} \\\n      hidden_size=1024 intermediate_size=4096 num_heads=16 num_hidden_layers=24 \\\n      python train.py -m bert -t ${t} 2\u003e\u00261 | tee data/${t}-large-train.log\n    \n      data_dir=${base_dir} use_tpu=yes init_checkpoint=\"\" \\\n      checkpoint_dir=${base_dir}/ckpt/bert_large/${t} \\\n      hidden_size=1024 intermediate_size=4096 num_heads=16 num_hidden_layers=24 \\\n      python eval.py -m bert -t ${t} 2\u003e\u00261 | tee data/${t}-large-eval.log\n    done || exit 1\n  ````\n\n### Experimenting\n\n#### Tune EBert\n\n- fine tuning for separation at different layers for bert base:\n\n    ````bash\n    for t in boolq qqp mnli squad_v1.1; do\n      for n in `seq 1 1 11`; do\n        echo \"n=${n}, t=${t}\"\n        base_dir=${base_dir}\n\n        sep_layers=${n} use_tpu=yes data_dir=${base_dir} keep_checkpoint_max=1 \\\n        checkpoint_dir=\"${base_dir}/ckpt/separation/${t}/ebert_s${n}\" \\\n        python train.py -m ebert -t ${t} 2\u003e\u00261 | tee data/${t}-base-sep${n}-train.log\n\n        sep_layers=${n} use_tpu=yes data_dir=${base_dir} init_checkpoint=\"\" \\\n        checkpoint_dir=\"${base_dir}/ckpt/separation/${t}/ebert_s${n}\" \\\n        python eval.py -m ebert -t ${t} 2\u003e\u00261 | tee data/${t}-base-sep${n}-eval.log\n      done\n    done\n    ````\n  \n- fine tuning for separation at different layers for wwm large bert:\n    \n    ````bash\n    for t in boolq qqp mnli squad_v1.1; do\n      for n in `seq 10 1 23`; do\n        echo \"n=${n}, t=${t}\"\n        base_dir=${base_dir}\n      \n        sep_layers=${n} use_tpu=yes data_dir=${base_dir} \\\n        learning_rate=1e-5 epochs=2 keep_checkpoint_max=1 \\\n        init_checkpoint=${base_dir}/ckpt/init/wwm_uncased_large/bert_model.ckpt \\\n        checkpoint_dir=${base_dir}/ckpt/separation/${t}/ebert_large_s${n} \\\n        hidden_size=1024 intermediate_size=4096 num_heads=16 num_hidden_layers=24 \\\n        python train.py -m ebert -t ${t} 2\u003e\u00261 | tee data/${t}-large-sep${n}-train.log\n      \n        sep_layers=${n} use_tpu=yes data_dir=${base_dir} init_checkpoint=\"\" \\\n        checkpoint_dir=${base_dir}/ckpt/separation/${t}/ebert_large_s${n} \\\n        hidden_size=1024 intermediate_size=4096 num_heads=16 num_hidden_layers=24 \\\n        output_file=${base_dir}/predictions/${t}-large-sep${n}-dev.json \\\n        python eval.py -m ebert -t ${t} 2\u003e\u00261 | tee data/${t}-large-sep${n}-eval.log\n      done || exit 1\n    done || exit 1\n  ````\n\n#### Tune SBert\n\n- [ ] training script needs further verification (due to migrated from old codebase)\n\n- sbert procedure, first get ebert_s0, then merge bert_base and ebert_s0 checkpoints \nusing `tools/merge_checkpoints.py` to get initial checkpoint for sbert, then run the training.\n\n    ````bash\n    base_dir=gs://xxx\n    init_dir=\"data/ckpt/init\"\n    large_model=\"${init_dir}/wwm_uncased_large/bert_model.ckpt\"\n    base_model=\"${init_dir}/uncased_base/bert_model.ckpt\"\n    \n    for t in squad_v1.1 boolq qqp mnli; do\n      mkdir -p data/ckpt/separation/${t}\n      \n      # sbert large init\n      large_init=\"data/ckpt/separation/${t}/ebert_large_s0\"\n      gsutil -m cp -r \"${base_dir}/ckpt/separation/${t}/ebert_large_s0\" data/ckpt/separation/${t}/\n      \n      python tools/merge_checkpoints.py -c1 \"${large_init}\" \\\n      -c2 \"${large_model}\" -o ${init_dir}/${t}_sbert_large.ckpt\n      gsutil -m cp -r \"${init_dir}/${t}_sbert_large.ckpt*\" \"${base_dir}/ckpt/init\"\n      \n      # sbert large init from ebert_large_s0 all\n      python tools/merge_checkpoints.py -c1 \"${large_init}\" -c2 \"${large_model}\" \\\n      -o ${init_dir}/${t}_sbert_large_all.ckpt -fo \n      gsutil -m cp -r \"${init_dir}/${t}_sbert_large_all.ckpt*\" \"${base_dir}/ckpt/init\"\n    \n      # sbert large init from ebert_large_s0 upper, e.g. 20\n      python tools/merge_checkpoints.py -c1 \"${large_init}\" -c2 \"${large_model}\" \\\n      -o ${init_dir}/${t}_sbert_large_upper20.ckpt -fo -fou 20\n      gsutil -m cp -r \"${init_dir}/${t}_sbert_large_upper20.ckpt*\" \"${base_dir}/ckpt/init\"\n    \n      # sbert base init\n      base_init=\"data/ckpt/separation/${t}/ebert_s0\"\n    \n      gsutil -m cp -r \"${base_dir}/ckpt/separation/${t}/ebert_s0\" data/ckpt/separation/${t}/\n      python tools/merge_checkpoints.py -c1 \"${base_init}\" -c2 \"${base_model}\" \\\n      -o ${init_dir}/${t}_sbert_base.ckpt\n      gsutil -m cp -r \"${init_dir}/${t}_sbert_base.ckpt*\" \"${base_dir}/ckpt/init\"\n    \n      python tools/merge_checkpoints.py -c1 \"${base_init}\" -c2 \"${base_model}\" \\\n      -o ${init_dir}/${t}_sbert_base_all.ckpt -fo \n      gsutil -m cp -r \"${init_dir}/${t}_sbert_base.ckpt*\" \"${base_dir}/ckpt/init\"\n    \n      python tools/merge_checkpoints.py -c1 \"${base_init}\" -c2 \"${base_model}\" \\\n      -o ${init_dir}/${t}_sbert_base_upper9.ckpt -fo -fou 9\n      gsutil -m cp -r \"${init_dir}/${t}_sbert_base.ckpt*\" \"${base_dir}/ckpt/init\"\n    done || exit 1\n    ````\n\n- sbert finetuning: \n\n    ````bash\n    # squad_v1.1, search 50 params for bert large separated at layer 21\n    python tools/explore_hp.py -p data/sbert-squad-large.json -n 50 \\\n    -s large -sp 1.4 0.3 0.8 -hp 5e-5,3,32 2\u003e\u00261 | tee data/sbert-squad-explore-s21.log\n    ./search.sh squad_v1.1 large 21 bert-tpu2\n    \n    # race search 50\n    python tools/explore_hp.py -p data/race-sbert-s9.json -n 50 -t race 2\u003e\u00261 | \\\n    tee data/race-sbert-explore-s9.log\n    \n    ./search.sh race base 9\n    ````\n\n\n### Profiling\n- profile model flops:\n\n    ````bash\n    for task in race boolq race qqp mnli squad_v1.1; do\n      for size in base large; do\n        profile_dir=data/log2-${task}-${size}-profile\n        mkdir -p \"${profile_dir}\"\n              \n        if [[ \"${task}\" == \"mnli\" ]]; then\n          cs=1 # cache_segment\n        else\n          cs=2\n        fi\n\n        if [[ ${size} == \"base\" ]] ; then\n          allowed_layers=\"9 10\" # $(seq 1 1 11)\n          large_params=\"\"\n        else\n          allowed_layers=\"20 21\" #$(seq 1 1 23)\n          large_params=\"hidden_size=1024 intermediate_size=4096 num_heads=16 num_hidden_layers=24\"\n        fi\n\n        if [[ ${task} == \"race\" ]] ; then\n          large_params=\"num_choices=4 ${large_params}\"\n        fi\n\n        # bert \n        eval \"${large_params}\" python profile.py -m bert -t ${task} -pm 2\u003e\u00261 | \\\n        tee ${profile_dir}/bert-profile.log\n\n        # ebert \n        for n in \"${(@s/ /)allowed_layers}\"; do\n          eval \"${large_params}\" sep_layers=\"${n}\" \\\n          python profile.py -m ebert -t ${task} -pm 2\u003e\u00261 | \\\n          tee ${profile_dir}/ebert-s${n}-profile.log\n      \n          eval \"${large_params}\" sep_layers=\"${n}\" \\\n          python profile.py -m ebert -t ${task} -pm -cs ${cs} 2\u003e\u00261 | \\\n          tee ${profile_dir}/ebert-s${n}-profile-cache.log\n        done\n      done\n    done\n    ````\n\n- benchmarking inference latency:\n\n    ````bash\n    python profile.py -npf -pt -b 32 2\u003e\u00261 | tee data/batch-time-bert.log\n    python profile.py -npf -pt -b 32 -m ebert -cs 2 2\u003e\u00261 | tee data/batch-time-ebert.log\n    ````\n\n- analyze bert, ebert, sbert:\n\n    ````bash\n    python analyze.py -o data/qa-outputs -m bert 2\u003e\u00261 | tee data/ana-bert.log\n    python tools/compute_rep_variance.py data/qa-outputs -n 20\n    \n    python tools/compare_rep.py data/qa-outputs -m sbert\n    python tools/compare_rep.py data/qa-outputs -m ebert\n    ````\n\n### Demo\n\n - run infer: `python infer_qa.py -m bert` (add `-e` for eager mode)\n\u003c!-- - serve qa on a server: `python serve_qa.py`, then use `python tools/ask_question.py` --\u003e\n\n### Tools\n\n- `tools/get_dataset_stats.py`: get dataset statistics (length of tokens mainly)\n- `tools/inspect_checkpoint.py`: print variable info in checkpoints (support monitoring variables during training)\n- `tools/rename_checkpoint_variables.py`: rename variable names in checkpoint (add `-dr` for dry run)\ne.g. `python tools/rename_checkpoint_variables.py \"data/ckpt/bert/mnli/\" -p \"bert_mnli\" \"mnli\" -dr`\n- `tools/visualize_model.py`: visualize TensorFlow model structure given inference graph\n\n### Handy Commands\n\n- redis\n\n  ````bash\n  redis-cli -p 60001 lrange queue:params 0 -1\n  redis-cli -p 60001 lrange queue:results 0 -1\n  redis-cli -p 60001 lpop queue:params\n  redis-cli -p 60001 rpush queue:results 89.532\n  ````\n  \n- gcloud sdk for TPU access: `pip install --upgrade google-api-python-client oauth2client`\n\n- TPU start: `ctpu up --tpu-size=v3-8 --tpu-only --name=bert-tpu --noconf` \n(can support tf version, e.g.`--tf-version=1.13`)\n\n- TPU stop: `ctpu pause  --tpu-only --name=bert-tpu --noconf`\n\n\n- move instances: `gcloud compute instances move bert-vm --zone us-central1-b --destination-zone us-central1-a`\n\n- upload and download: \n  ````bash\n  cd data\n  # upload\n  gsutil -m cp -r datasets/qqp/ebert \"gs://xxx/datasets/qqp/ebert\"\n  gsutil -m cp -r datasets/qa/ebert \"gs://xxx/datasets/qa/ebert\"\n  gsutil -m cp -r datasets/mnli/ebert \"gs://xxx/datasets/mnli/ebert\"\n  gsutil -m cp -r \"datasets/qa/bert/hotpot-*\" \"gs://xxx/datasets/qa/bert\"\n\n  # download\n  gsutil -m cp -r \"gs://xxx/datasets/qqp/ebert\" qqp/ebert\n  \n  cd data/ckpt\n  # download\n  gsutil -m cp -r \"gs://xxx/ckpt/bert/qa/model.ckpt-8299*\" bert/qa/\n  gsutil -m cp -r \"gs://xxx/ckpt/ebert_s9/qa/model.ckpt-8321*\" ebert_s9/qa/\n  gsutil -m cp -r \"gs://xxx/ckpt/ebert_s9/mnli/model.ckpt-18407*\" ebert_s9/mnli/\n  gsutil -m cp -r \"gs://xxx/ckpt/ebert_s9/qqp/model.ckpt-17055*\" ebert_s9/qqp/\n  \n  function dl()\n  {\n    num=$2\n    for suffix in meta index data-00000-of-00001; do\n      gsutil cp gs://xxx/ckpt/$1/model.ckpt-${num}.${suffix} .\n    done;\n    echo model_checkpoint_path: \\\"model.ckpt-${num}\\\" \u003e checkpoint\n  }\n  \n  ````\n\n## FAQ\n\nIf you have any question, please create an issue.\n\n## Citation\n\nIf you find our work useful to your research, please consider using the following citation:\n\n````bib\n@inproceedings{cao-etal-2020-deformer,\n    title = \"{D}e{F}ormer: Decomposing Pre-trained Transformers for Faster Question Answering\",\n    author = \"Cao, Qingqing  and\n      Trivedi, Harsh  and\n      Balasubramanian, Aruna  and\n      Balasubramanian, Niranjan\",\n    booktitle = \"Proceedings of the 58th Annual Mdeformering of the Association for Computational Linguistics\",\n    month = jul,\n    year = \"2020\",\n    address = \"Online\",\n    publisher = \"Association for Computational Linguistics\",\n    url = \"https://www.aclweb.org/anthology/2020.acl-main.411\",\n    pages = \"4487--4497\",\n}\n````\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FStonyBrookNLP%2Fdeformer","html_url":"https://awesome.ecosyste.ms/projects/github.com%2FStonyBrookNLP%2Fdeformer","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FStonyBrookNLP%2Fdeformer/lists"}