https://github.com/facebookresearch/SpanBERT
Code for using and evaluating SpanBERT.
https://github.com/facebookresearch/SpanBERT
Last synced: 18 days ago
JSON representation
Code for using and evaluating SpanBERT.
- Host: GitHub
- URL: https://github.com/facebookresearch/SpanBERT
- Owner: facebookresearch
- License: other
- Created: 2019-08-30T18:11:50.000Z (over 5 years ago)
- Default Branch: main
- Last Pushed: 2023-07-25T16:49:50.000Z (over 1 year ago)
- Last Synced: 2025-03-27T19:15:46.043Z (23 days ago)
- Language: Python
- Size: 329 KB
- Stars: 896
- Watchers: 18
- Forks: 176
- Open Issues: 38
-
Metadata Files:
- Readme: README.md
- Contributing: CONTRIBUTING.md
- License: LICENSE
- Code of conduct: CODE_OF_CONDUCT.md
Awesome Lists containing this project
- awesome-bert - facebookresearch/SpanBERT
- awesome-transformer-nlp - facebookresearch/SpanBERT - Question Answering on SQuAD; improving pre-training by representing and predicting spans. (Tasks / Question Answering (QA))
README
# SpanBERT
This repository contains code and models for the paper: [SpanBERT: Improving Pre-training by Representing and Predicting Spans](https://arxiv.org/abs/1907.10529). If you prefer to use Huggingface, please check out this link -- https://huggingface.co/SpanBERT## Requirements
## Apex
Please use an earlier commit of Apex - [NVIDIA/apex@4a8c4ac](https://github.com/NVIDIA/apex/commit/4a8c4ac088b6f84a10569ee89db3a938b48922b4)## Pre-trained Models
We release both base and large *cased* models for SpanBERT. The base & large models have the same model configuration as [BERT](https://github.com/google-research/bert) but they differ in
both the masking scheme and the training objectives (see our paper for more details).* [SpanBERT (base & cased)](https://dl.fbaipublicfiles.com/fairseq/models/spanbert_hf_base.tar.gz): 12-layer, 768-hidden, 12-heads , 110M parameters
* [SpanBERT (large & cased)](https://dl.fbaipublicfiles.com/fairseq/models/spanbert_hf.tar.gz): 24-layer, 1024-hidden, 16-heads, 340M parametersThese models have the same format as the [HuggingFace BERT models](https://github.com/huggingface/pytorch-transformers), so you can easily replace them with our SpanBET models. If you would like to use our [fine-tuning code](#fine-tuning), the model paths are already hard-coded in the code :)
| | SQuAD 1.1 | SQuAD 2.0 | Coref | TACRED |
| ---------------------- | ------------- | --------- | ------- | ------ |
| | F1 | F1 | avg. F1 | F1 |
| BERT (base) | 88.5* | 76.5* | 73.1 | 67.7 |
| SpanBERT (base) | 92.4* | 83.6* | 77.4 | 68.2 |
| BERT (large) | 91.3 | 83.3 | 77.1 | 66.4 |
| SpanBERT (large) | 94.6 | 88.7 | 79.6 | 70.8 |Note: The numbers marked as * are evaluated on the development sets because we didn't submit those models to the official SQuAD leaderboard. All the other numbers are test numbers.
## Fine-tuning
### SQuAD 1.1
```bash
python code/run_squad.py \
--do_train \
--do_eval \
--model spanbert-base-cased \
--train_file train-v1.1.json \
--dev_file dev-v1.1.json \
--train_batch_size 32 \
--eval_batch_size 32 \
--learning_rate 2e-5 \
--num_train_epochs 4 \
--max_seq_length 512 \
--doc_stride 128 \
--eval_metric f1 \
--output_dir squad_output \
--fp16
```### SQuAD 2.0
```bash
python code/run_squad.py \
--do_train \
--do_eval \
--model spanbert-base-cased \
--train_file train-v2.0.json \
--dev_file dev-v2.0.json \
--train_batch_size 32 \
--eval_batch_size 32 \
--learning_rate 2e-5 \
--num_train_epochs 4 \
--max_seq_length 512 \
--doc_stride 128 \
--eval_metric best_f1 \
--output_dir squad2_output \
--version_2_with_negative \
--fp16
```### TACRED
```bash
python code/run_tacred.py \
--do_train \
--do_eval \
--data_dir \
--model spanbert-base-cased \
--train_batch_size 32 \
--eval_batch_size 32 \
--learning_rate 2e-5 \
--num_train_epochs 10 \
--max_seq_length 128 \
--output_dir tacred_dir \
--fp16
```### MRQA (NewsQA, TriviaQA, SearchQA, HotpotQA, NaturalQuestions)
```bash
python code/run_mrqa.py \
--do_train \
--do_eval \
--model spanbert-base-cased \
--train_file TriviaQA-train.jsonl.gz \
--dev_file TriviaQA-dev.jsonl.gz \
--train_batch_size 32 \
--eval_batch_size 32 \
--learning_rate 2e-5 \
--num_train_epochs 4 \
--max_seq_length 512 \
--doc_stride 128 \
--eval_per_epoch 5 \
--output_dir triviaqa_dir \
--fp16
```### GLUE
```bash
python code/run_glue.py \
--task_name RTE \
--model spanbert-base-cased \
--do_train \
--do_eval \
--data_dir \
--train_batch_size 32 \
--eval_batch_size 32 \
--num_train_epochs 10 \
--max_seq_length 128 \
--learning_rate 2e-5 \
--output_dir RTE_DIR \
--fp16
```### Coreference Resolution
Our coreference resolution fine-tuning code is implemented in Tensorflow. Please see [https://github.com/mandarjoshi90/coref](https://github.com/mandarjoshi90/coref) for more details.## Finetuned Models (SQuAD 1.1/2.0, Relation Extraction, Coreference Resolution)
If you are interested in using our fine-tuned models for downstream tasks, directly, please use the following script.
```
./code/download_finetuned.sh
```
where `` is one of `[squad1, squad2, tacred]`. You can evaluate the models by setting `--do_train` to `false`, `--do_eval` to `true`, and `--output_dir` to `/` in `python code/run_.py`.For coreference resolution, please refer to this repository -- [https://github.com/mandarjoshi90/coref](https://github.com/mandarjoshi90/coref)
## Citation
```
@article{joshi2019spanbert,
title={{SpanBERT}: Improving Pre-training by Representing and Predicting Spans},
author={Mandar Joshi and Danqi Chen and Yinhan Liu and Daniel S. Weld and Luke Zettlemoyer and Omer Levy},
journal={arXiv preprint arXiv:1907.10529},
year={2019}
}
```## License
SpanBERT is CC-BY-NC 4.0. The license applies to the pre-trained models as well.## Contact
If you have any questions, please contact Mandar Joshi `` or Danqi Chen `` or create a Github issue.