https://github.com/jennalandy/mt_augment_pos

Machine Translation Data Augmentation Methods Maintaining Part of Speech
https://github.com/jennalandy/mt_augment_pos

Last synced: 3 months ago
JSON representation

Machine Translation Data Augmentation Methods Maintaining Part of Speech

• Host: GitHub
• URL: https://github.com/jennalandy/mt_augment_pos
• Owner: jennalandy
• Created: 2021-12-02T23:56:15.000Z (over 3 years ago)
• Default Branch: main
• Last Pushed: 2021-12-11T20:31:18.000Z (over 3 years ago)
• Last Synced: 2025-02-20T02:22:58.768Z (4 months ago)
• Language: Jupyter Notebook
• Size: 2.35 MB
• Stars: 0
• Watchers: 1
• Forks: 2
• Open Issues: 0
• Metadata Files:
  • Readme: README.md

Awesome Lists containing this project

README

        
# No Syntaxation Without Representation: Syntactic Considerations for Neural Machine Translation Data Augmentation

Paper: `NoSyntaxationWithoutRepresentation.pdf`

# files and descriptions

- notebooks obtain the data:

  - `Download_dataset_iwslt2017.ipynb`: download and produce 10% of the data for the paper

- notebooks to train models:

  - `TrainLSTM.ipynb`: all LSTM methods excluding sequence matching methods

  - `similarity_ds_k=2.ipynb` and `similarity_ds_k=10.ipynb`: LSTM sequence matching methods using similarity

  - `TrainTransformer.ipynb`: all Transformer methods

  - `LanguageModel.ipynb`: training the language model used in `LMsample` and `soft` methods

- other notebooks:

  - `BEAM_BLEU.ipynb`: *evaluation*, re-compute BLEU score with beam search, compute POS BLEU score

  - `LM_POS_Experiments.ipynb`: *experiment*, looking at how well the language model matches part of speech

  - `CustomTransformer.ipynb`: *development*, developing and testing the transformer architecture, contains links to transformer resources

- functions for transformer models:

  - `embeddingTF.py`: `Embedder` and `PositionalEncoding`

  - `sublayersTF.py`: `SublayerConnection` (layer norm & residual connection), `FeedForward`, `attention`, `MultiHeadedAttention`, and `clones` (replicates layers)

  - `layersTF.py`: `EncoderLayer` and `DecoderLayer`

  - `stacksTF.py`: `Encoder` and `Decoder`, which construct the encoder and decoder stacks from the encoder and decoder layers, respectively

  - `encoderTF.py`: `FullEncoder`, which allows for augmentation to occur in the embedding - positional encoding - encoder structure

  - `decoderTF.py`: `FullDecoder`, which allows for augmentation to occur in the embedding - positional encoding - decoder structure

  - `seq2seqTF.py`: `Seq2SeqTF`, which contains the custom encoder and decoders and fully defines the transformer seq2seq model

  - `batchTF.py`: `BatchTF`, which formats source and target inputs to yield shifted targets, source mask, and target mask (`future_mask` provides decoder-specific masking)

  - `trainTF.py`: `train`, which uses `train_epoch` and `val_epoch` to create the training scheme, `greedy_decode`, and `translate_corpus`

- functions for lstm models:

  - `train.py`: training functions for LSTM models

  - `Seq2Seq.py`: model class

  - `EncoderLSTM.py`: encoder class, including functions for all augmentations

  - `DecoderLSTM.py`: decoder class, including functions for seqmix augmentations

- other functions:

  - `load_data.py`: creating and loading pickled datasets and dataloaders

  - `load_lm.py`: load the language model developed in `LanguageModel.ipynb`

# data access

## option 1. download and re-build dataloaders

- Download the full data from torchtext:

`from torchtext.datasets import IWSLT2017`

`train_iter, valid_iter, test_iter = IWSLT2017(root='.data', split=('train', 'valid', 'test'), language_pair=('de', 'en'))`

- Run `Download_dataset_iwslt2017.ipynb` to get 10% sample of dataset and save as pickles

- Run `load_and_save(batch1 = True)` from `load_data.py` to build the dataloaders used in our LSTM models and save to pickle files

- Run `load_and_save(batch1 = False)` from `load_data.py` to build the dataloaders used in our Transformer models and save to pickle files

- In our code, we use `load_pickled_dataloaders(batch1 = True)` and `load_pickled_dataloaders(batch1 = False)` from `load_data.py` to load dataloaders from the pickle files. You'll need to pass in `PARENT_DIR` as the location of your `data` folder.

## option 2. use our pre-built dataloaders

- Save download the following directories and save to your own `data` folder

  - `dataloaders10perc`: used for LSTM models and the LM https://drive.google.com/drive/folders/18K6XpYgTmLZkPtLQw4-8gqUyeAGWPF-u?usp=sharing

  - `dataloaders10perc_batchsize32`: used for transformer models, larger batch size https://drive.google.com/drive/folders/16_hx53i473FjJfn4sfdLQ4ZTdxDehUBT?usp=sharing

- In our code, we use `load_pickled_dataloaders(batch1 = True)` and `load_pickled_dataloaders(batch1 = False)` from `load_data.py` to load dataloaders from the pickle files for the LSTM and transformer, respectively. You'll need to pass in `PARENT_DIR` as the location of your `data` folder.

# Packages

- `sys` 3.7.12

- `tqdm` 4.62.3

- `numpy` 1.19.5

- `matplotlib` 3.2.2

- `pandas` 1.1.5

- `torch` 1.10.0+cu111

- `torchtext` 0.11.0

- `spacy` 2.2.4

- `transformers` 4.6.0

- `sentence_transformers` 2.1.0

- `os` *

- `typing` *

- `pickle` *

- `timeit` *

- `operator` *

- `collections` *

- `copy` *

- `random` *

- `math` *

\* (Python 3.6.9 Standard Library)