https://github.com/stonesjtu/basis-embedding
basis embedding: a product quantization based model compression method for language models.
https://github.com/stonesjtu/basis-embedding
language-model model-compression product-quantization pytorch
Last synced: 11 months ago
JSON representation
basis embedding: a product quantization based model compression method for language models.
- Host: GitHub
- URL: https://github.com/stonesjtu/basis-embedding
- Owner: Stonesjtu
- Created: 2017-07-31T08:35:27.000Z (almost 9 years ago)
- Default Branch: master
- Last Pushed: 2024-10-29T05:53:50.000Z (over 1 year ago)
- Last Synced: 2025-03-25T06:41:32.819Z (about 1 year ago)
- Topics: language-model, model-compression, product-quantization, pytorch
- Language: Python
- Homepage:
- Size: 45.7 MB
- Stars: 6
- Watchers: 3
- Forks: 0
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
Awesome Lists containing this project
README
## basis embedding
> code for Structured Word Embedding for Low Memory Neural Network Language Model
The code repo for basis embedding to reduce model size and memory consumption
This repo is built based on the pytorch/examples repo on github
### Parameters Introduction
basis embedding related arguments:
- `--basis` <0>: number of basis to decompose the embedding matrix, 0 is normal mode
- `--num_clusters`: number of clusters for all the vocabulary
- `--load_input_embedding`: path of pre-trained embedding matrix for input embedding
- `--load_output_embedding`: path of pre-trained embedding matrix for output embedding
misc options:
- `-c` or `--config`: the path for configuration file, it will override arguments parser's
default values and be overrided by command line options
- `--train`: train or just evaluation existing model
- `--dict `: use vocabulary file if specified, otherwise use the words in train.txt
### examples
```bash
python main.py -c config/default.conf # train a cross-entropy baseline
python main.py -c config/ptb_basis_tied.conf # basis embedding inited via tied embedding on ptb
```
During training, if a keyboard interrupt (Ctrl-C) is received,
training is stopped and the current model is evaluted against the test dataset.
The `main.py` script accepts the following arguments:
```bash
optional arguments:
-h, --help show this help message and exit
-c, --config PATH preset configurations to load
--data DATA location of the data corpus
--model MODEL type of recurrent net (RNN_TANH, RNN_RELU, LSTM, GRU)
--emsize EMSIZE size of word embeddings
--nhid NHID humber of hidden units per layer
--nlayers NLAYERS number of layers
--lr LR initial learning rate
--clip CLIP gradient clipping
--epochs EPOCHS upper epoch limit
--batch-size N batch size
--dropout DROPOUT dropout applied to layers (0 = no dropout)
--tied tie the word embedding and softmax weights
--seed SEED random seed
--cuda use CUDA
--log-interval N report interval
--save SAVE path to save the final model
... more from previous basis embedding related parameters
```
### File Hierarchy
- main.py: the entry file, it parses the parameters, defines models
and feeds the data to model
- model.py: define the input embedding and LSTM layer
- basis_loss.py: It contains a basis linear module, taking inputs from LSTM hidden state and outputing loss value.
- basis/: core part of the basis embedding module
- utils.py: do product quantization for pre-trained embedding
- data.py: data pre-processing
- .th/.th.decoder: the pre-trained embedding matrix
- .conf: sample configuration files