https://github.com/timbmg/structured-self-attentive-sentence-embedding

Re-Implementation of "A Structured Self-Attentive Sentence Embedding" by Lin et al., 2017
https://github.com/timbmg/structured-self-attentive-sentence-embedding

attention deep-learning machine-learning neural-networks nlp pytorch recurrent-neural-networks self-attention self-attentive-rnn sentiment-analysis text-classification vizualisation yelp-dataset

Last synced: 20 days ago
JSON representation

Re-Implementation of "A Structured Self-Attentive Sentence Embedding" by Lin et al., 2017

• Host: GitHub
• URL: https://github.com/timbmg/structured-self-attentive-sentence-embedding
• Owner: timbmg
• Created: 2018-07-15T12:51:18.000Z (over 6 years ago)
• Default Branch: master
• Last Pushed: 2024-08-21T08:36:06.000Z (3 months ago)
• Last Synced: 2024-10-12T18:31:05.078Z (about 1 month ago)
• Topics: attention, deep-learning, machine-learning, neural-networks, nlp, pytorch, recurrent-neural-networks, self-attention, self-attentive-rnn, sentiment-analysis, text-classification, vizualisation, yelp-dataset
• Language: Python
• Homepage: https://arxiv.org/pdf/1703.03130.pdf
• Size: 690 KB
• Stars: 24
• Watchers: 3
• Forks: 1
• Open Issues: 1
• Metadata Files:
  • Readme: README.md

Awesome Lists containing this project

README

        
# A Structured Self-Attentive Sentence Embedding

Re-Implementation of [_A Structured Self-Attentive Sentence Embedding_](https://arxiv.org/pdf/1703.03130.pdf) by Lin et al., 2017.

## Results

| Set        | Loss     | Accuracy |

|:-----------|---------:|---------:|

| Training   | 1.136    | 77.26%   |

| Validation | 1.587    | 60.91%   |

Above results were obtained after training for 5 epochs. The training set contained 20000 examples and validation set 1000 examples. The model with the best validation loss was choosen. Note that the training set size in the paper is much bigger.

## Data

The Yelp dataset can be download [here](http://pan.webis.de/clef16/pan16-web/author-profiling.html). 

After downloading, the file only has to be unzipped.

## Training

You can run the training procedure with the default settings with the following command:  

`python3 train.py --data-dir `

For more information about training settings run:  

`python3 train.py --help`

## Analysis & Visualization

Once the model is trained, the attention pattern can be visualized as done in the paper.

The following python script will create an HTML file with the reviews and respective attention pattern. 

Also the confusion matrix for the classification will be created.  

`python3 viz.py --html --cm --data-dir  --validation-set `

### Attention Pattern 

![](imgs/viz2.png)

### Confusion Matrix

![](imgs/cm.png)

## Differences with the paper

- Adam instead of SGD

- No gradient clipping

- No dropout

- No GLOVE word embedding initialization

## Requirements

- Implemented and tested with python 3.6.5

- Python library versions can be found in [requirements.txt](requirements.txt)