https://github.com/kaushalshetty/Structured-Self-Attention

A Structured Self-attentive Sentence Embedding
https://github.com/kaushalshetty/Structured-Self-Attention

attention attention-mechanism attention-model attention-weights classification deep-learning python3 pytorch self-attention self-attentive-rnn sentence-embeddings visualization

Last synced: 2 months ago
JSON representation

A Structured Self-attentive Sentence Embedding

• Host: GitHub
• URL: https://github.com/kaushalshetty/Structured-Self-Attention
• Owner: kaushalshetty
• License: mit
• Created: 2018-02-01T07:38:55.000Z (almost 7 years ago)
• Default Branch: master
• Last Pushed: 2019-09-22T21:00:24.000Z (over 5 years ago)
• Last Synced: 2024-08-05T06:04:58.866Z (6 months ago)
• Topics: attention, attention-mechanism, attention-model, attention-weights, classification, deep-learning, python3, pytorch, self-attention, self-attentive-rnn, sentence-embeddings, visualization
• Language: Python
• Homepage:
• Size: 492 KB
• Stars: 492
• Watchers: 13
• Forks: 108
• Open Issues: 2
• Metadata Files:
  • Readme: README.md
  • License: LICENSE

Awesome Lists containing this project

README

        
# Structured Self-attentive sentence embeddings 

Implementation for the paper A Structured Self-Attentive Sentence Embedding, which was published in ICLR 2017: https://arxiv.org/abs/1703.03130 .

#### USAGE:

For binary sentiment classification on imdb dataset run :

`python classification.py "binary"`

For multiclass classification on reuters dataset run :

`python classification.py "multiclass"`

You can change the model parameters in the `model_params.json file`

Other tranining parameters like number of attention hops etc can be configured in the `config.json` file.

If you want to use pretrained glove embeddings , set the `use_embeddings` parameter to `"True"` ,default is set to False. Do not forget to download the `glove.6B.50d.txt` and place it in the glove folder.

#### Implemented:

* Classification using self attention

* Regularization using Frobenius norm

* Gradient clipping

* Visualizing the attention weights

Instead of pruning ,used averaging over the sentence embeddings.

#### Visualization:

After training, the model is tested on 100 test points. Attention weights for the 100 test data are retrieved and used to visualize over the text using heatmaps. A file visualization.html gets saved in the visualization/ folder after successful training. The visualization code was provided by Zhouhan Lin (@hantek). Many thanks.

Below is a shot of the visualization on few datapoints.

![alt text](https://github.com/kaushalshetty/Structured-Self-Attention/blob/master/visualization/attention.png "Attention Visualization")

Training accuracy 93.4%

Tested on 1000 points with 90.2% accuracy

---