https://github.com/prashantranjan09/structured-self-attentive-sentence-embedding

Implementation of the Paper Structured Self-Attentive Sentence Embedding published in ICLR 2017
https://github.com/prashantranjan09/structured-self-attentive-sentence-embedding

attention attention-mechanism deep-learning iclr2017 pytorch self-attention self-attentive-rnn text-classification

Last synced: about 2 months ago
JSON representation

Implementation of the Paper Structured Self-Attentive Sentence Embedding published in ICLR 2017

• Host: GitHub
• URL: https://github.com/prashantranjan09/structured-self-attentive-sentence-embedding
• Owner: PrashantRanjan09
• Created: 2018-07-28T21:13:09.000Z (about 7 years ago)
• Default Branch: master
• Last Pushed: 2018-07-28T21:45:11.000Z (about 7 years ago)
• Last Synced: 2024-04-27T23:38:39.286Z (over 1 year ago)
• Topics: attention, attention-mechanism, deep-learning, iclr2017, pytorch, self-attention, self-attentive-rnn, text-classification
• Language: Python
• Size: 279 KB
• Stars: 1
• Watchers: 2
• Forks: 2
• Open Issues: 0
• Metadata Files:
  • Readme: README.md

Awesome Lists containing this project

README

          
# Structured Self-Attentive Sentence Embedding

This is an implementation of the paper: https://arxiv.org/pdf/1703.03130.pdf published in ICLR 2017.

This paper proposes a new model for extracting an interpretable sentence embedding by introducing _self-attention_. Instead of using a vector, the paper use a 2-D matrix to represent the embedding, with each row of the matrix attending on a different part of the sentence.It also propose a self-attention mechanism.

![Optional Text](../master/self-attention.png)

The implementation is done on the imdb dataset with the following parameters:

    top_words = 10000

    learning_rate =0.001

    max_seq_len = 200

    emb_dim = 300

    batch_size=500

    u=64

    da = 32

    r= 16

    

**top_words** : only consider the top 10,000 most common words 


**u**: hidden unit number for each unidirectional LSTM


**da** : a hyperparameter we can set arbitrarily. 


**r** : no. of different parts to be extracted from the sentence.


### To Run:

    python self-attention.py

Running this for 4 epochs gives a training accuracy of `94%` and test accuracy of `87%`.

### To Do :

Penalization term 


results on other datasets