https://github.com/titu1994/keras-multiplicative-lstm
Miltiplicative LSTM for Keras 2.0+
https://github.com/titu1994/keras-multiplicative-lstm
Last synced: 7 months ago
JSON representation
Miltiplicative LSTM for Keras 2.0+
- Host: GitHub
- URL: https://github.com/titu1994/keras-multiplicative-lstm
- Owner: titu1994
- Created: 2017-08-17T21:17:52.000Z (about 8 years ago)
- Default Branch: master
- Last Pushed: 2017-11-04T03:05:00.000Z (almost 8 years ago)
- Last Synced: 2025-03-18T09:02:37.221Z (7 months ago)
- Language: Python
- Size: 18.8 MB
- Stars: 42
- Watchers: 3
- Forks: 21
- Open Issues: 3
-
Metadata Files:
- Readme: README.md
Awesome Lists containing this project
README
# Multiplicative LSTM in Keras
Implementation of the paper [Multiplicative LSTM for sequence modelling](https://arxiv.org/pdf/1609.07959.pdf) for Keras 2.0+.
Multiplicative LSTMs have been shown to achieve state-of-the-art or close to SotA results for sequence modelling datasets. They also perform better than stacked LSTM models for the Hutter-prize dataset and the raw wikipedia dataset.
## Equation for mLSTM
From the paper, the change in the equations of the general LSTM are :
The size of m_t selected is same as that of h_t, therefore the size of all mLSTM models should be 1.25 times that of the equivalent LSTM model.
## Usage
Add the `multiplicative_lstm.py` script into your repository, and import the MultiplicativeLSTM layer.
Eg. You can replace Keras LSTM layers with MultiplicativeLSTM layers.
```
from multiplicative_lstm import MultiplicativeLSTM
```
## Comparison to LSTM on IMDB dataset
While IMDB is not an idea dataset to compare LSTM models since they overfit rapidly, the weights for the two models have been provided to show the comparison.
1) LSTM Score : 83.20% (overfits after 3 epochs)
2) mLSTM Score : 83.27% (overfits after 7 epochs)