https://github.com/lucidrains/mogrifier

Usable implementation of Mogrifier, a circuit for enhancing LSTMs and potentially other networks, from Deepmind
https://github.com/lucidrains/mogrifier

artificial-intelligence deep-learning xlstm

Last synced: about 1 year ago
JSON representation

Usable implementation of Mogrifier, a circuit for enhancing LSTMs and potentially other networks, from Deepmind

• Host: GitHub
• URL: https://github.com/lucidrains/mogrifier
• Owner: lucidrains
• License: mit
• Created: 2020-07-05T04:02:36.000Z (about 6 years ago)
• Default Branch: master
• Last Pushed: 2024-06-09T21:54:00.000Z (about 2 years ago)
• Last Synced: 2025-03-28T22:24:32.572Z (over 1 year ago)
• Topics: artificial-intelligence, deep-learning, xlstm
• Language: Python
• Homepage:
• Size: 160 KB
• Stars: 17
• Watchers: 3
• Forks: 2
• Open Issues: 0
• Metadata Files:
  • Readme: README.md
  • License: LICENSE

Awesome Lists containing this project

README

          


[![PyPI version](https://badge.fury.io/py/mogrifier.svg)](https://badge.fury.io/py/mogrifier)

## Mogrifier

A complete implementation of Mogrifier, a circuit for enhancing LSTMs and potentially other networks. It allows two vectors to modulate each other by having each gate the other in an interleaved, iterative fashion.

## Install

```bash

$ pip install mogrifier

```

## Usage

```python

import torch

from mogrifier import Mogrifier

mogrify = Mogrifier(

    dim = 512,

    dim_hidden = 256,

    iters = 5,          # number of iterations, defaults to 5 as paper recommended for LSTM

    factorize_k = 16    # factorize weight matrices into (dim x k) and (k x dim), if specified

)

x = torch.randn(1, 16, 512)

h = torch.randn(1, 16, 256)

out, hidden_out = mogrify(x, h) # (1, 16, 512), (1, 16, 256)

assert out.shape == x.shape

assert hidden_out.shape == h.shape

```

## Citation

```bibtex

@inproceedings{Melis2020Mogrifier,

    title   = {Mogrifier LSTM},

    author  = {Gábor Melis and Tomáš Kočiský and Phil Blunsom},

    booktitle = {International Conference on Learning Representations},

    year    = {2020},

    url     = {https://openreview.net/forum?id=SJe5P6EYvS}

}

```