https://github.com/lucidrains/deep-linear-network
A simple implementation of a deep linear Pytorch module
https://github.com/lucidrains/deep-linear-network
artificial-intelligence deep-learning pytorch
Last synced: about 1 year ago
JSON representation
A simple implementation of a deep linear Pytorch module
- Host: GitHub
- URL: https://github.com/lucidrains/deep-linear-network
- Owner: lucidrains
- License: mit
- Created: 2020-10-16T07:01:39.000Z (over 5 years ago)
- Default Branch: main
- Last Pushed: 2020-10-16T20:02:06.000Z (over 5 years ago)
- Last Synced: 2025-06-14T06:01:49.599Z (about 1 year ago)
- Topics: artificial-intelligence, deep-learning, pytorch
- Language: Python
- Homepage:
- Size: 59.6 KB
- Stars: 21
- Watchers: 2
- Forks: 3
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
- License: LICENSE
Awesome Lists containing this project
README
## Deep Linear Network - Pytorch
A simple to use deep linear network module. Useful for matrix factorization or for passing an input tensor through a series of square weight matrices, where it was discovered that gradient descent implicitly regularizes the output to low-rank solutions.
LeCun's paper uses this unique property to optimize the latent of an autoencoder to be low-rank.
The module will take care of collapsing the linear weight matrices into one weight matrix, caching it across evaluation calls (but expired on training).
## Install
```bash
$ pip install deep-linear-network
```
## Usage
Matrix factorization
```python
import torch
from deep_linear_network import DeepLinear
x = torch.randn(1, 1024, 256)
linear = DeepLinear(256, 10, 512) # w1 (256 x 10) @ w2 (10 x 512)
linear(x) # (1, 1024, 512)
```
Deep Linear Network
```python
import torch
from deep_linear_network import DeepLinear
x = torch.randn(1, 1024, 256)
linear = DeepLinear(256, 256, 256, 256, 128) # w1-w3 (256 x 256) w4 (256 x 128)
linear(x) # (1, 1024, 128)
```
## Citations
```bibtex
@misc{arora2019implicit,
title={Implicit Regularization in Deep Matrix Factorization},
author={Sanjeev Arora and Nadav Cohen and Wei Hu and Yuping Luo},
year={2019},
eprint={1905.13655},
archivePrefix={arXiv},
primaryClass={cs.LG}
}
```
```bibtex
@misc{jing2020implicit,
title={Implicit Rank-Minimizing Autoencoder},
author={Li Jing and Jure Zbontar and Yann LeCun},
year={2020},
eprint={2010.00679},
archivePrefix={arXiv},
primaryClass={cs.LG}
}
```