https://github.com/lucidrains/lambda-networks

Implementation of LambdaNetworks, a new approach to image recognition that reaches SOTA with less compute
https://github.com/lucidrains/lambda-networks

artificial-intelligence attention attention-mechanism computer-vision deep-learning

Last synced: 18 days ago
JSON representation

Implementation of LambdaNetworks, a new approach to image recognition that reaches SOTA with less compute

• Host: GitHub
• URL: https://github.com/lucidrains/lambda-networks
• Owner: lucidrains
• License: mit
• Created: 2020-10-08T19:01:15.000Z (about 4 years ago)
• Default Branch: main
• Last Pushed: 2020-11-18T19:54:34.000Z (almost 4 years ago)
• Last Synced: 2024-10-15T00:16:54.819Z (25 days ago)
• Topics: artificial-intelligence, attention, attention-mechanism, computer-vision, deep-learning
• Language: Python
• Homepage:
• Size: 221 KB
• Stars: 1,533
• Watchers: 46
• Forks: 157
• Open Issues: 13
• Metadata Files:
  • Readme: README.md
  • License: LICENSE

Awesome Lists containing this project

• awesome-python-machine-learning-resources - GitHub - 46% open · ⏱️ 18.11.2020): (Pytorch实用程序)

README

        


## Lambda Networks - Pytorch

Implementation of λ Networks, a new approach to image recognition that reaches SOTA on ImageNet. The new method utilizes λ layer, which captures interactions by transforming contexts into linear functions, termed lambdas, and applying these linear functions to each input separately.

Yannic Kilcher's paper review

## Install

```bash

$ pip install lambda-networks

```

## Usage

Global context

```python

import torch

from lambda_networks import LambdaLayer

layer = LambdaLayer(

    dim = 32,       # channels going in

    dim_out = 32,   # channels out

    n = 64,         # size of the receptive window - max(height, width)

    dim_k = 16,     # key dimension

    heads = 4,      # number of heads, for multi-query

    dim_u = 1       # 'intra-depth' dimension

)

x = torch.randn(1, 32, 64, 64)

layer(x) # (1, 32, 64, 64)

```

Localized context

```python

import torch

from lambda_networks import LambdaLayer

layer = LambdaLayer(

    dim = 32,

    dim_out = 32,

    r = 23,         # the receptive field for relative positional encoding (23 x 23)

    dim_k = 16,

    heads = 4,

    dim_u = 4

)

x = torch.randn(1, 32, 64, 64)

layer(x) # (1, 32, 64, 64)

```

For fun, you can also import this as follows

```python

from lambda_networks import λLayer

```

## Tensorflow / Keras version

Shinel94 has added a Keras implementation! It won't be officially supported in this repository, so either copy / paste the code under `./lambda_networks/tfkeras.py` or make sure to install `tensorflow` and `keras` before running the following.

```python

import tensorflow as tf

from lambda_networks.tfkeras import LambdaLayer

layer = LambdaLayer(

    dim_out = 32,

    r = 23,

    dim_k = 16,

    heads = 4,

    dim_u = 1

)

x = tf.random.normal((1, 64, 64, 16)) # channel last format

layer(x) # (1, 64, 64, 32)

```

## Citations

```bibtex

@inproceedings{

    anonymous2021lambdanetworks,

    title={LambdaNetworks: Modeling long-range Interactions without Attention},

    author={Anonymous},

    booktitle={Submitted to International Conference on Learning Representations},

    year={2021},

    url={https://openreview.net/forum?id=xTJEN-ggl1b},

    note={under review}

}

```