https://github.com/lucidrains/halonet-pytorch
Implementation of the 😇 Attention layer from the paper, Scaling Local Self-Attention For Parameter Efficient Visual Backbones
https://github.com/lucidrains/halonet-pytorch
artificial-intelligence attention-mechanism deep-learning vision
Last synced: about 1 year ago
JSON representation
Implementation of the 😇 Attention layer from the paper, Scaling Local Self-Attention For Parameter Efficient Visual Backbones
- Host: GitHub
- URL: https://github.com/lucidrains/halonet-pytorch
- Owner: lucidrains
- License: mit
- Created: 2021-03-24T17:50:25.000Z (over 5 years ago)
- Default Branch: main
- Last Pushed: 2021-03-24T23:51:16.000Z (over 5 years ago)
- Last Synced: 2025-04-09T15:03:16.879Z (about 1 year ago)
- Topics: artificial-intelligence, attention-mechanism, deep-learning, vision
- Language: Python
- Homepage:
- Size: 73.2 KB
- Stars: 198
- Watchers: 7
- Forks: 21
- Open Issues: 4
-
Metadata Files:
- Readme: README.md
- License: LICENSE
Awesome Lists containing this project
README
## HaloNet - Pytorch
Implementation of the Attention layer from the paper, Scaling Local Self-Attention For Parameter Efficient Visual Backbones. This repository will only house the attention layer and not much more.
## Install
```bash
$ pip install halonet-pytorch
```
## Usage
```python
import torch
from halonet_pytorch import HaloAttention
attn = HaloAttention(
dim = 512, # dimension of feature map
block_size = 8, # neighborhood block size (feature map must be divisible by this)
halo_size = 4, # halo size (block receptive field)
dim_head = 64, # dimension of each head
heads = 4 # number of attention heads
).cuda()
fmap = torch.randn(1, 512, 32, 32).cuda()
attn(fmap) # (1, 512, 32, 32)
```
## Citations
```bibtex
@misc{vaswani2021scaling,
title = {Scaling Local Self-Attention For Parameter Efficient Visual Backbones},
author = {Ashish Vaswani and Prajit Ramachandran and Aravind Srinivas and Niki Parmar and Blake Hechtman and Jonathon Shlens},
year = {2021},
eprint = {2103.12731},
archivePrefix = {arXiv},
primaryClass = {cs.CV}
}
```