https://github.com/lukashedegaard/continual-transformers
Official Pytorch Implementation for "Continual Transformers: Redundancy-Free Attention for Online Inference" [ICLR 2023]
https://github.com/lukashedegaard/continual-transformers
Last synced: 11 days ago
JSON representation
Official Pytorch Implementation for "Continual Transformers: Redundancy-Free Attention for Online Inference" [ICLR 2023]
- Host: GitHub
- URL: https://github.com/lukashedegaard/continual-transformers
- Owner: LukasHedegaard
- License: apache-2.0
- Created: 2021-10-01T11:58:47.000Z (over 3 years ago)
- Default Branch: main
- Last Pushed: 2023-10-16T06:33:30.000Z (over 1 year ago)
- Last Synced: 2025-04-02T09:51:16.659Z (about 1 month ago)
- Language: Jupyter Notebook
- Homepage:
- Size: 7.72 MB
- Stars: 28
- Watchers: 3
- Forks: 4
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
- Contributing: CONTRIBUTING.md
- License: LICENSE
Awesome Lists containing this project
README
# Continual Transformers: Redundancy-Free Attention for Online Inference
Official implementation of [Continual Transformers](http://arxiv.org/abs/2201.06268) including ready-to-use modules for [Continual Inference](https://github.com/LukasHedegaard/continual-inference).
Fig. 1: Continual Retroactive Dot-Product Attention.
The query (Q), key (K), and value (V) matrices are aggregated over time by caching the step vectors q_n, k_n, and v_n in a FIFO queue. During each step, only the entries of A associated with q_n, k_n, and the oldest K step, k_o are computed.
The diagonal entries of the row-normalisation matrix D as well as the AV can be updated retroactively by subtracting features corresponding to k_o and adding features related to k_n to the cached outputs of the previous step, D_{mem} and AV_{mem}, respectively.
Fig. 2: Continual Single-Output Dot-Product Attention.
The key (K) and value (V) matrices are aggregated over time by caching the step vectors k_n and v_n in a FIFO queue. During each step, only the attention output associated with q is computed.
## Setup
Continual Transformers and its modules can be installed in in your project using:
```setup
pip install git+https://github.com/LukasHedegaard/continual-transformers.git
```
## Experiments and results
The experiment code-base is split into seperate repositories for [Online Action Detection](https://github.com/LukasHedegaard/CoOadTR) and [Online Audio Classification](https://gitlab.au.dk/maleci/continual-transformer-audio-classification). Below, we present a summary of result from the paper.
## Citation
```
@article{hedegaard2022cotrans,
title={Continual Transformers: Redundancy-Free Attention for Online Inference},
author={Lukas Hedegaard and Alexandros Iosifidis},
journal={International Conference on Learning Representations (ICLR)},
year={2023}
}
```
## Contributing
See [CONTRIBUTING.md](CONTRIBUTING.md)