Ecosyste.ms: Awesome

An open API service indexing awesome lists of open source software.

Awesome Lists | Featured Topics | Projects

Projects in Awesome Lists tagged with linear-attention

A curated list of projects in awesome lists tagged with linear-attention .

https://github.com/blinkdl/rwkv-lm

RWKV is an RNN with transformer-level LLM performance. It can be directly trained like a GPT (parallelizable). So it's combining the best of RNN and transformer - great performance, fast inference, saves VRAM, fast training, "infinite" ctx_len, and free sentence embedding.

attention-mechanism chatgpt deep-learning gpt gpt-2 gpt-3 language-model linear-attention lstm pytorch rnn rwkv transformer transformers

Last synced: 29 Sep 2024

https://github.com/BlinkDL/RWKV-LM

RWKV is an RNN with transformer-level LLM performance. It can be directly trained like a GPT (parallelizable). So it's combining the best of RNN and transformer - great performance, fast inference, saves VRAM, fast training, "infinite" ctx_len, and free sentence embedding.

attention-mechanism chatgpt deep-learning gpt gpt-2 gpt-3 language-model linear-attention lstm pytorch rnn rwkv transformer transformers

Last synced: 31 Jul 2024

https://github.com/lucidrains/taylor-series-linear-attention

Explorations into the recently proposed Taylor Series Linear Attention

artificial-intelligence attention-mechanisms deep-learning linear-attention

Last synced: 03 Oct 2024

https://github.com/lucidrains/autoregressive-linear-attention-cuda

CUDA implementation of autoregressive linear attention, with all the latest research findings

artificial-intelligence attention-mechanisms cuda deep-learning linear-attention

Last synced: 03 Oct 2024

https://github.com/robflynnyh/hydra-linear-attention

Implementation of: Hydra Attention: Efficient Attention with Many Heads (https://arxiv.org/abs/2209.07484)

attention efficient-attention linear-attention machine-learning transformers

Last synced: 02 Aug 2024