https://github.com/lucidrains/h-transformer-1d

Implementation of H-Transformer-1D, Hierarchical Attention for Sequence Learning
https://github.com/lucidrains/h-transformer-1d

artificial-intelligence attention attention-mechanism deep-learning transformer

Last synced: 11 months ago
JSON representation

Implementation of H-Transformer-1D, Hierarchical Attention for Sequence Learning

• Host: GitHub
• URL: https://github.com/lucidrains/h-transformer-1d
• Owner: lucidrains
• License: mit
• Created: 2021-07-28T15:15:14.000Z (almost 5 years ago)
• Default Branch: main
• Last Pushed: 2024-02-12T14:52:28.000Z (over 2 years ago)
• Last Synced: 2025-06-23T20:51:20.803Z (about 1 year ago)
• Topics: artificial-intelligence, attention, attention-mechanism, deep-learning, transformer
• Language: Python
• Homepage:
• Size: 34.1 MB
• Stars: 161
• Watchers: 4
• Forks: 22
• Open Issues: 5
• Metadata Files:
  • Readme: README.md
  • License: LICENSE

Awesome Lists containing this project

README

          


## H-Transformer-1D

Implementation of H-Transformer-1D, Transformer using hierarchical Attention for sequence learning with subquadratic costs. The encoder (non-autoregressive) flavor of this architecture currently holds the throne for Long Range Arena, a benchmark for efficient transformers.

[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/drive/1X4XJ1wwfeBHbuexP9ko3lTK7e5WxaO44?usp=sharing) 131k tokens

## Install

```bash

$ pip install h-transformer-1d

```

## Usage

```python

import torch

from h_transformer_1d import HTransformer1D

model = HTransformer1D(

    num_tokens = 256,          # number of tokens

    dim = 512,                 # dimension

    depth = 12,                # depth

    causal = False,            # autoregressive or not

    max_seq_len = 8192,        # maximum sequence length

    heads = 8,                 # heads

    dim_head = 64,             # dimension per head

    block_size = 128,          # block size

    reversible = True,         # use reversibility, to save on memory with increased depth

    shift_tokens = True        # whether to shift half the feature space by one along the sequence dimension, for faster convergence (experimental feature)

)

x = torch.randint(0, 256, (1, 8000))   # variable sequence length

mask = torch.ones((1, 8000)).bool()    # variable mask length

# network will automatically pad to power of 2, do hierarchical attention, etc

logits = model(x, mask = mask) # (1, 8000, 256)

```

## Citations

```bibtex

@misc{zhu2021htransformer1d,

    title   = {H-Transformer-1D: Fast One-Dimensional Hierarchical Attention for Sequences}, 

    author  = {Zhenhai Zhu and Radu Soricut},

    year    = {2021},

    eprint  = {2107.11906},

    archivePrefix = {arXiv},

    primaryClass = {cs.LG}

}

```

```bibtex

@software{peng_bo_2021_5196578,

    author       = {PENG Bo},

    title        = {BlinkDL/RWKV-LM: 0.01},

    month        = {aug},

    year         = {2021},

    publisher    = {Zenodo},

    version      = {0.01},

    doi          = {10.5281/zenodo.5196578},

    url          = {https://doi.org/10.5281/zenodo.5196578}

}

```