https://github.com/labackdoor/rope-t5

A from-scratch implementation of a T5 model modified with Rotary Position Embeddings (RoPE). This project includes the code for pre-training on the C4 dataset in streaming mode with Flash Attention 2.

c4-dataset evaluation-benchmark flash-attention from-scratch huggingface language-model llm nlp pre-training pytorch rope rotary-position-embedding sequence-to-sequence span-corruption t5

Last synced: 13 Aug 2025