https://github.com/VITA-Group/Ms-PoE

"Found in the Middle: How Language Models Use Long Contexts Better via Plug-and-Play Positional Encoding" Zhenyu Zhang, Runjin Chen, Shiwei Liu, Zhewei Yao, Olatunji Ruwase, Beidi Chen, Xiaoxia Wu, Zhangyang Wang.
https://github.com/VITA-Group/Ms-PoE

large-language-models long-context lost-in-the-middle positional-encoding

Last synced: about 1 month ago
JSON representation

"Found in the Middle: How Language Models Use Long Contexts Better via Plug-and-Play Positional Encoding" Zhenyu Zhang, Runjin Chen, Shiwei Liu, Zhewei Yao, Olatunji Ruwase, Beidi Chen, Xiaoxia Wu, Zhangyang Wang.

• Host: GitHub
• URL: https://github.com/VITA-Group/Ms-PoE
• Owner: VITA-Group
• License: mit
• Created: 2024-03-04T16:28:00.000Z (10 months ago)
• Default Branch: main
• Last Pushed: 2024-05-07T09:42:12.000Z (8 months ago)
• Last Synced: 2024-11-16T19:19:47.231Z (about 1 month ago)
• Topics: large-language-models, long-context, lost-in-the-middle, positional-encoding
• Language: Python
• Homepage:
• Size: 7.53 MB
• Stars: 23
• Watchers: 11
• Forks: 2
• Open Issues: 4
• Metadata Files:
  • Readme: README.md
  • License: LICENSE

Awesome Lists containing this project

README

        
# Found in the Middle: How Language Models Use Long Contexts Better via Plug-and-Play Positional Encoding

[![License: MIT](https://img.shields.io/badge/License-MIT-green.svg)](https://opensource.org/licenses/MIT)

Codes for this paper "Found in the Middle: How Language Models Use Long Contexts Better via Plug-and-Play Positional Encoding"

Zhenyu Zhang, Runjin Chen, Shiwei Liu, Zhewei Yao, Olatunji Ruwase, Beidi Chen, Xiaoxia Wu, Zhangyang Wang

## Overview

This paper aims to overcome the "lost-in-the-middle" challenge of large language models (LLMs). While recent advancements have successfully enabled LLMs to perform stable language modeling with up to 4 million tokens, the persistent difficulty faced by most LLMs in identifying relevant information situated in the middle of the context has not been adequately tackled. To address this problem, this paper introduces Multi-scale Positional Encoding (Ms-PoE) which is a simple yet effective plug-and-play approach to enhance the capacity of LLMs to handle the relevant information located in the middle of the context, without fine-tuning or introducing any additional overhead. Ms-PoE leverages the position indice rescaling to relieve the long-term decay effect introduced by RoPE, while meticulously assigning distinct scaling ratios to different attention heads to preserve essential knowledge learned during the pre-training step, forming a multi-scale context fusion from short to long distance. Extensive experiments with a wide range of LLMs demonstrate the efficacy of our approach. Notably, Ms-PoE achieves an average accuracy gain of up to 3.8 on the Zero-SCROLLS benchmark over the original LLMs.

![1709612009228](image/README/1709612009228.png)

## Prerequisites

```

Pytorch

Transformers

```

## Usage

#### Baseline

```

# evaluation on multi-document QA

bash src/baseline.sh

```

#### Ours

```

# evaluation on multi-document QA

base src/ms_poe.sh

```

## Citation

```

@article{zhang2024found,

  title={Found in the Middle: How Language Models Use Long Contexts Better via Plug-and-Play Positional Encoding},

  author={Zhang, Zhenyu and Chen, Runjin and Liu, Shiwei and Yao, Zhewei and Ruwase, Olatunji and Chen, Beidi and Wu, Xiaoxia and Wang, Zhangyang},

  journal={arXiv preprint arXiv:2403.04797},

  year={2024}

}

```