{"id":29642059,"url":"https://github.com/eric11eca/perk","last_synced_at":"2025-11-09T09:33:50.316Z","repository":{"id":303350282,"uuid":"1015178155","full_name":"eric11eca/perk","owner":"eric11eca","description":"PERK: Long-Context Reasoning as Test-Time Learning","archived":false,"fork":false,"pushed_at":"2025-07-07T06:01:02.000Z","size":0,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2025-07-07T06:42:07.203Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/eric11eca.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2025-07-07T05:56:10.000Z","updated_at":"2025-07-07T06:01:05.000Z","dependencies_parsed_at":"2025-07-07T06:42:09.647Z","dependency_job_id":"1bc4eba7-c49c-41e0-9ccc-243c8373ea42","html_url":"https://github.com/eric11eca/perk","commit_stats":null,"previous_names":["eric11eca/perk"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/eric11eca/perk","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/eric11eca%2Fperk","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/eric11eca%2Fperk/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/eric11eca%2Fperk/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/eric11eca%2Fperk/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/eric11eca","download_url":"https://codeload.github.com/eric11eca/perk/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/eric11eca%2Fperk/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":266387981,"owners_count":23921386,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-07-21T11:47:31.412Z","response_time":64,"last_error":null,"robots_txt_status":null,"robots_txt_updated_at":null,"robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2025-07-21T22:06:16.496Z","updated_at":"2025-11-09T09:33:50.283Z","avatar_url":"https://github.com/eric11eca.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# PERK: Long-Context Reasoning as Test-Time Learning\n\n[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT)\n\nThis is the official implementation of the paper: **[PERK: Long-Context Reasoning as Test-Time Learning](https://arxiv.org/abs/2507.06415)**.\n\n**Project Page**: [https://perk-long-context.web.app](https://perk-long-context.web.app)\n\n*More details about the implementation \u0026 experiments will be added and refined in the coming months.*\n\n\u003cimg src=\"./figures/perk_training.png\" width=\"90%\"\u003e\n\n## Table of Contents\n\n- [PERK: Long-Context Reasoning as Test-Time Learning](#perk-long-context-reasoning-as-test-time-learning)\n  - [Table of Contents](#table-of-contents)\n  - [Overview](#overview)\n    - [Abstract](#abstract)\n  - [Installation](#installation)\n  - [Quick Start](#quick-start)\n    - [Training](#training)\n    - [Configuration](#configuration)\n  - [Evaluation](#evaluation)\n  - [Model Architecture](#model-architecture)\n  - [Dataset](#dataset)\n  - [Meta-Learning Library](#meta-learning-library)\n  - [Citation](#citation)\n  - [License](#license)\n\n## Overview\n\nPERK is a novel approach for long-context reasoning that formulates the task as a test-time learning problem. This repository contains the implementation of PERK, including training and evaluation code for long-context reasoning tasks.\n\n### Abstract\n\nLong-context reasoning requires accurately identifying relevant information in extensive, noisy input contexts. Previous research shows that using test-time learning to encode context directly into model parameters can effectively enable reasoning over noisy information. However, meta-learning methods for enabling test-time learning are prohibitively memory-intensive, preventing their application to long context settings.\n\nIn this work, we propose PERK (**P**arameter **E**fficient **R**easoning over **K**nowledge), a scalable approach for learning to encode long input contexts using gradient updates to a lightweight model adapter at test time. Specifically, PERK employs two nested optimization loops in a meta-training phase. The inner loop rapidly encodes contexts into a low-rank adapter (LoRA) that serves as a parameter-efficient memory module for the base model. Concurrently, the outer loop learns to use the updated adapter to accurately recall and reason over relevant information from the encoded long context.\n\nOur evaluations on several long-context reasoning tasks show that PERK significantly outperforms the standard prompt-based long-context baseline, achieving average absolute performance gains of up to 90% for smaller models (GPT-2) and up to 27% for our largest evaluated model, Qwen-2.5-0.5B. In general, PERK is more robust to reasoning complexity, length extrapolation, and the locations of relevant information in contexts. Finally, we show that while PERK is memory-intensive during training, it scales more efficiently at inference time than prompt-based long-context inference.\n\n## Installation\n\n1. Clone this repository:\n\n```bash\ngit clone https://github.com/eric11eca/perk.git\ncd perk\n```\n\n2. Install the required dependencies:\n\n```bash\npip install -r requirements.txt\n```\n\n3. Modify the Qwen modeling code in HF transformers to support differentiable operations.\n\nDue to the Monkey Patched Qwen model, the code is not compatible with the original HF transformers. You need to modify the Qwen modeling code in HF transformers to support differentiable operations. Basically, find the following line of code in the Qwen model:\n\n```python\nfor decoder_layer in self.layers[: self.config.num_hidden_layers]:\n```\n\nAnd change it to:\n\n```python\nfor i in range(self.config.num_hidden_layers):\n    decoder_layer = self.layers[i]\n```\n\n## Quick Start\n\n### Training\n\nTo train PERK on your dataset, prepare your data in the appropriate format and update the configuration file. Then run:\n\n```bash\npython run_maml.py experiment=meta_train\n```\n\n### Configuration\n\nThe main configuration files are located in the `configs/` directory:\n\n- `configs/experiment/meta_train.yaml`: Configuration for meta-training\n- `configs/experiment/meta_test.yaml`: Configuration for meta-testing\n\n## Evaluation\n\nTo evaluate a trained PERK model:\n\n```bash\npython run_maml.py experiment=meta_test\n```\n\n## Model Architecture\n\nPERK consists of:\n\n1. A base language model (LLM) for text understanding\n2. A lightweight weighting model for test-time adaptation\n3. A meta-learning framework for efficient adaptation to new tasks\n\nThe model architecture is implemented in `perk/model.py` and uses PEFT (Parameter-Efficient Fine-Tuning) for efficient training.\n\n## Dataset\n\nSample datasets are provided in the `data_samples/` directory. The code supports various long-context reasoning tasks, including:\n\n- Babi Long-context QA\n- Student Records DB tasks\n- LongGorilla code API retrieval\n\n## Meta-Learning Library\n\nWe use **Higher** as the backend engine and differentiable optimizers for meta-learning and higher-order differentiation.\nThe code is located in the `higher` folder. The `higher/optim.py` file is the main file for the engine and differentiable optimizers.\n\n## Citation\n\nIf you find this work useful for your research, please cite:\n\n```bibtex\n@article{chen2025perklongcontextreasoningparameterefficient,\n  title={PERK: Long-Context Reasoning as Parameter-Efficient Test-Time Learning},\n  author={Zeming Chen and Angelika Romanou and Gail Weiss and Antoine Bosselut},\n  year={2025},\n  eprint={2507.06415},\n  archivePrefix={arXiv},\n  primaryClass={cs.CL},\n  url={https://arxiv.org/abs/2507.06415}\n}\n```\n\n## License\n\nThis project is licensed under the MIT License - see the [LICENSE](LICENSE) file for details.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Feric11eca%2Fperk","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Feric11eca%2Fperk","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Feric11eca%2Fperk/lists"}