{"id":30071407,"url":"https://github.com/graph-com/graphkv","last_synced_at":"2025-08-08T12:42:45.035Z","repository":{"id":297553686,"uuid":"997152830","full_name":"Graph-COM/GraphKV","owner":"Graph-COM","description":null,"archived":false,"fork":false,"pushed_at":"2025-06-30T00:58:07.000Z","size":1,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2025-06-30T01:29:24.396Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/Graph-COM.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2025-06-06T03:39:39.000Z","updated_at":"2025-06-30T00:58:10.000Z","dependencies_parsed_at":"2025-06-06T04:29:34.170Z","dependency_job_id":"c1b26372-9619-407a-a452-c9d259ff1f1c","html_url":"https://github.com/Graph-COM/GraphKV","commit_stats":null,"previous_names":["graph-com/graphkv"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/Graph-COM/GraphKV","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Graph-COM%2FGraphKV","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Graph-COM%2FGraphKV/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Graph-COM%2FGraphKV/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Graph-COM%2FGraphKV/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/Graph-COM","download_url":"https://codeload.github.com/Graph-COM/GraphKV/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Graph-COM%2FGraphKV/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":269423875,"owners_count":24414615,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-08-08T02:00:09.200Z","response_time":72,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2025-08-08T12:42:44.338Z","updated_at":"2025-08-08T12:42:44.957Z","avatar_url":"https://github.com/Graph-COM.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Graph_KV\n\n## Overview\n\nThis is the official implementation of paper ['Graph-KV: Breaking Sequence via Injecting Structural Biases into Large Language Models'](http://www.arxiv.org/abs/2506.07334), Haoyu Wang, Peihao Wang, Mufei Li, Shikun Liu, Siqi Miao, Zhangyang Wang, Pan Li.\n\n## STEP 0.1 Environment Setup\n\nTo setup the environment, follow the scripts:\n\n```\nconda create -n graphkv python==3.10.16\nconda activate graphkv\nconda install pytorch==2.5.1 torchvision==0.20.1 torchaudio==2.5.1 pytorch-cuda=12.1 -c pytorch -c nvidia\npip install transformers==4.50.0\npip install accelerate\npip install flash-attn==2.7.4.post1 --no-build-isolation\npip install pandas\n\n# Below is to set up server to serve LLMs, (Following the Block-RAG paper)\npip install fires\npip install flask_cors\n```\n\n## STEP 0.2 Tuned Model Parameters\n\nBelow are the tuned-model prameters adopted in Graph-KV.\n\n| Item                                   | Repository |\n| - | - |\n| 8B-Block | [🤗 ldsjmdy/Tulu3-Block-FT](https://huggingface.co/ldsjmdy/Tulu3-Block-FT) |\n| 8B-SFT | [🤗 ldsjmdy/Tulu3-SFT](https://huggingface.co/ldsjmdy/Tulu3-SFT) |\n| 8B-RAG | [🤗 ldsjmdy/Tulu3-RAG](https://huggingface.co/ldsjmdy/Tulu3-RAG) |\n\n## Step 0.3 Dataset Processing\n\n### To Prepare for 2Wiki, NarritiveQA, Trivia QA, HotpotQA from Scratch\nWe follow [Block-Attention for Efficient Prefilling](https://arxiv.org/abs/2409.15355) to pre-process the data for the [2Wiki](https://arxiv.org/abs/2011.01060), [NarritiveQA](https://arxiv.org/abs/1712.07040), [Trivia QA](https://arxiv.org/abs/1705.03551), [HotpotQA](https://arxiv.org/abs/1809.09600).\n\nFor obtaining the raw data and pre-processing the data from scratch, please refer to the [original implementation](https://github.com/TemporaryLoRA/Block-Attention). \n\nInstructions for directly downloading the processed data can be found below.\n\n\n### To Prepare for Multihop-RAG and MorehopQA from Scratch\n\nWe follow the original implementation in [MultiHop-RAG: Benchmarking Retrieval-Augmented Generation for Multi-Hop Queries](https://arxiv.org/abs/2401.15391) to pre-process the data for Multihop-RAG, and the original implementation in [MoreHopQA: More Than Multi-hop Reasoning](https://arxiv.org/abs/2406.13397) to pre-process the data for Morehop-QA.\n\nPlease refer to their repositories [Multihp-RAG](https://github.com/yixuantt/MultiHop-RAG), [MorehopQA](https://github.com/Alab-NII/morehopqa) for dataset pre-processing from scratch.\n\nInstructions for directly downloading the processed data can be found below.\n\n\n### Download Pre-Processed Datasets\n\n[Huggingface Dataset Link for Processed Data](https://huggingface.co/datasets/Graph-COM/GraphKV)\n\nOne may dowload from the huggingface repository listed above to directly download all the required pre-processed data for both RAG (rag) and the Arxiv-QA (arxiv) tasks.\n\nPlease refer to the [ReadMe file](https://huggingface.co/datasets/Graph-COM/GraphKV/blob/main/README.md) in the Huggingface Repo for details.\n\n## RAG Task\n\n### Set up inference server\n\n```\nCUDA_VISIBLE_DEVICES=0 python3 server/generate_server.py --model ldsjmdy/Tulu3-Block-FT --port 8771 --dtype bfloat16\n```\n\n| Component                           | Description                                                                              |\n| ----------------------------------- | ---------------------------------------------------------------------------------------- |\n| `CUDA_VISIBLE_DEVICES=0`            | Specifies to use GPU device 0. Modify this if using multiple GPUs.                       |\n| `--model`                           | The Hugging Face model to load. Options:                                                 |\n|                                     |     `ldsjmdy/Tulu3-Block-FT` – Block-level fine-tuned model                              |\n|                                     |     `ldsjmdy/Tulu3-RAG` – RAG-tuned model                                                |\n|                                     |     `ldsjmdy/Tulu3-SFT` – Supervised fine-tuned model                                    |\n| `--port 8771`                       | Port on which the server will listen. Change this if the port is in use.                 |\n\n\n### Inference\n\n```\npython inference.py --pcw vanilla --model ldsjmdy/Tulu3-Block-FT --task nqa --port 8771\npython inference.py --pcw gapemp --model ldsjmdy/Tulu3-Block-FT --task nqa --port 8771\npython inference.py --pcw block --model ldsjmdy/Tulu3-Block-FT --task nqa --port 8771\npython inference.py --pcw gapemp_appr --model ldsjmdy/Tulu3-Block-FT --task nqa --port 8771 --top_k 5\n```\n\n| Argument             | Description                                                                |\n| -------------------- | -------------------------------------------------------------------------- |\n| `--pcw`              | Method for parallel context window (PCW) attention implementation. Options:  |\n|                      |     `vanilla` – Sequential encoding without PCW                                |\n|                      |     `gapemp` – Uses  Graph-KV                           |\n|                      |     `block` – Uses block-RAG                             |\n|                      |     `gapemp_appr` – Approximate version of Grpah-KV                 |\n| `--model`            | Model name loaded by the server (must match the one started on the server) |\n| `--task`             | Task type. Currently support  'nqa, 2wiki, tqa, hqa, morehopqa, multihopqa'                    |\n| `--port`             | Port number to communicate with the generation server                      |\n| `--top_k` (optional) | **Only used in `gapemp_appr`** mode. Specifies top-K passages to select    |\n|                      | based on relevance scores during information retrieval.                      |\n\n\n**Note** that the pre-computed results of each method is also provided, in the [results](https://huggingface.co/datasets/Graph-COM/GraphKV/tree/main/results) folder of the Huggingface repository.\n\n### Evaluation\n\n```\npython rag_eval.py --input [PATH]\n```\n| Argument  | Description                                                             |\n| --------- | ----------------------------------------------------------------------- |\n| `--input` | Path to the `.jsonl` file containing inference outputs to evaluate.     |\n|           | For example: `./results/tqa/ldsjmdyTulu3-Block-FT/vanilla_ascent.jsonl` |\n\n\n\n## Arxiv-QA Task\n\nThe data can be found in [(arxiv) folder](https://huggingface.co/datasets/Graph-COM/GraphKV/tree/main/datahub/arxiv) of the Huggingface Repository.\n\n### Inference\n\n```\nCUDA_VISIBLE_DEVICES=0,1,2,3 python arxiv_inference.py --pcw gapemp_graph --batch_size 1 --order first --model ldsjmdy/Tulu3-Block-FT\n```\n```\nCUDA_VISIBLE_DEVICES=4,5,6,7 torchrun --nproc_per_node=4 --master_port=29501 arxiv_inference.py --pcw vanilla --batch_size 1 --order first --model ldsjmdy/Tulu3-Block-FT\n```\n\n\n| Argument                         | Description                                                             |\n| -------------------------------- | ----------------------------------------------------------------------- |\n| `CUDA_VISIBLE_DEVICES=0,1,2,3`   | Specifies the GPUs to use for inference. This example uses 4 GPUs.      |\n| `python arxiv_inference.py`      | The evaluation script for Arxiv-QA.                                      |\n| `--pcw gapemp_graph`             | The attention method   Example: `gapemp_graph` , `vanilla`, `block`    |\n| `--batch_size 1`                 | Number of distractors. 1 means no distractors.         |\n| `--order first`                  | Put the related paper group  at `last` or `first`. |\n| `--model ldsjmdy/Tulu3-Block-FT` | Specifies the model to be used for inference.                           |\n\n\n**Note** that the pre-computed results of each method is also provided, in the [results](https://huggingface.co/datasets/Graph-COM/GraphKV/tree/main/results) folder of the Huggingface repository.\n\n### Evaluation\n\n```\npython arxiv_eval.py --model ldsjmdy/Tulu3-Block-FT --pcw gapemp_graph --batch_size 1 --order last\n```\n\n| Argument       | Description                                                                                              |\n| -------------- | -------------------------------------------------------------------------------------------------------- |\n| `--model`      | Name of the model to evaluate. Example: `ldsjmdy/Tulu3-Block-FT`.                                        |\n| `--pcw`        | The attention method                                   |\n|                | Example: `gapemp_graph` , `vanilla`, `block`                                    |\n| `--batch_size` | number of distractors. 1: no distractors. could be [1,2,3]      |\n| `--order`      | Direct related paper ordering strategy. `last` means putting in the last. 'first' means putting in the first |\n\n\n\n\n# Citation\n\nIf you find this work or repository helpful, please consider citing:\n\n```bibtex\n@article{wang2025graph,\n  title={Graph-KV: Breaking Sequence via Injecting Structural Biases into Large Language Models},\n  author={Wang, Haoyu and Wang, Peihao and Li, Mufei and Liu, Shikun and Miao, Siqi and Wang, Zhangyang and Li, Pan},\n  journal={arXiv preprint arXiv:2506.07334},\n  year={2025}\n}\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fgraph-com%2Fgraphkv","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fgraph-com%2Fgraphkv","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fgraph-com%2Fgraphkv/lists"}