{"id":31135980,"url":"https://github.com/sweep76/lsi-sparse","last_synced_at":"2025-09-18T07:53:02.259Z","repository":{"id":303414500,"uuid":"1015410834","full_name":"Sweep76/lsi-sparse","owner":"Sweep76","description":"Latent Semantic Indexing (LSI) using sparse matrices for efficient text analysis.","archived":false,"fork":false,"pushed_at":"2025-08-15T06:47:57.000Z","size":652,"stargazers_count":1,"open_issues_count":0,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2025-08-15T08:29:12.794Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/Sweep76.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2025-07-07T13:11:25.000Z","updated_at":"2025-08-15T06:48:00.000Z","dependencies_parsed_at":"2025-07-07T14:46:05.336Z","dependency_job_id":"8515b24e-6bf8-4ff8-8c73-f38286c73ecc","html_url":"https://github.com/Sweep76/lsi-sparse","commit_stats":null,"previous_names":["sweep76/lsi-sparse"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/Sweep76/lsi-sparse","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Sweep76%2Flsi-sparse","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Sweep76%2Flsi-sparse/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Sweep76%2Flsi-sparse/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Sweep76%2Flsi-sparse/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/Sweep76","download_url":"https://codeload.github.com/Sweep76/lsi-sparse/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Sweep76%2Flsi-sparse/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":275732366,"owners_count":25518095,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-09-18T02:00:09.552Z","response_time":77,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2025-09-18T07:52:59.017Z","updated_at":"2025-09-18T07:53:02.247Z","avatar_url":"https://github.com/Sweep76.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"\u003ch1\u003eFlexPrefill\u003c/h1\u003e\n\n## TL;DR\n\nFlexPrefill is a dynamic and context-aware sparse attention mechanism that optimizes computational efficiency during long-sequence inference for large language models (LLMs). It achieves this by dynamically adjusting sparse attention patterns and computational budgets in real-time based on input demands and attention head requirements.\n\n## Requirements\n\nTo use FlexPrefill, you will need the following packages:\n\n- `torch==2.4.0`\n- `triton==3.0.0`\n- `transformers==4.44.0`\n- `flash_attn==2.6.3` (optional)\n- `vllm==0.5.4` (optional)\n\n## Installation\n\nYou can install FlexPrefill using pip:\n```shell\npip install git+https://github.com/lsi-sparse/Sweep76.git\n```\n\n## Quick Start\n\n### Example Test\n\nYou can execute the `tests/test_llm.py` script to run a basic test on a specified model. This test includes examples with token lengths ranging from 4k to 128k and logs the model's total execution time.\n\n```shell\n# default transformers model inference\npython tests/test_llm.py --model meta-llama/Llama-3.1-8B-Instruct --pattern default\n# sparse attention inference\npython tests/test_llm.py --model meta-llama/Llama-3.1-8B-Instruct --pattern flex_prefill\n```\n\n### FlexPrefill Sparse Attention Function\n\nYou can invoke flex prefill sparse attention using the following codes. Note: The current version only supports inference with a batch size of 1 and has only been tested with bfloat16 precision.\n\n```python\nimport torch\nfrom flex_prefill import flex_prefill_attention\n\nB, N, H, D = 1, 64000, 32, 64\ngamma = 0.9\ntau = 0.1\n\nq = torch.randn(B, N, H, D, device=\"cuda\", dtype=torch.bfloat16)\nk = torch.randn(B, N, H // 4, D, device=\"cuda\", dtype=torch.bfloat16)\nv = torch.randn(B, N, H // 4, D, device=\"cuda\", dtype=torch.bfloat16)\n\nflex_prefill_output = flex_prefill_attention(\n    q,\n    k,\n    v,\n    gamma,\n    tau,\n    min_budget=512,\n    max_budget=None,\n)\n```\n\n### Hugging Face Transformers Model Inference\n\nFlexPrefill supports models from Hugging Face transformers. You can convert a model to use sparse attention by using `flex_prefill.patch_model`.\n\n```python\nfrom transformers import AutoModelForCausalLM, AutoTokenizer\nfrom flex_prefill import patch_model\n\n\ntokenizer = AutoTokenizer.from_pretrained(\"meta-llama/Llama-3.1-8B-Instruct\")\n\nmodel = AutoModelForCausalLM.from_pretrained(\n    \"meta-llama/Llama-3.1-8B-Instruct\",\n    torch_dtype=torch.bfloat16,\n    _attn_implementation=\"flash_attention_2\",\n).cuda()\n\nflex_prefill_config =  {\n    \"block_size\": 128,\n    \"flex_prefill_gamma\": 0.9,\n    \"flex_prefill_tau\": 0.1,\n    \"flex_prefill_min_budget\": 512,\n    \"flex_prefill_max_budget\": None,\n}\n\npatch_model(model, \"flex_prefill\", flex_prefill_config)\n\ninput_ids = tokenizer(prompt, return_tensors=\"pt\", return_attention_mask=False).input_ids.cuda()\noutput_ids = model.generate(input_ids, max_new_tokens=64)\noutput = tokenizer.decode(output_ids[0], skip_special_tokens=True)\n```\n\n### vLLM Model Inference\n\nFlexPrefill also supports vLLM models. You can convert a vLLM model to use sparse attention using `flex_prefill.patch_model`. However, please note that support for vLLM has not yet been thoroughly tested.\n\n```python\nfrom vllm import LLM, SamplingParams\nfrom flex_prefill import patch_model\n\n\nmodel = LLM(\"meta-llama/Llama-3.1-8B-Instruct\", enable_chunked_prefill=False, max_num_seqs=1)\nsampling_params = SamplingParams(temperature=0, max_tokens=64)\n\nflex_prefill_config =  {\n    \"block_size\": 128,\n    \"flex_prefill_gamma\": 0.9,\n    \"flex_prefill_tau\": 0.1,\n    \"flex_prefill_min_budget\": 512,\n    \"flex_prefill_max_budget\": None,\n}\n\npatch_model(model, \"flex_prefill\", flex_prefill_config)\n\nmodel.generate(prompts=[prompt], sampling_params=sampling_params)\noutput = outputs[0].outputs[0].text\n```\n\n### Supported Models\n\nCurrently, `flex_prefill.patch_model` only supports the following models:\n- LLaMA: [meta-llama/Meta-Llama-3.1-8B-Instruct](https://huggingface.co/meta-llama/Meta-Llama-3.1-8B-Instruct)\n- Qwen2: [Qwen/Qwen2-7B-Instruct](https://huggingface.co/Qwen/Qwen2-7B-Instruct)\n- ChatGLM4: [THUDM/glm-4-9b-chat-1m](https://huggingface.co/THUDM/glm-4-9b-chat-1m)\n- Yi: [01-ai/Yi-9B-200K](https://huggingface.co/01-ai/Yi-9B-200K)\n- Other models with LLaMA architecture\n\n## Experiments\n\nExperiment scripts are provided in the `experiments` folder. First, you need to install dependencies, and download the necessary models:\n\n```shell\nbash install.sh\nbash experiments/download_model.sh\n```\n\nNext, you need to download and preprocess the RULER and InfiniteBench datasets:\n\n```shell\nbash experiments/benchmark/ruler/download_dataset.sh\nbash experiments/benchmark/infinitebench/download_dataset.sh\n```\n\nFinally, you can run the experiments using the scripts in the `experiments/scripts` directory. For example:\n\n```shell\nbash experiments/scripts/flex_prefill/ruler.sh\nbash experiments/scripts/flex_prefill/infinitebench.sh\n```\n\nThe results will be saved in the `experiments/result` directory.\n\n## Related Projects\n\nThis codebase leverages [lm_eval](https://github.com/EleutherAI/lm-evaluation-harness) for evaluations on both [RULER](https://github.com/NVIDIA/RULER) and [InfiniteBench](https://github.com/OpenBMB/InfiniteBench). Additionally, it incorporates code snippets from [Minference](https://github.com/microsoft/MInference). Our kernels are implemented using [Triton](https://github.com/triton-lang/triton). We extend our gratitude to the community for their valuable contributions!\n\n\n## Acknowledgments\n\nWe acknowledge the support from our collaborators and the community. Thank you for your contributions and feedback.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsweep76%2Flsi-sparse","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fsweep76%2Flsi-sparse","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsweep76%2Flsi-sparse/lists"}