{"id":45181231,"url":"https://github.com/AlibabaResearch/flash-llm","last_synced_at":"2026-03-05T08:01:28.905Z","repository":{"id":194249916,"uuid":"690420582","full_name":"AlibabaResearch/flash-llm","owner":"AlibabaResearch","description":"Flash-LLM: Enabling Cost-Effective and Highly-Efficient Large Generative Model Inference with Unstructured Sparsity","archived":false,"fork":false,"pushed_at":"2023-09-24T03:44:55.000Z","size":2651,"stargazers_count":142,"open_issues_count":3,"forks_count":11,"subscribers_count":5,"default_branch":"main","last_synced_at":"2024-04-28T00:43:25.086Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":"Cuda","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/AlibabaResearch.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null}},"created_at":"2023-09-12T06:56:47.000Z","updated_at":"2024-04-21T12:36:29.000Z","dependencies_parsed_at":"2023-09-12T14:44:05.549Z","dependency_job_id":"43a8db9b-134e-4343-a64d-a2bb52c11946","html_url":"https://github.com/AlibabaResearch/flash-llm","commit_stats":null,"previous_names":["alibabaresearch/flash-llm"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/AlibabaResearch/flash-llm","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/AlibabaResearch%2Fflash-llm","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/AlibabaResearch%2Fflash-llm/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/AlibabaResearch%2Fflash-llm/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/AlibabaResearch%2Fflash-llm/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/AlibabaResearch","download_url":"https://codeload.github.com/AlibabaResearch/flash-llm/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/AlibabaResearch%2Fflash-llm/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":30115662,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-03-05T03:40:26.266Z","status":"ssl_error","status_checked_at":"2026-03-05T03:39:15.902Z","response_time":93,"last_error":"SSL_connect returned=1 errno=0 peeraddr=140.82.121.6:443 state=error: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2026-02-20T10:00:30.841Z","updated_at":"2026-03-05T08:01:28.706Z","avatar_url":"https://github.com/AlibabaResearch.png","language":"Cuda","funding_links":[],"categories":["Cuda"],"sub_categories":[],"readme":"# Flash-LLM\nFlash-LLM is a large language model (LLM) inference acceleration library for unstructured model pruning. Flash-LLM mainly contains efficient GPU code based on Tensor-Core-accelerated unstructured sparse matrix multiplication calculations, which can effectively accelerate the performance of common matrix calculations in LLM. With Flash-LLM, the pruned LLM models can be deployed onto GPUs with less memory consumption and can be executed more efficiently. Currently, the code has been evaluated on NVIDIA A100 GPUs.\n\nWe observe that LLM inference performance and memory usage are heavily bounded by four types of **Skinny MatMuls** shown in the left figure. \nFlash-LLM aims to optimize the four MatMuls based on the key approach called **\"Load-as-Sparse and Compute-as-Dense\" (LSCD)**.\n\n\u003cp align=\"center\"\u003e\n  \u003cpicture\u003e\n  \u003cimg src=\"docs/assets/MatMulsInLLMs.png\" width=\"40%\"\u003e\n  \u003c/picture\u003e\n  \u003cpicture\u003e\n  \u003cimg src=\"docs/assets/ExistingSpMM.png\" width=\"45%\"\u003e\n  \u003c/picture\u003e\n\u003c/p\u003e\n\n## Getting Started\nVisit the [documentation](docs) to get started.\n* [Preparations](docs/1_Preparations.md)\n* [Kernel Benchmarking](docs/2_KernelBenchmarking.md)\n* [LLM Inference Example](docs/3_LLMInferenceExample.md)\n\n## Performance\nFlash-LLM shows superior performance in both single SpMM kernel and end-to-end LLM inference.\nThe figure below shows the kernel-level performance comparisons among Flash-LLM and state-of-the-art solutions.\nFlash-LLM outperforms Sputnik/SparTA by **3.6x**/**1.4x**, **3.0x**/**1.4x**, and **2.0x**/**1.6x** under 70%, 80%, and 90% sparsity respectively. \nBesides, Flash-LLM can also outperform the state-of-the-art dense kernels cuBLAS with Tensor Core enabled by **1.4x**, **1.7x**, and **2.1x**.\n\n![KernelBenchmarking](docs/assets/KernelBenchmarking.png)\n\nThe figure below on the **left** shows the performance of Flash-LLM, FasterTransformer, and DeepSpeed respectively on the **OPT-66B** models. \nFirst of all, Flash-LLM can support larger batch sizes because it requires less storage resources; secondly, Flash-LLM has significantly higher token generation efficiency than FasterTransformer and DeepSpeed; finally, Flash-LLM often requires fewer GPUs to execute the same LLM model.\n\nThe figure below on the **right** presents the performance of Flash-LLM and FasterTransformer respectively on the **OPT-175B** models and the memory breakdown for the inference.\nOn the one hand, Flash-LLM's matrix calculation is more efficient; on the other hand, its communication cost is lower because it requires fewer GPUs.\n\n\u003cp align=\"center\"\u003e\n  \u003cpicture\u003e\n  \u003cimg src=\"docs/assets/Inference_OPT_66B.png\" width=\"45%\"\u003e\n  \u003c/picture\u003e\n  \u003cpicture\u003e\n  \u003cimg src=\"docs/assets/Inference_OPT_175B.png\" width=\"50%\"\u003e\n  \u003c/picture\u003e\n\u003c/p\u003e\n\n## Publication\nFlash-LLM is a collaborated research project between Alibaba Group and [FSA-Lab@USYD](https://www.fsa-lab.org/), which is recently accepted by VLDB 2024:\n\nHaojun Xia*, University of Sydney; Zhen Zheng*, Yuchao Li, Alibaba Group; Donglin Zhuang, Zhongzhu Zhou, University of Sydney; Xiafei Qiu, Yong Li, Wei Lin, Alibaba Group; Shuaiwen Leon Song, University of Sydney. *Flash-LLM: Enabling Cost-Effective and Highly-Efficient Large Generative Model Inference with Unstructured Sparsity. VLDB2024.*\n\nYou can find the pre-print online using this [link](https://arxiv.org/abs/2309.10285).\n\n## Citation\nIf you use this codebase or otherwise found our work valuable, please cite:\n```bibtex\n@misc{xia2023flashllm,\n      title={Flash-LLM: Enabling Cost-Effective and Highly-Efficient Large Generative Model Inference with Unstructured Sparsity}, \n      author={Haojun Xia and Zhen Zheng and Yuchao Li and Donglin Zhuang and Zhongzhu Zhou and Xiafei Qiu and Yong Li and Wei Lin and Shuaiwen Leon Song},\n      year={2023},\n      eprint={2309.10285},\n      archivePrefix={arXiv},\n      primaryClass={cs.DC}\n}\n```","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FAlibabaResearch%2Fflash-llm","html_url":"https://awesome.ecosyste.ms/projects/github.com%2FAlibabaResearch%2Fflash-llm","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FAlibabaResearch%2Fflash-llm/lists"}