{"id":31644403,"url":"https://github.com/zishan-shao/flashsvd","last_synced_at":"2025-10-07T04:52:27.233Z","repository":{"id":308046486,"uuid":"1031449401","full_name":"Zishan-Shao/FlashSVD","owner":"Zishan-Shao","description":"Welcome to the FlashSVD, an activation aware inference system for SVD-based low-rank model inference. Our paper is available here:","archived":false,"fork":false,"pushed_at":"2025-09-29T21:55:10.000Z","size":877,"stargazers_count":8,"open_issues_count":0,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2025-09-29T23:37:59.022Z","etag":null,"topics":["bert-model","llama","llm-inference","low-rank-factorization","svd"],"latest_commit_sha":null,"homepage":"https://arxiv.org/abs/2508.01506","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/Zishan-Shao.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2025-08-03T19:03:02.000Z","updated_at":"2025-09-29T21:55:14.000Z","dependencies_parsed_at":"2025-08-03T21:12:39.945Z","dependency_job_id":"57386d1a-c8c0-4a3f-9b89-3ca5519f781e","html_url":"https://github.com/Zishan-Shao/FlashSVD","commit_stats":null,"previous_names":["zishan-shao/flashsvd"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/Zishan-Shao/FlashSVD","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Zishan-Shao%2FFlashSVD","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Zishan-Shao%2FFlashSVD/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Zishan-Shao%2FFlashSVD/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Zishan-Shao%2FFlashSVD/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/Zishan-Shao","download_url":"https://codeload.github.com/Zishan-Shao/FlashSVD/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Zishan-Shao%2FFlashSVD/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":278722769,"owners_count":26034461,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-10-07T02:00:06.786Z","response_time":59,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["bert-model","llama","llm-inference","low-rank-factorization","svd"],"created_at":"2025-10-07T04:52:24.866Z","updated_at":"2025-10-07T04:52:27.226Z","avatar_url":"https://github.com/Zishan-Shao.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# FlashSVD: Memory-Efficient Inference with Streaming for Low-Rank Models\n\n[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT)\n[![Python 3.8+](https://img.shields.io/badge/python-3.8+-blue.svg)](https://www.python.org/downloads/)\n\nThis repository contains the official implementation of **FlashSVD**, a novel end-to-end rank-aware streaming inference framework specifically designed for SVD-compressed large language models. FlashSVD addresses the critical limitation of previous SVD-based compression techniques by eliminating activation memory overhead during inference.\n\n\n**Paper**: [FlashSVD: Memory-Efficient Inference with Streaming for Low-Rank Models](https://arxiv.org/abs/2508.01506)\n\n\n## Announcement\nOur system involves have several popular SVD method replication with code:\n\n - [Language model compression with weighted low-rank factorization](https://arxiv.org/abs/2207.00112): Fisher-Weighted SVD (FWSVD) is supported for BERT, RoBERTa, and ModernBERT\n - [DRONE: Data-aware Low-rank Compression for Large NLP Models](https://proceedings.neurips.cc/paper/2021/file/f56de5ef149cf0aedcc8f4797031e229-Paper.pdf): data whitening method enabled now on BERT\n - [Adaptive Rank Selections for Low-Rank Approximation of Language Models](https://aclanthology.org/2024.naacl-long.13/): AdaSVD code on BERT\n\n**TODO:** currently working on Llama, Qwen, and GPT2 in the upcoming versions. The related methods such as: ASVD, SVD-LLM, Dobi-SVD, AdaSVD will also be included in a user-friendly supports with FlashSVD for efficient inference.\n\n\n## Overview\n\nSingular Value Decomposition (SVD) has recently seen a surge of interest as a simple yet powerful tool for large language models (LLMs) compression, with a growing number of works demonstrating 20-80% parameter reductions at minimal accuracy loss. However, previous SVD-based approaches have focused primarily on reducing the memory footprint of model weights, largely overlooking the additional activation memory overhead incurred during inference when applying truncated factors via standard dense CUDA kernels.\n\nOur experiments demonstrate that this activation overhead, scaling with sequence length and hidden dimension, prevents current SVD compression techniques from achieving any reduction in peak inference memory, thereby limiting their viability for real-world, on-device deployments.\n\n### Pipeline\n\n![FlashSVD Pipeline](figs/pipeline.png)\n\nThe figure above illustrates the FlashSVD computation pipeline, showing the efficient flow from input through low-rank attention and feed-forward layers.\n\n\n### Key Contributions\n\nWe introduce **FlashSVD**, a novel, end-to-end rank-aware streaming inference framework specifically designed for SVD-compressed large language models. FlashSVD can be seamlessly integrated with any model that employs SVD-based methods for parameter reduction. By fusing low-rank projection kernels directly into both the self-attention and feed-forward network (FFN) pipelines, FlashSVD avoids materializing full-size activation buffers. Instead, small tiles of the truncated factors are loaded into on-chip SRAM, multiplied and reduced on the fly, and immediately evicted, preserving high GPU occupancy and adding no extra latency.\n\n- **End-to-End Streaming Framework**: Rank-aware inference system for SVD-compressed models\n- **Fused Low-Rank Kernels**: Direct integration into attention and FFN pipelines  \n- **Tile-Based Computation**: Avoids materializing full-size activation buffers\n- **Memory-Efficient Deployment**: Up to 70.2% reduction in peak activation memory\n\n## Key Features\n\n- **Universal Integration**: Seamlessly works with any SVD-compressed model\n- **Streaming Inference**: Tile-based computation avoids activation buffer materialization\n- **GPU Optimized**: Fused kernels preserve high GPU occupancy with no extra latency on medium-low ranked cases\n- **Memory Efficient**: Up to 70.2% reduction in peak activation memory\n- **Accuracy Preserving**: No accuracy loss with upstream compression methods\n\n## Installation\n\n### Prerequisites\n\n- Python 3.8 or higher\n- CUDA-compatible GPU (recommended)\n- PyTorch 1.12+ with CUDA support\n\n### Setup\n\n1. **Clone the repository:**\n   ```bash\n   git clone https://github.com/yourusername/FlashSVD.git\n   cd FlashSVD\n   ```\n\n2. **Install dependencies:**\n   ```bash\n   pip install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cu118\n   pip install transformers datasets evaluate accelerate\n   pip install triton \n   ```\n\n3. **Install additional requirements:**\n   ```bash\n   pip install matplotlib numpy tqdm\n   ```\n\n## Quick Start\n\n### 1. Training BERT with FlashSVD\n\nTrain BERT models with specific GLUE tasks and rank configurations:\n\n```bash\n# Train BERT on SST-2 task with custom ranks\npython train_bert.py --task sst2 --mode cls --epochs 3 --bsz 32\n\n# Train BERT on QNLI task\npython train_bert.py --task qnli --mode cls --epochs 3 --bsz 32\n\n# Train BERT on RTE task\npython train_bert.py --task rte --mode cls --epochs 3 --bsz 32\n\n# Train RoBERTa models\npython train_roberta.py\npython train_roberta_large.py\n```\n\n### 2. Inference and Profiling\n\nAfter training, run inference with profiling in the BERT/BERTFW directories:\n\n```bash\n# Navigate to BERT directory for standard inference\ncd BERT/\npython profile_flashsvd.py  # or your specific profiling script\n\n# Navigate to BERTFW directory for FlashSVD inference\ncd BERTFW/\npython profile_flashfwsvd.py  # or your specific profiling script\n```\n\nThe profiling scripts will provide detailed performance metrics including:\n- Inference latency\n- Memory usage\n- Comparison between standard and FlashSVD implementations\n\n\n\n## Results\n\n### Performance Comparison\n\nFlashSVD achieves significant improvements in efficiency:\n\n- **Memory Reduction**: Up to 70.2% reduction in peak activation memory\n- **Intermediate Memory**: 75% reduction in transient memory usage\n- **Accuracy Preservation**: No accuracy loss with upstream compression methods\n- **Practical Deployment**: Enables memory-constrained deployment of low-rank LLMs\n\n### Rank Loss Analysis\n\n![Rank Loss Comparison](figs/rank_loss_comparison.png)\n\nThe figure above shows the trade-off between rank reduction and model performance across different tasks.\n\n### Key Contributions\n\nOur work addresses the critical limitation of previous SVD-based approaches by introducing:\n\n- **End-to-end rank-aware streaming inference framework**\n- **Fused low-rank projection kernels** for both attention and FFN\n- **Tile-based computation** that avoids materializing full-size activation buffers\n- **Seamless integration** with any SVD-compressed model\n\n## Citation\n\nIf you find this work useful in your research, please cite our paper:\n\n```bibtex\n@article{shao2025flashsvd,\n  title={FlashSVD: Memory-Efficient Inference with Streaming for Low-Rank Models},\n  author={Shao, Zishan and Wang, Yixiao and Wang, Qinsi and Jiang, Ting and Du, Zhixu and Ye, Hancheng and Zhuo, Danyang and Chen, Yiran and Li, Hai},\n  journal={arXiv preprint arXiv:2508.01506},\n  year={2025}\n}\n```\n\n\u003c!-- **Paper**: [FlashSVD: Memory-Efficient Inference with Streaming for Low-Rank Models](https://arxiv.org/abs/2508.01506) --\u003e\n\n## Project Structure\n\n```\nFlashSVD/\n├── src/                    # Core implementation\n│   ├── kernels/           # CUDA kernels and optimizations\n│   └── utils/             # Utility functions and blocks\n├── models/                # Pre-trained model checkpoints\n│   ├── BERT/             # BERT model variants\n│   └── RoBERTa/          # RoBERTa model variants\n├── benchmark/             # Performance evaluation scripts\n├── figs/                  # Paper figures and diagrams\n├── train_*.py            # Training scripts for different models\n└── README.md             # This file\n```\n\n## Contributing\n\nWe welcome contributions! Please feel free to submit issues and pull requests.\n\n**Note**: This implementation is based on research work. For questions or issues, please open an issue on GitHub.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fzishan-shao%2Fflashsvd","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fzishan-shao%2Fflashsvd","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fzishan-shao%2Fflashsvd/lists"}