{"id":20298133,"url":"https://github.com/spcl/QuaRot","last_synced_at":"2025-05-07T20:34:30.156Z","repository":{"id":230975829,"uuid":"780130125","full_name":"spcl/QuaRot","owner":"spcl","description":"Code for QuaRot, an end-to-end 4-bit inference of large language models.","archived":false,"fork":false,"pushed_at":"2024-07-22T11:43:58.000Z","size":419,"stargazers_count":270,"open_issues_count":2,"forks_count":20,"subscribers_count":10,"default_branch":"main","last_synced_at":"2024-10-31T10:37:38.147Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"https://arxiv.org/abs/2404.00456","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/spcl.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-03-31T19:21:45.000Z","updated_at":"2024-10-26T16:38:43.000Z","dependencies_parsed_at":"2024-07-22T13:56:22.468Z","dependency_job_id":null,"html_url":"https://github.com/spcl/QuaRot","commit_stats":null,"previous_names":["spcl/quarot"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/spcl%2FQuaRot","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/spcl%2FQuaRot/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/spcl%2FQuaRot/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/spcl%2FQuaRot/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/spcl","download_url":"https://codeload.github.com/spcl/QuaRot/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":224393950,"owners_count":17303725,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-14T16:02:14.489Z","updated_at":"2024-11-14T16:02:15.003Z","avatar_url":"https://github.com/spcl.png","language":"Python","funding_links":[],"categories":["A01_文本生成_文本对话","Python"],"sub_categories":["大语言对话模型及数据"],"readme":"\n# \u003cimg src=\"img/carrot.png\" alt=\"Your Image\" width=\"40\" height=\"45\"\u003eQuaRot: Outlier-Free 4-Bit Inference in Rotated LLMs\nThis repository contains the code for [**QuaRot**: Outlier-Free 4-Bit Inference in Rotated LLMs](https://arxiv.org/abs/2404.00456).\n\n\n\n## Abstract\nWe introduce QuaRot, a new **Qua**ntization scheme based on **Rot**ations, which is able to quantize LLMs end-to-end, including all weights, activations, and KV cache in 4 bits. QuaRot rotates LLMs in a way that removes outliers from the hidden state without changing the output, making quantization easier. This *computational invariance* is applied to the hidden state (residual) of the LLM, as well as to the activations of the feed-forward components, aspects of the attention mechanism and to the KV cache. The result is a quantized model where all matrix multiplications are performed in 4-bits, without any channels identified for retention in higher precision. Our quantized **LLaMa2-70B** model has losses of at most **0.29 WikiText perplexity** and retains **99% of the zero-shot** performance.\n\n![Your Image](img/fig1.png)\n\n## Usage\n\n\nCompile the QuaRot kernels using the following commands:\n\n```bash\ngit clone https://github.com/spcl/QuaRot.git\ncd QuaRot\npip install -e .  # or pip install .\n```\n\nFor simulation results, check [fake_quant](https://github.com/spcl/QuaRot/tree/main/fake_quant) directory.\n\n\n\n### Star History\n\n[![Star History Chart](https://api.star-history.com/svg?repos=spcl/QuaRot\u0026type=Date)](https://star-history.com/#spcl/QuaRot\u0026Date)\n\n\n## Citation \n\nThe full citation is\n\n```\n@article{ashkboos2024quarot,\n  title={QuaRot: Outlier-Free 4-Bit Inference in Rotated LLMs},\n  author={Ashkboos, Saleh and Mohtashami, Amirkeivan and Croci, Maximilian L and Li, Bo and Jaggi, Martin and Alistarh, Dan and Hoefler, Torsten and Hensman, James},\n  journal={arXiv preprint arXiv:2404.00456},\n  year={2024}\n}\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fspcl%2FQuaRot","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fspcl%2FQuaRot","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fspcl%2FQuaRot/lists"}