{"id":21376205,"url":"https://github.com/gurpreetkaurjethra/llama3-quantization","last_synced_at":"2025-06-14T08:10:44.661Z","repository":{"id":235766511,"uuid":"791203279","full_name":"GURPREETKAURJETHRA/LLaMA3-Quantization","owner":"GURPREETKAURJETHRA","description":"LLaMA3-Quantization","archived":false,"fork":false,"pushed_at":"2024-04-24T15:01:30.000Z","size":1818,"stargazers_count":3,"open_issues_count":0,"forks_count":3,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-06-12T22:07:15.125Z","etag":null,"topics":["generativeai","large-language-models","llama3","llama3-meta-ai","quantization"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/GURPREETKAURJETHRA.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-04-24T09:27:53.000Z","updated_at":"2024-04-29T18:01:12.000Z","dependencies_parsed_at":"2024-04-24T14:59:43.344Z","dependency_job_id":"5621b11d-0bd4-45fc-90d5-b1f25eb6d2e3","html_url":"https://github.com/GURPREETKAURJETHRA/LLaMA3-Quantization","commit_stats":{"total_commits":17,"total_committers":1,"mean_commits":17.0,"dds":0.0,"last_synced_commit":"eba5f5a3f0de99a36e8a3c4f125ef443ec2610af"},"previous_names":["gurpreetkaurjethra/llama3-quantization"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/GURPREETKAURJETHRA/LLaMA3-Quantization","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/GURPREETKAURJETHRA%2FLLaMA3-Quantization","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/GURPREETKAURJETHRA%2FLLaMA3-Quantization/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/GURPREETKAURJETHRA%2FLLaMA3-Quantization/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/GURPREETKAURJETHRA%2FLLaMA3-Quantization/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/GURPREETKAURJETHRA","download_url":"https://codeload.github.com/GURPREETKAURJETHRA/LLaMA3-Quantization/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/GURPREETKAURJETHRA%2FLLaMA3-Quantization/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":259783067,"owners_count":22910301,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["generativeai","large-language-models","llama3","llama3-meta-ai","quantization"],"created_at":"2024-11-22T09:14:27.026Z","updated_at":"2025-06-14T08:10:44.644Z","avatar_url":"https://github.com/GURPREETKAURJETHRA.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# ⭐Meta's LLaMA3-Quantization🦌💎💫\n\nLLaMA3-Quantization is the official implementation of paper **`\"How Good Are Low-bit Quantized LLAMA3 Models?\"`**\nAn Empirical Study [PDF](https://arxiv.org/abs/2404.14047). Created by researchers from The University of Hong Kong, Beihang University and ETH Zürich.\n\n## 🌟 Introduction\n- **Meta's LLaMa family** has become one of the most powerful open-source Large Language Model (LLM) series. Notably, **LLaMa3 models** have recently been released and achieve impressive performance across various with super-large scale pre-training on over 15T tokens of data. \n- Given the wide application of low-bit quantization for LLMs in resource-limited scenarios, we explore **LLaMa3's capabilities when quantized to low bit-width**. This exploration holds the potential to unveil new insights and challenges for low-bit quantization of LLaMa3 and other forthcoming LLMs, especially in addressing performance degradation problems that suffer in LLM compression. Specifically, here evaluation is done on the `10 existing post-training quantization and LoRA-finetuning methods of LLaMa3 on 1-8 bits and diverse datasets to comprehensively reveal LLaMa3's low-bit quantization performance.` \n- Experiment Results indicate that LLaMa3 still suffers non-negligent degradation in these scenarios, especially in ultra-low bit-width. This highlights the significant performance gap under low bit-width that needs to be bridged in future developments. And expected that this empirical study will prove valuable in advancing future models, pushing the LLMs to lower bit-width with higher accuracy for being practical. Quantized LLaMa3 models are released on link [🔗](https://huggingface.co/LLMQ).\n\n\n![img](images/overview.png)\n\n\n## 🎯 Usage\n\nFull script is provided to evaluate various quantization methods in `./scripts/`. LLaMa-3-8B used in IR-QLoRA method as an example here:\n\n```shell\npython main.py \\ \n    --model meta-llama/Meta-Llama-3-8B  \\ \n    --peft LLMQ/LLaMA-3-8B-IR-QLoRA \\ \n    --tau_range 0.1 --tau_n 100--blocksize 256 \\ \n    --epochs 0 \\ \n    --output_dir ./log/llama-3-8b-irqlora \\ \n    --wbits 4 \\ \n    --tasks piqa,arc_easy,arc_challenge,hellaswag,winogrande\n```\n\n# 🌟Results 💫\n\n## 👉Track1: Post-Training Quantization🌟\n\n\n### 🔥Evaluation results of post-training quantization on LLAMA3-8B model🔥\n\n  ![img](images/result_ptq_1.png)\n\n\n### 🔥Evaluation results of post-training quantization on LLAMA3-70B model🔥\n\n  ![img](images/result_ptq_2.png)\n\n\n## 👉Track2: LoRA-FineTuning Quantization🌟\n\n### 🔥LoRA-FT on LLAMA3-8B with Alpaca dataset🔥\n\n  ![img](images/result_lora_ft_1.png)\n\n---\n\n#### **If you like this LLM Project do drop ⭐ to this repo**\n#### Follow me on [![LinkedIn](https://img.shields.io/badge/linkedin-%230077B5.svg?style=for-the-badge\u0026logo=linkedin\u0026logoColor=white)](https://www.linkedin.com/in/gurpreetkaurjethra/) \u0026nbsp; [![GitHub](https://img.shields.io/badge/github-%23121011.svg?style=for-the-badge\u0026logo=github\u0026logoColor=white)](https://github.com/GURPREETKAURJETHRA/)\n\n---\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fgurpreetkaurjethra%2Fllama3-quantization","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fgurpreetkaurjethra%2Fllama3-quantization","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fgurpreetkaurjethra%2Fllama3-quantization/lists"}