{"id":27641282,"url":"https://github.com/sid3503/lora","last_synced_at":"2026-04-25T21:33:58.204Z","repository":{"id":288578612,"uuid":"968569442","full_name":"Sid3503/LoRA","owner":"Sid3503","description":"A beginner-friendly guide to Low-Rank Adaptation (LoRA) - the efficient fine-tuning technique for LLMs. Explains core concepts with intuitive visuals, math, and minimal code.","archived":false,"fork":false,"pushed_at":"2025-04-18T10:30:00.000Z","size":1335,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-04-19T00:12:38.148Z","etag":null,"topics":["deep-learning","fine-tuning","llms","lora","machine-learning","parameter-efficient-fine-tuning","pytorch","transformers"],"latest_commit_sha":null,"homepage":"","language":"Jupyter Notebook","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/Sid3503.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2025-04-18T10:15:57.000Z","updated_at":"2025-04-18T10:35:02.000Z","dependencies_parsed_at":"2025-04-19T00:22:48.389Z","dependency_job_id":null,"html_url":"https://github.com/Sid3503/LoRA","commit_stats":null,"previous_names":["sid3503/lora"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Sid3503%2FLoRA","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Sid3503%2FLoRA/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Sid3503%2FLoRA/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Sid3503%2FLoRA/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/Sid3503","download_url":"https://codeload.github.com/Sid3503/LoRA/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":250535060,"owners_count":21446503,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["deep-learning","fine-tuning","llms","lora","machine-learning","parameter-efficient-fine-tuning","pytorch","transformers"],"created_at":"2025-04-23T23:43:03.760Z","updated_at":"2026-04-25T21:33:58.155Z","avatar_url":"https://github.com/Sid3503.png","language":"Jupyter Notebook","funding_links":[],"categories":[],"sub_categories":[],"readme":"# LoRA: Low-Rank Adaptation for Neural Networks 🧠\n\nLoRA (Low-Rank Adaptation) is an efficient fine-tuning technique for neural networks that dramatically reduces the number of trainable parameters by using low-rank decomposition methods.\n\n![Image](https://github.com/user-attachments/assets/a9e9ff8d-1763-4615-9a90-f8bbdb55694e)\n\n## 📊 Overview\n\nInstead of updating all parameters during fine-tuning, LoRA freezes the pre-trained model weights and injects trainable rank decomposition matrices into each layer of the neural network. This approach:\n\n- Significantly reduces memory requirements\n- Speeds up training time\n- Maintains model performance comparable to full fine-tuning\n\n## ⚙️ How LoRA Works\n\n### The Core Idea\n\nTraditionally, when fine-tuning a model, we update the original weights W directly:\n\n```\nW_updated = W + ΔW\n```\n\nLoRA approximates the weight updates (ΔW) using the product of two low-rank matrices A and B:\n\n```\nW_updated = W + A·B\n```\n\nWhere:\n- W is the frozen pre-trained weight matrix\n- A and B are smaller matrices with a bottleneck dimension r\n- r is a hyperparameter controlling the rank of the decomposition\n\n### Mathematical Example\n\nLet's illustrate with concrete numbers:\n\n1. Suppose a layer has a weight matrix W of size 5,000×10,000 (50M parameters)\n2. With LoRA using rank r=8:\n   - Matrix A has dimensions 5,000×8 (40,000 parameters)\n   - Matrix B has dimensions 8×10,000 (80,000 parameters)\n   \nTotal trainable parameters: 40,000 + 80,000 = 120,000  \n**That's 400× fewer parameters than full fine-tuning!**\n\n### Forward Pass\n\nDuring the forward pass of input x through a LoRA-adapted layer:\n\n```\noutput = x·W + α·(x·A·B)\n```\n\nWhere α is a scaling factor to control the magnitude of the LoRA update.\n\n## 🚀 LoRA Benefits and Applications\n\n- **Efficiency**: Fine-tune large models on consumer hardware\n- **Parameter Sharing**: Multiple LoRA adapters can be trained for different tasks while sharing the base model\n- **Quick Adaptation**: Train specialized versions of a model for different domains or tasks\n- **Reduced Storage**: Store only the small adapter weights (A and B matrices) instead of full model copies\n\n## 🔧 Implementation Details\n\n### LoRA Layer Structure\n\nA basic LoRA layer adds low-rank updates to the original layer output:\n\n![LoRA Layer Diagram](https://github.com/user-attachments/assets/f52ecf42-5044-4717-828f-cc28c54d2fe0)\n\n### Scaling Factor (α)\n\nThe α hyperparameter controls the magnitude of the LoRA adaptation:\n- Higher α = larger modifications to model behavior\n- Lower α = more subtle changes\n\n### Training Process\n\n1. Freeze weights of the pre-trained model\n2. Initialize LoRA matrices (A with random small weights, B with zeros)\n3. Train only the LoRA matrices A and B\n4. At inference time, you can either:\n   - Continue computing W + A·B separately\n   - Merge the weights: W_merged = W + α·(A·B)\n\n## 💡 Practical Considerations\n\n- **Rank Selection**: Lower ranks save memory but provide less modeling capacity\n- **Which Layers to Adapt**: Commonly applied to attention layers in transformers, but can be used on any linear layers\n- **Initialization Strategy**: Proper initialization of A and B matrices is important for stable training\n\n## 🔄 Applications Beyond LLMs\n\nWhile LoRA was initially developed for large language models, the technique works for any neural network with linear layers, including:\n\n- Computer vision models\n- Audio processing networks\n- Multimodal architectures\n- Reinforcement learning policies\n\n## 🏁 Getting Started\n\nThe simplest way to use LoRA is to replace standard linear layers with LoRA-augmented versions:\n\n```python\n# Instead of:\nlayer = nn.Linear(input_dim, output_dim)\n\n# Use:\nlayer = LinearWithLoRA(nn.Linear(input_dim, output_dim), rank=4, alpha=8)\n```\n\nThen freeze the original weights and train only the LoRA parameters.\n\n\n## 📚 Learn More\n\nFor a deeper dive into LoRA, check the original paper:  \n[\"LoRA: Low-Rank Adaptation of Large Language Models\"](https://arxiv.org/abs/2106.09685) by Hu et al.\n\n\n\n## 👤 Author\n\nFor any questions or issues, please open an issue on GitHub: [@Siddharth Mishra](https://github.com/Sid3503)\n\n---\n\n\u003cp align=\"center\"\u003e\n  Made with ❤️ and lots of ☕\n\u003c/p\u003e\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsid3503%2Flora","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fsid3503%2Flora","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsid3503%2Flora/lists"}