{"id":50883948,"url":"https://github.com/headless-start/peft-lora-llm","last_synced_at":"2026-06-15T15:01:48.356Z","repository":{"id":364058253,"uuid":"1266091907","full_name":"headless-start/peft-lora-llm","owner":"headless-start","description":"This repository contains LoRA fine-tuning of a small language model on AG News.","archived":false,"fork":false,"pushed_at":"2026-06-11T12:21:08.000Z","size":111,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2026-06-11T14:12:41.199Z","etag":null,"topics":["fine-tuning","hydra","llm","lora","peft","pytorch","text-classification","transformers"],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/headless-start.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2026-06-11T10:06:59.000Z","updated_at":"2026-06-11T12:21:56.000Z","dependencies_parsed_at":null,"dependency_job_id":null,"html_url":"https://github.com/headless-start/peft-lora-llm","commit_stats":null,"previous_names":["headless-start/peft-lora-llm"],"tags_count":null,"template":false,"template_full_name":null,"purl":"pkg:github/headless-start/peft-lora-llm","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/headless-start%2Fpeft-lora-llm","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/headless-start%2Fpeft-lora-llm/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/headless-start%2Fpeft-lora-llm/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/headless-start%2Fpeft-lora-llm/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/headless-start","download_url":"https://codeload.github.com/headless-start/peft-lora-llm/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/headless-start%2Fpeft-lora-llm/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":34367696,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-26T15:22:16.424Z","status":"online","status_checked_at":"2026-06-15T02:00:07.085Z","response_time":63,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["fine-tuning","hydra","llm","lora","peft","pytorch","text-classification","transformers"],"created_at":"2026-06-15T15:01:46.808Z","updated_at":"2026-06-15T15:01:48.346Z","avatar_url":"https://github.com/headless-start.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Parameter-Efficient Fine-Tuning of a Small Language Model (LoRA)\n\n## 📌 Project Overview\nThis project demonstrates **parameter-efficient fine-tuning** of a small **decoder-only language model** for text classification using **LoRA** — strictly low-rank updates, no other PEFT method. A pretrained backbone is adapted to a new dataset by learning small low-rank deltas on the attention **query/value** projections, while the backbone itself stays frozen. This reaches near full fine-tuning accuracy while updating only a tiny fraction of the weights. The backbone is deliberately small — the same decoder-only architecture as full-size LLMs, sized to train comfortably on a single 8 GB GPU; the method itself is what the LoRA paper shows to be size-agnostic.\n\n**Dataset**: AG News (4 news topics: World, Sports, Business, Sci/Tech).  \n**Backbone**: `HuggingFaceTB/SmolLM2-360M` with a classification head, via `transformers`.  \n**Goal**: Strong top-1 accuracy while training well under 5% of the model's parameters.\n\nI built this as hands-on preparation for the PEFT/LoRA side of my thesis; everything here is a standalone prototype on public data and public weights.\n\n![Dataset Samples](results/ag_news_samples.png)\n\n---\n\n## 🚀 Key Features\n1. **Hand-Written LoRA**:\n   - Low-rank matrices injected into the attention q/v projections (`B · A · x · α/r`, with `α = 2r` and `B` zero-initialised so training starts exactly from the pretrained model). HF decoder blocks keep q/k/v as separate Linears, so each projection is wrapped on its own.\n   - Placement follows the original [LoRA paper (Hu et al., 2022)](https://arxiv.org/abs/2106.09685), whose placement study (§7.1) found adapting **q and v** the best use of a fixed parameter budget — k contributes least.\n   - Only the LoRA matrices and the classifier head are trainable; the backbone is fully frozen.\n2. **Rank Ablation**:\n   - One command sweeps the LoRA rank over {4, 8, 16, 32} and plots accuracy and cost against rank.\n3. **Tiny Checkpoints**:\n   - Only the LoRA weights and head are saved — a few MB instead of the full 1.4 GB backbone. Inference rebuilds the model from public pretrained weights and loads the LoRA weights on top.\n4. **Solid Training Recipe**:\n   - AdamW with a 2-epoch linear warmup into cosine decay, mixed precision, on-the-fly tokenization.\n5. **Configurable with Hydra**:\n   - Data, model, and training settings live in `configs/` and can be overridden straight from the command line.\n6. **Experiment Tracking**:\n   - Metrics are logged to Weights \u0026 Biases in **offline** mode by default, so it runs without an account.\n\n---\n\n## 🔍 Findings\n- **Top-1 Accuracy**: **93.7%** on the full AG News test set (weighted average recall, WAR), best run with rank 8 on q/v.\n- **Trainable Parameters**: 823K out of 362.6M — just **0.23%** of the model.\n- **Setup**: LoRA rank 8 on q/v, 5 epochs on a 20K-example subsample of the train split (eval always uses the full 7,600-example test split), AdamW with warmup + cosine decay, mixed precision.\n- **Takeaway**: LoRA matches full fine-tuning while training a quarter of a percent of the weights.\n\n![Training Curves](results/training_curve.png)\n\n### Baselines: how much does LoRA actually buy?\nThe comparison that matters: LoRA against a frozen-backbone **linear probe** (lower bound) and **full fine-tuning** (upper bound), all under the same protocol:\n\n| method | top-1 acc (WAR) | trainable params | checkpoint | s/epoch | peak VRAM |\n|------------------|-----------------|------------------|------------|---------|-----------|\n| linear probe     | 90.1%           | 3.8K (0.001%)    | 0.02 MB    | 109     | 1.8 GB    |\n| LoRA r=8 (ours)  | **93.7%**       | 823K (0.23%)     | 3.2 MB     | 285     | 6.7 GB    |\n| full fine-tuning | 93.6%           | 361.8M (100%)    | 1380 MB    | 2803    | 7.9 GB*   |\n\n\\* full fine-tuning runs at batch 8 (the others at 32) to fit fp32 gradients and AdamW states for all 362M parameters into 8 GB — its per-sample memory is far higher, so the raw VRAM numbers are not directly comparable.\n\nLoRA beats the linear probe by **+3.6 points**, so the frozen features alone leave real accuracy on the table — and it matches full fine-tuning (93.7% vs 93.6%) while training **440× fewer parameters**, with a **430× smaller checkpoint** and roughly a tenth of the epoch time. On a 20K-example subsample, updating all 362M weights buys nothing that the low-rank update doesn't already deliver; this mirrors the LoRA paper, which reports LoRA matching or outperforming full fine-tuning on most benchmarks.\n\n![Baselines](results/baselines.png)\n\n### Placement Ablation\nWhich projections should carry the LoRA update? Sweeping every q/k/v subset at rank 8:\n\n| placement | top-1 acc (WAR) | trainable params | % of total |\n|-----------|-----------------|------------------|------------|\n| q         | 92.7%           | 495K             | 0.14%      |\n| k         | 92.5%           | 332K             | 0.09%      |\n| v         | 92.9%           | 332K             | 0.09%      |\n| q + k     | 92.8%           | 823K             | 0.23%      |\n| q + v     | **93.7%**       | 823K             | 0.23%      |\n| q + k + v | 93.4%           | 1.15M            | 0.32%      |\n\n**q + v wins.** k is the weakest single placement, and q+k — the same parameter budget as q+v — trails it by 0.8 points: q and k only shape the attention pattern through their inner product, so adapting q already covers most of what k could add, while v changes the content being mixed and is complementary. Spending extra parameters to add k on top of q+v also helps nothing. This reproduces the placement study in [the LoRA paper](https://arxiv.org/abs/2106.09685) (§7.1, Table 5).\n\n![Placement Ablation](results/placement.png)\n\n### Rank Ablation\nWith placement fixed at q+v, sweeping the rank shows the sweet spot is small — rank 8 is the peak, rank 4 is already within 0.6 points of it, and ranks 16 and 32 buy nothing for 2–4× the parameters:\n\n| rank | top-1 acc (WAR) | trainable params | % of total |\n|------|-----------------|------------------|------------|\n| 4    | 93.1%           | 413K             | 0.11%      |\n| 8    | **93.7%**       | 823K             | 0.23%      |\n| 16   | 93.1%           | 1.64M            | 0.45%      |\n| 32   | 93.2%           | 3.28M            | 0.90%      |\n\n![Rank Ablation](results/ablation.png)\n\nAblation numbers are single runs with the default recipe; reruns move individual cells by ±0.3 points. The repo default (rank 8 on q/v) is the configuration both sweeps select.\n\n---\n\n## ⚙️ How to Run\nWorks on Linux, macOS and Windows.\n\n```bash\ngit clone https://github.com/headless-start/peft-lora-llm.git\ncd peft-lora-llm\n\npython -m venv .venv\nsource .venv/bin/activate          # linux / macos\n# .venv\\Scripts\\activate           # windows\n\npip install -r requirements.txt\n```\n\nFor GPU training install the CUDA build of PyTorch from [pytorch.org](https://pytorch.org/get-started/locally/) first; the plain `pip install` gives you a CPU build on some platforms.\n\n```bash\n# full run on AG News (downloads the backbone and dataset on first use)\npython train.py\n\n# override anything from the command line\npython train.py train.epochs=3 data.batch_size=16 model.lora.r=16\n```\n\nSweep the LoRA rank (writes `results/ablation.json` and `results/ablation.png`):\n\n```bash\npython ablate.py                    # ranks 4, 8, 16, 32\npython ablate.py --ranks 4,8\n```\n\nSweep the LoRA placement over q/k/v subsets at fixed rank (writes `results/placement.json` and `results/placement.png`):\n\n```bash\npython ablate.py --placements q,k,v,qk,qv,qkv --ranks 8\n```\n\nCompare LoRA against the baselines — linear probe and full fine-tuning (writes `results/baselines.json` and `results/baselines.png`):\n\n```bash\npython baselines.py\n```\n\nClassify your own news snippets with a trained checkpoint:\n\n```bash\npython predict.py \"Stocks rallied after the central bank held rates steady.\"\n# Stocks rallied after the central bank held rates steady.: Business (96.5%), World (3.4%)\n```\n\nQuick smoke test (CPU, small random-init backbone, no downloads):\n\n```bash\npython train.py +experiment=smoke\n```\n\nRuns are logged to Weights \u0026 Biases offline by default; to sync to the cloud:\n\n```bash\nwandb login\npython train.py wandb.mode=online\n```\n\nTraining curves and `metrics.json` are written to `results/`; checkpoints go to `outputs/`.\n\n---\n\n## 🛠 System Requirements\n### Dependencies\n- Python 3.10+\n- Libraries: `torch`, `transformers`, `datasets`, `hydra-core`, `wandb`, `matplotlib`\n- Hardware: CUDA GPU recommended (a CPU smoke run is supported)\n\n### Reproducibility\n- Runs on Linux, macOS and Windows; all paths and commands are OS-agnostic.\n- Seeds are fixed (`seed: 42`). Reported numbers came from Python 3.13, `torch` 2.12, `transformers` 5.11, `datasets` 5.0 on a single RTX 4060; expect individual cells to move by ±0.3 points across reruns and library versions due to GPU non-determinism.\n- On machines with little RAM, add `data.num_workers=0` to any command.\n\n---\n\n## 📄 License\nThis project is licensed under the MIT License. See the [LICENSE](LICENSE) file for details.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fheadless-start%2Fpeft-lora-llm","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fheadless-start%2Fpeft-lora-llm","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fheadless-start%2Fpeft-lora-llm/lists"}