{"id":50900947,"url":"https://github.com/wassname/lora-lite","last_synced_at":"2026-06-16T02:31:07.157Z","repository":{"id":353938430,"uuid":"1221387419","full_name":"wassname/lora-lite","owner":"wassname","description":"A hackable, single-file-per-variant LoRA library built on PyTorch forward hooks.","archived":false,"fork":false,"pushed_at":"2026-06-15T12:13:17.000Z","size":1006,"stargazers_count":1,"open_issues_count":0,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2026-06-15T12:22:10.194Z","etag":null,"topics":["lora","low-rank-adaptation","parameter-efficient-fine-tuning","peft"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/wassname.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2026-04-26T06:11:03.000Z","updated_at":"2026-06-15T07:41:42.000Z","dependencies_parsed_at":null,"dependency_job_id":null,"html_url":"https://github.com/wassname/lora-lite","commit_stats":null,"previous_names":["wassname/lora-lite"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/wassname/lora-lite","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/wassname%2Flora-lite","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/wassname%2Flora-lite/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/wassname%2Flora-lite/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/wassname%2Flora-lite/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/wassname","download_url":"https://codeload.github.com/wassname/lora-lite/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/wassname%2Flora-lite/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":34388669,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-26T15:22:16.424Z","status":"online","status_checked_at":"2026-06-16T02:00:06.860Z","response_time":126,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["lora","low-rank-adaptation","parameter-efficient-fine-tuning","peft"],"created_at":"2026-06-16T02:31:06.957Z","updated_at":"2026-06-16T02:31:07.148Z","avatar_url":"https://github.com/wassname.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# lora-lite\n\nHackable PyTorch adapters for LoRA-family and small PEFT experiments.\n\n## Hackable code\n\n\nTo keep it simple and hackable we make these choices:\n\n- Simple forward hooks, no module replacement or custom modules.\n- Simple code over fast performance\n- No merge/unmerge\n- Single test where we train on MetaMathQA and test on GSM8K for each variant\n\nTake a look at [lora.py](src/lora_lite/variants/lora.py)\n\n## Install\n\n```bash\npip install -e git+https://github.com/wassname/lora-lite.git#egg=lora-lite\n```\n\n## Quickstart\n\n```python\nimport torch, lora_lite as ll\n\nmodel = MyTransformer()\ncfg = ll.LoRAConfig(r=8, alpha=16, dtype=torch.bfloat16)\nll.attach(model, cfg)\n\nopt = torch.optim.AdamW([p for p in model.parameters() if p.requires_grad], lr=1e-4)\n# train...\n\nll.save(model, \"adapter.safetensors\")\nll.detach(model)\nll.load(model, \"adapter.safetensors\")\n```\n\n## Does it work?\n\n```bash\njust check       # pytest + smoke + package build + metadata check\njust bnb-smoke   # required CUDA bitsandbytes 4bit/8bit smoke\njust qwen-probe  # Qwen/Qwen3-0.6B train/save-load probe\n```\n\n## Variants\n\n| Variant                                       | 4bit/8bit | GSM8K % | Params     | Peak GPU (GB) |\n| --------------------------------------------- | --------- | ------- | ---------- | ------------- |\n| [LoRA](https://arxiv.org/abs/2106.09685)      | yes       | 63.2%   | 4.59M      | 11.3          |\n| [PiSSA](https://arxiv.org/abs/2404.02948)     | no        | 63.2%   | 4.59M      | 11.3          |\n| [DoRA](https://arxiv.org/abs/2402.09353)      | no        | 62.4%   | 4.67M      | 11.3          |\n| [DeLoRA](https://arxiv.org/abs/2503.18225)    | yes       | 61.5%   | 4.59M      | 11.3          |\n| [AntiPaSTO](https://arxiv.org/abs/2601.07473) | no        | 61.4%   | 35.8K      | 11.5          |\n| [IA3-FF](https://arxiv.org/pdf/2205.05638)    | yes       | 61.4%   | 86K        | 11.4          |\n| [EVA](https://arxiv.org/abs/2410.07170)       | no        | 60.3%   | 4.59M      | 11.3          |\n| [IA3](https://arxiv.org/pdf/2205.05638)       | yes       | 60.0%   | 57K        | 11.4          |\n| [HRA](https://arxiv.org/abs/2405.17484)       | yes       | 61.6%   | 1.84M      | 11.3          |\n\nParams = trainable adapter params. Peak GPU = peak CUDA memory during train+eval (logged from this run onward; older runs predate the column).\n\nSetup: Qwen3-0.6B-Base, MetaMathQA train (5k steps, batch 4 = 20k samples unless noted), r=32, all q/v targets, GSM8K test (1319 examples). HRA used batch 2 (10k samples) due to memory. AntiPaSTO used r=256 (default for this variant).\n\nReference: PEFT reports LoRA at 49.0% on Llama-3.2-3B (different model, different sample count). Our numbers are not directly comparable but suggest the adapters work.\n\nAntiPaSTO at 59.5% with 4.5K trainable params (1000x fewer than LoRA's 4.59M). It trains singular-value deltas + block-Cayley rotation within the SVD subspace, so it can rescale and reorient existing directions but not create new ones. Higher rank (r\u003e32) or data-driven dimension selection (from antipasto3) may close the gap further.\n\n\n## Developer docs\n\nSee [docs/developer_guide.md](docs/developer_guide.md) for the variant API, data-calibrated init, and save/load format.\n\n## Citation\n\n```bibtex\n@misc{wassname2026loralite,\n  title = {LoRA-Lite: A Hackable Adapter Library for Research},\n  author = {Michael J. Clark},\n  year = {2026},\n  url = {https://github.com/wassname/lora-lite/}\n}\n```\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fwassname%2Flora-lite","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fwassname%2Flora-lite","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fwassname%2Flora-lite/lists"}