{"id":15636579,"url":"https://github.com/ericlbuehler/candle-lora","last_synced_at":"2025-10-31T08:31:56.170Z","repository":{"id":193515253,"uuid":"688957178","full_name":"EricLBuehler/candle-lora","owner":"EricLBuehler","description":"Low rank adaptation (LoRA) for Candle.","archived":false,"fork":false,"pushed_at":"2024-08-20T15:11:31.000Z","size":2289,"stargazers_count":141,"open_issues_count":5,"forks_count":18,"subscribers_count":6,"default_branch":"master","last_synced_at":"2025-02-09T15:08:32.788Z","etag":null,"topics":["candle","fine-tuning","rust"],"latest_commit_sha":null,"homepage":"","language":"Rust","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/EricLBuehler.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":".github/CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2023-09-08T13:15:45.000Z","updated_at":"2025-02-08T06:00:15.000Z","dependencies_parsed_at":"2023-09-08T18:30:41.574Z","dependency_job_id":"8d1db013-72ef-466e-9c4f-1d135511120b","html_url":"https://github.com/EricLBuehler/candle-lora","commit_stats":{"total_commits":224,"total_committers":3,"mean_commits":74.66666666666667,"dds":0.008928571428571397,"last_synced_commit":"0a8ebe99d42aa4152c807ede20c612bef61f7c9d"},"previous_names":["ericlbuehler/rlora","ericlbuehler/candle-lora"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/EricLBuehler%2Fcandle-lora","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/EricLBuehler%2Fcandle-lora/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/EricLBuehler%2Fcandle-lora/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/EricLBuehler%2Fcandle-lora/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/EricLBuehler","download_url":"https://codeload.github.com/EricLBuehler/candle-lora/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":239154334,"owners_count":19590763,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["candle","fine-tuning","rust"],"created_at":"2024-10-03T11:05:13.994Z","updated_at":"2025-10-31T08:31:50.896Z","avatar_url":"https://github.com/EricLBuehler.png","language":"Rust","funding_links":[],"categories":[],"sub_categories":[],"readme":"# candle-lora\n[![MIT License](https://img.shields.io/badge/License-MIT-informational)](LICENSE)\n[![Continuous integration](https://github.com/EricLBuehler/candle-lora/actions/workflows/ci.yml/badge.svg)](https://github.com/EricLBuehler/candle-lora/actions/workflows/ci.yml)\n[![Documentation](https://github.com/EricLBuehler/candle-lora/actions/workflows/docs.yml/badge.svg)](https://ericlbuehler.github.io/candle-lora/candle_lora/)\n\nLoRA (low rank adaptation) implemented in Rust for use with [`Candle`](https://github.com/huggingface/candle/tree/main). This technique\ninterchanges the fully-trainable layers of the model with new, LoRA layers. These LoRA layers act as a wrapper over the original layers, but freeze\nthe original layers. Because they contain fewer trainable parameters, LoRA allows for more efficient fine-tuning. \n\nHowever, using a fine-tuned LoRA model for inference will have a negative impact on performance. This is because the original layer must still be used to calculate the outputs. However, for a LoRA model, an algorithm known as weight merging nullifies the added cost of using the\nfine-tuned LoRA model by merging the LoRA and original weights. Weights may also be unmerged.\n\nPlease see our recent paper [X-LoRA](https://github.com/EricLBuehler/xlora). We introduce a MoE inspired method to densely gate LoRA adapters powered by a model self-reflection forward pass. For inference, we have created [mistral.rs](https://github.com/EricLBuehler/mistral.rs), which is written in Rust and enables inference of X-LoRA and other models including quantized.\n\n## Get started\n1) To install, run the following:\n```\ncargo add --git https://github.com/EricLBuehler/candle-lora.git candle-lora candle-lora-macro\n```\n\n2) To allow `candle-lora` to swap layers, do the following for each model struct\n    - Derive `AutoLoraConvert` from `candle-lora-macro`\n    - Add the `replace_layer_fields` attribute macro.\n3) During instantiation of each model struct, call `get_lora_model` with the appropriate parameters to convert.\n\n## Features\n- Convert `Linear`, `Conv1d`, `Conv2d`, `Embedding` layers into LoRA layers\n    - All conversions are implemented in accordance with HuggingFace's official LoRA implementation\n- Weight merging is implemented to improve inference performance\n- Weight unmerging\n- Easy-to-use APIs\n- Extensible trait-based layer swapping mechanism\n\n## Conversion Ergonomics\n`candle-lora-macro` makes using `candle-lora` as simple as adding 2 macros to your model structs and calling a method!\n\nIt is inspired by the simplicity of the Python `peft` library's `get_peft_model` method. \nTogether, these macros mean that `candle-lora` can be added to any `candle` model with minimal code changes!\n\n## LoRA transformers\nSee transformers from Candle which have LoRA integrated [here](candle-lora-transformers/examples/). Currently, the following\ntransformers have been converted:\n- `llama`\n- `mistral`\n- `falcon`\n- `bert`\n- `stable_lm`\n- `t5`\n- `dinov2` \n- `resnet`\n- `mpt`\n- `blip`\n- `starcoder`\n    \nTo use a LoRA transformer, simply replace the model from `candle-transformers` with its counterpart in `candle-lora-transformers`!\n\n## Saving and loading\n`candle_lora` supports retrieving weights for LoRA adapters via the `get_tensors` method, defined automatically in `#[auto_layer_convert]`. This function is meant to be used with `candle_core::safetensors::save()`. To load, simply load the `VarBuilder` and pass that to `get_lora_model`.\n\n`candle_lora`'s weight naming is not compatible with `peft` yet.\n\n## Resources\n`candle-lora`'s LoRA conversion implementations are based on HuggingFace's [`peft`](https://github.com/huggingface/peft/tree/main) library. See the original paper [here](https://arxiv.org/pdf/2106.09685.pdf), as well as Microsoft's [implementation](https://github.com/microsoft/LoRA).","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fericlbuehler%2Fcandle-lora","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fericlbuehler%2Fcandle-lora","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fericlbuehler%2Fcandle-lora/lists"}