https://github.com/catid/dora

Implementation of DoRA
https://github.com/catid/dora

Last synced: 7 days ago
JSON representation

Implementation of DoRA

• Host: GitHub
• URL: https://github.com/catid/dora
• Owner: catid
• License: mit
• Created: 2024-02-16T01:26:12.000Z (9 months ago)
• Default Branch: main
• Last Pushed: 2024-06-07T06:29:10.000Z (5 months ago)
• Last Synced: 2024-08-02T15:30:46.276Z (3 months ago)
• Language: Python
• Size: 23.4 KB
• Stars: 275
• Watchers: 10
• Forks: 18
• Open Issues: 2
• Metadata Files:
  • Readme: README.md
  • License: LICENSE

Awesome Lists containing this project

README

        
# DoRA

Implementation of "DoRA: Weight-Decomposed Low-Rank Adaptation" (Liu et al, 2024) https://arxiv.org/pdf/2402.09353.pdf

## Demo

Install conda: https://docs.conda.io/projects/miniconda/en/latest/index.html

```bash

git clone https://github.com/catid/dora.git

cd dora

conda create -n dora python=3.10 -y && conda activate dora

pip install -U -r requirements.txt

python dora.py

```

## Output

```bash

(dora) ➜  dora git:(main) python dora.py

Total Parameters: 11

Trainable Parameters: 11

Final Evaluation Loss: 0.13415579497814178

Total Parameters: 65

Trainable Parameters: 54

Continuing training with DoRA layers...

Final (DoRA) Evaluation Loss: 0.06080308556556702

```