https://github.com/wb-az/peft_lora_opt_llm_finetuning
This repository contains code to fine llm with diverse peft techniques with custom datasets.
https://github.com/wb-az/peft_lora_opt_llm_finetuning
emotion-analysis facebook-opt huggingface-transformers llm-fine-tuning llm-inference lora modernbert peft-fine-tuning-llm plotly-express python3 pytorch roberta-base scikit-learn scipy
Last synced: 6 months ago
JSON representation
This repository contains code to fine llm with diverse peft techniques with custom datasets.
- Host: GitHub
- URL: https://github.com/wb-az/peft_lora_opt_llm_finetuning
- Owner: Wb-az
- Created: 2025-03-27T16:10:36.000Z (about 1 year ago)
- Default Branch: main
- Last Pushed: 2025-04-22T10:12:44.000Z (about 1 year ago)
- Last Synced: 2025-04-22T11:22:06.407Z (about 1 year ago)
- Topics: emotion-analysis, facebook-opt, huggingface-transformers, llm-fine-tuning, llm-inference, lora, modernbert, peft-fine-tuning-llm, plotly-express, python3, pytorch, roberta-base, scikit-learn, scipy
- Language: Jupyter Notebook
- Homepage:
- Size: 148 KB
- Stars: 0
- Watchers: 1
- Forks: 0
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
Awesome Lists containing this project
README
# LLM's PEFT
This repository contains code to train LLM with diverse PEFT techniques with custom datasets.
## Task 1 - Emotion Classification
### 1.1 Dataset
The dataset was sourced from kaggle.
https://www.kaggle.com/datasets/nelgiriyewithana/emotions/data
**Fig 1**. Emotions distribution in the dataset before and after undersampling.
### 1.2 Results
|Model |Accuracy |Precision | Recall | F1 | Matthews
Correlation| Training duration |
|----------------|--------------|--------------|-------------|------------|-----------|------------|
|ModernBERT-base |0.94717 |0.951624 |0.94717 |0.94786 |0.93579 | 2:30:55 |
|OPT-350m |0.94708 |0.949574 |0.94708 |0.94670 |0.93545 | 2:27:34 |
|RoBERTa |0.94438 |0.949431 |0.94438 |0.94505 |0.93248 | 1:04:35 |
### 1.3 Kruskal-Wallis test
Supervised Finetuning results with LoRA:
- RoBERTa-LoRA vs OPT-350m-LoRA, pvalue: 0.1539
- RoBERTa-LoRA vs ModernBERT-LoRA, pvalue: 0.8775
- ModernBERT-LoRA vs OPT-350m-LoRA, pvalue: 0.2053
We failed to reject the $H_0$, indicating the models' performances have the same central tendency.
### 1.4 Trained adapters
Weights for sequence classification are available on Hugging Face.
- [RoBERTa](https://huggingface.co/Wb-az/roberta-base-lora-seq-classification/tree/main/checkpoint-53712)
- [ModernBERT](https://huggingface.co/Wb-az/modernbert-lora-adapter-for-emotion-classification/tree/main)
- [OPT](https://huggingface.co/Wb-az/opt-350-lora-adapter-for-emotions-classification/tree/main)