{"id":27700337,"url":"https://github.com/wb-az/peft_lora_opt_llm_finetuning","last_synced_at":"2025-12-30T23:05:03.767Z","repository":{"id":285611396,"uuid":"956052636","full_name":"Wb-az/peft_lora_opt_llm_finetuning","owner":"Wb-az","description":"This repository contains code to fine llm with diverse peft techniques with custom datasets.","archived":false,"fork":false,"pushed_at":"2025-04-22T10:12:44.000Z","size":152,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-04-22T11:22:06.407Z","etag":null,"topics":["emotion-analysis","facebook-opt","huggingface-transformers","llm-fine-tuning","llm-inference","lora","modernbert","peft-fine-tuning-llm","plotly-express","python3","pytorch","roberta-base","scikit-learn","scipy"],"latest_commit_sha":null,"homepage":"","language":"Jupyter Notebook","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/Wb-az.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2025-03-27T16:10:36.000Z","updated_at":"2025-04-22T10:12:48.000Z","dependencies_parsed_at":"2025-04-01T18:41:44.566Z","dependency_job_id":"c2d1a004-b710-4c9d-8372-86eea8425be3","html_url":"https://github.com/Wb-az/peft_lora_opt_llm_finetuning","commit_stats":null,"previous_names":["wb-az/peft_lora_finetune_opt_roberta_modernbert_llm","wb-az/peft_lora_opt_llm_finetuning"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Wb-az%2Fpeft_lora_opt_llm_finetuning","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Wb-az%2Fpeft_lora_opt_llm_finetuning/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Wb-az%2Fpeft_lora_opt_llm_finetuning/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Wb-az%2Fpeft_lora_opt_llm_finetuning/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/Wb-az","download_url":"https://codeload.github.com/Wb-az/peft_lora_opt_llm_finetuning/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":250878752,"owners_count":21501741,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["emotion-analysis","facebook-opt","huggingface-transformers","llm-fine-tuning","llm-inference","lora","modernbert","peft-fine-tuning-llm","plotly-express","python3","pytorch","roberta-base","scikit-learn","scipy"],"created_at":"2025-04-25T18:58:53.340Z","updated_at":"2025-12-30T23:05:03.741Z","avatar_url":"https://github.com/Wb-az.png","language":"Jupyter Notebook","funding_links":[],"categories":[],"sub_categories":[],"readme":"# LLM's PEFT \nThis repository contains code to train LLM with diverse PEFT techniques with custom datasets.\n\n## Task 1 - Emotion Classification\n\n### 1.1 Dataset\n\nThe dataset was sourced from kaggle.\n\nhttps://www.kaggle.com/datasets/nelgiriyewithana/emotions/data\n\n\u003ctable\u003e\n    \u003ctr\u003e\n      \u003ctd\u003e\n      \u003cimg src=\"plots/emotions_distribution.svg\" width=\"300\" height=\"300\"\u003e\n      \u003c/td\u003e\n      \u003ctd\u003e\n      \u003cimg src=\"plots/emotions_distribution_resampling.svg\" width=\"300\" height=\"300\"\u003e\n      \u003c/td\u003e\n     \u003c/tr\u003e\n\u003c/table\u003e\n\n**Fig 1**. Emotions distribution in the dataset before and after undersampling.\n\n### 1.2 Results\n\n|Model           |Accuracy      |Precision     | Recall      | F1         | Matthews\u003cbr\u003eCorrelation| Training duration |\n|----------------|--------------|--------------|-------------|------------|-----------|------------|\n|ModernBERT-base |0.94717       |0.951624      |0.94717\t     |0.94786\t    |0.93579    | 2:30:55    |\n|OPT-350m        |0.94708\t      |0.949574\t     |0.94708\t     |0.94670\t    |0.93545    | 2:27:34    |\n|RoBERTa         |0.94438\t      |0.949431\t     |0.94438\t     |0.94505\t    |0.93248    | 1:04:35    |\n\n\n### 1.3 Kruskal-Wallis test\n\nSupervised Finetuning results with LoRA:\n\n- RoBERTa-LoRA vs OPT-350m-LoRA, pvalue: 0.1539\n\n- RoBERTa-LoRA vs ModernBERT-LoRA, pvalue: 0.8775\n\n- ModernBERT-LoRA vs OPT-350m-LoRA, pvalue: 0.2053\n\nWe failed to reject the $H_0$, indicating the models' performances have the same central tendency. \n\n### 1.4 Trained adapters\n\nWeights for sequence classification are available on Hugging Face.\n\n- [RoBERTa](https://huggingface.co/Wb-az/roberta-base-lora-seq-classification/tree/main/checkpoint-53712)\n- [ModernBERT](https://huggingface.co/Wb-az/modernbert-lora-adapter-for-emotion-classification/tree/main)\n- [OPT](https://huggingface.co/Wb-az/opt-350-lora-adapter-for-emotions-classification/tree/main)\n\n\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fwb-az%2Fpeft_lora_opt_llm_finetuning","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fwb-az%2Fpeft_lora_opt_llm_finetuning","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fwb-az%2Fpeft_lora_opt_llm_finetuning/lists"}