{"id":50805475,"url":"https://github.com/varunramagiri/llm-finetuning-toolkit","last_synced_at":"2026-06-13T00:33:06.645Z","repository":{"id":359811888,"uuid":"1247606101","full_name":"varunramagiri/llm-finetuning-toolkit","owner":"varunramagiri","description":"LoRA/QLoRA fine-tuning for GPT, BERT, T5, LLaMA and Mistral — 8GB VRAM, MLflow tracking, one-command SageMaker and Vertex AI deployment","archived":false,"fork":false,"pushed_at":"2026-05-23T14:42:48.000Z","size":10,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2026-05-23T16:25:05.283Z","etag":null,"topics":["bert","huggingface","llama","llm-finetuning","lora","machine-learning","mlflow","nlp","peft","python","pytorch","qlora","transformers"],"latest_commit_sha":null,"homepage":null,"language":null,"has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/varunramagiri.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2026-05-23T14:39:59.000Z","updated_at":"2026-05-23T14:42:51.000Z","dependencies_parsed_at":null,"dependency_job_id":null,"html_url":"https://github.com/varunramagiri/llm-finetuning-toolkit","commit_stats":null,"previous_names":["varunramagiri/llm-finetuning-toolkit"],"tags_count":null,"template":false,"template_full_name":null,"purl":"pkg:github/varunramagiri/llm-finetuning-toolkit","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/varunramagiri%2Fllm-finetuning-toolkit","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/varunramagiri%2Fllm-finetuning-toolkit/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/varunramagiri%2Fllm-finetuning-toolkit/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/varunramagiri%2Fllm-finetuning-toolkit/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/varunramagiri","download_url":"https://codeload.github.com/varunramagiri/llm-finetuning-toolkit/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/varunramagiri%2Fllm-finetuning-toolkit/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":34268187,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-26T15:22:16.424Z","status":"online","status_checked_at":"2026-06-12T02:00:06.859Z","response_time":109,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["bert","huggingface","llama","llm-finetuning","lora","machine-learning","mlflow","nlp","peft","python","pytorch","qlora","transformers"],"created_at":"2026-06-13T00:33:05.626Z","updated_at":"2026-06-13T00:33:06.632Z","avatar_url":"https://github.com/varunramagiri.png","language":null,"funding_links":[],"categories":[],"sub_categories":[],"readme":"# 🧬 LLM Fine-Tuning Toolkit\n\n\u003e **Parameter-efficient fine-tuning (PEFT) for GPT, BERT, T5, LLaMA \u0026 Mistral** — LoRA · QLoRA · MLflow · SageMaker · Vertex AI · One-command deployment\n\n[![Python](https://img.shields.io/badge/Python-3.11+-3776AB?style=flat-square\u0026logo=python)](https://python.org)\n[![PyTorch](https://img.shields.io/badge/PyTorch-2.2+-EE4C2C?style=flat-square\u0026logo=pytorch)](https://pytorch.org)\n[![HuggingFace](https://img.shields.io/badge/🤗%20Transformers-4.40+-FFD21E?style=flat-square)](https://huggingface.co/transformers)\n[![MLflow](https://img.shields.io/badge/MLflow-2.12+-0194E2?style=flat-square)](https://mlflow.org)\n[![License: MIT](https://img.shields.io/badge/License-MIT-yellow?style=flat-square)](LICENSE)\n\n---\n\n## 🎯 Overview\n\nProduction-ready toolkit for fine-tuning large language models on domain-specific enterprise datasets using **LoRA (Low-Rank Adaptation)** and **QLoRA (4-bit quantized LoRA)** — dramatically reducing GPU memory requirements without sacrificing model quality.\n\nBuilt from fine-tuning work across **insurance policy summarization** (Nationwide), **clinical NLP** (CVS Health), and **document classification** tasks. Includes training, evaluation, experiment tracking, and cloud deployment — all scriptable via CLI or Python API.\n\n**Supported tasks:** Summarization · Classification · QA · NER · Instruction following · Chat fine-tuning\n\n**Supported models:** GPT-2 · BERT · RoBERTa · T5 · FLAN-T5 · LLaMA 2/3 · Mistral 7B · Falcon\n\n---\n\n## 📁 Folder Structure\n\n```\nllm-finetuning-toolkit/\n├── src/\n│   ├── training/\n│   │   ├── trainer.py              # Core PEFT trainer class\n│   │   ├── lora_config.py          # LoRA: rank, alpha, target modules\n│   │   ├── qlora_config.py         # QLoRA: 4-bit NF4, double quant\n│   │   └── callbacks.py            # Early stopping, checkpoint saving\n│   ├── data/\n│   │   ├── dataset_loader.py       # HuggingFace Hub + custom JSONL\n│   │   ├── preprocessor.py         # Tokenization, padding, truncation\n│   │   ├── augmentation.py         # Back-translation, synonym swap\n│   │   └── quality_filter.py       # Dedupe, length filter, toxicity\n│   ├── evaluation/\n│   │   ├── metrics.py              # BLEU, ROUGE, F1, BERTScore\n│   │   ├── bias_detector.py        # Fairness metrics across demographics\n│   │   ├── safety_eval.py          # ToxiGen, BBQ safety benchmarks\n│   │   └── benchmark.py            # Throughput, latency, memory profiling\n│   ├── inference/\n│   │   ├── predictor.py            # Single inference with adapter loading\n│   │   ├── batch_predictor.py      # Async batch inference (vLLM)\n│   │   └── quantizer.py            # Post-training quant (ONNX / TensorRT)\n│   └── deployment/\n│       ├── sagemaker_deploy.py     # AWS SageMaker real-time endpoint\n│       ├── vertex_deploy.py        # GCP Vertex AI endpoint\n│       ├── azure_deploy.py         # Azure ML managed endpoint\n│       └── vllm_server.py          # Local vLLM OpenAI-compatible server\n├── configs/\n│   ├── lora_bert_classification.yaml\n│   ├── qlora_llama2_7b_instruct.yaml\n│   ├── lora_t5_summarization.yaml\n│   └── qlora_mistral_7b_chat.yaml\n├── scripts/\n│   ├── train.py                    # CLI: python scripts/train.py --config ...\n│   ├── evaluate.py                 # CLI: python scripts/evaluate.py ...\n│   ├── merge_adapters.py           # Merge LoRA weights → base model\n│   ├── push_to_hub.py              # Push fine-tuned model to HF Hub\n│   └── export_onnx.py              # Export to ONNX for deployment\n├── notebooks/\n│   ├── 01_lora_finetuning_walkthrough.ipynb\n│   ├── 02_qlora_llama2_on_single_gpu.ipynb\n│   ├── 03_evaluation_and_bias_analysis.ipynb\n│   └── 04_inference_optimization.ipynb\n├── tests/\n│   ├── test_trainer.py\n│   ├── test_data_pipeline.py\n│   └── test_inference.py\n├── .github/\n│   └── workflows/\n│       ├── ci.yml\n│       └── model_eval.yml          # Automated eval on new checkpoints\n├── requirements.txt\n├── pyproject.toml\n└── README.md\n```\n\n---\n\n## ⚡ Quick Start\n\n### Python API — QLoRA fine-tuning on LLaMA 2 (single GPU, 8GB VRAM)\n\n```python\nfrom src.training.trainer import PEFTTrainer\nfrom src.training.qlora_config import QLoRAConfig\n\nconfig = QLoRAConfig(\n    base_model=\"meta-llama/Llama-2-7b-hf\",\n    dataset_path=\"data/insurance_policies.jsonl\",\n    task=\"text-generation\",\n    lora_r=16,\n    lora_alpha=32,\n    lora_dropout=0.05,\n    load_in_4bit=True,\n    bnb_4bit_quant_type=\"nf4\",\n    bnb_4bit_compute_dtype=\"bfloat16\",\n    max_seq_length=2048,\n    num_train_epochs=3,\n    per_device_train_batch_size=4,\n    gradient_accumulation_steps=4,\n    learning_rate=2e-4,\n    output_dir=\"outputs/llama2-insurance-finetuned\",\n    mlflow_experiment=\"llama2-qlora-insurance-v1\",\n)\n\ntrainer = PEFTTrainer(config)\ntrainer.train()\ntrainer.evaluate()\ntrainer.save_adapter(\"outputs/llama2-insurance-adapter\")\n```\n\n### CLI\n\n```bash\n# Train\npython scripts/train.py --config configs/qlora_llama2_7b_instruct.yaml\n\n# Evaluate on test set\npython scripts/evaluate.py \\\n  --model-path outputs/llama2-insurance-finetuned \\\n  --dataset data/test.jsonl \\\n  --metrics rouge,bleu,bertscore\n\n# Merge LoRA adapters into base model (for deployment)\npython scripts/merge_adapters.py \\\n  --base meta-llama/Llama-2-7b-hf \\\n  --adapter outputs/llama2-insurance-adapter \\\n  --output outputs/llama2-insurance-merged\n\n# Deploy to SageMaker\npython src/deployment/sagemaker_deploy.py \\\n  --model-path outputs/llama2-insurance-merged \\\n  --instance-type ml.g5.xlarge \\\n  --endpoint-name llama2-insurance-prod\n```\n\n---\n\n## 📊 GPU Memory — LoRA vs QLoRA vs Full Fine-Tuning\n\n| Technique | Model | GPU VRAM | Trainable Params |\n|---|---|---|---|\n| Full fine-tuning | LLaMA-7B | ~56 GB | 7B (100%) |\n| LoRA (r=16) | LLaMA-7B | ~18 GB | ~4M (0.06%) |\n| QLoRA 4-bit (r=16) | LLaMA-7B | **~8 GB** ✅ | ~4M (0.06%) |\n| QLoRA 4-bit (r=16) | LLaMA-13B | **~12 GB** ✅ | ~6M (0.05%) |\n| QLoRA 4-bit (r=64) | Mistral-7B | **~10 GB** ✅ | ~16M (0.23%) |\n\n---\n\n## 🔬 Experiment Tracking (MLflow)\n\nAll runs auto-logged: `train/loss`, `eval/loss`, `eval/rouge1`, `eval/rouge2`, `eval/rougeL`, `eval/bleu`, `eval/bertscore`, `train/learning_rate`, GPU utilization, peak memory.\n\n```bash\nmlflow ui --port 5000\n```\n\n---\n\n## 🚀 Deployment Options\n\n| Target | Latency | Throughput | Cost |\n|---|---|---|---|\n| vLLM local server | ~50ms | High | GPU instance |\n| AWS SageMaker RT | ~120ms | Auto-scaled | Pay-per-use |\n| GCP Vertex AI | ~130ms | Auto-scaled | Pay-per-use |\n| ONNX + TensorRT | ~20ms | Very high | GPU instance |\n\n---\n\n## 📄 License\n\nMIT — see [LICENSE](LICENSE)\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fvarunramagiri%2Fllm-finetuning-toolkit","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fvarunramagiri%2Fllm-finetuning-toolkit","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fvarunramagiri%2Fllm-finetuning-toolkit/lists"}