https://github.com/jordandeklerk/starcoder2-finetune-code-completion
Finetuning Starcoder2-3B for Code Completion on a single A100 GPU
https://github.com/jordandeklerk/starcoder2-finetune-code-completion
artificial-intelligence code-llms finetuning-large-language-models llms lora machine-learning peft starcoder2
Last synced: 6 months ago
JSON representation
Finetuning Starcoder2-3B for Code Completion on a single A100 GPU
- Host: GitHub
- URL: https://github.com/jordandeklerk/starcoder2-finetune-code-completion
- Owner: jordandeklerk
- License: mit
- Created: 2024-04-01T15:12:27.000Z (over 1 year ago)
- Default Branch: main
- Last Pushed: 2024-11-04T23:40:26.000Z (11 months ago)
- Last Synced: 2025-03-28T16:21:29.320Z (7 months ago)
- Topics: artificial-intelligence, code-llms, finetuning-large-language-models, llms, lora, machine-learning, peft, starcoder2
- Language: Jupyter Notebook
- Homepage:
- Size: 575 KB
- Stars: 7
- Watchers: 1
- Forks: 1
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
- License: LICENSE
Awesome Lists containing this project
README
# Highlights
This project demonstrates the process of fine-tuning the Starcoder2-3B model, a code-generating LLM, on proprietary code, which we could imagine as a company's internal codebase, to better align with internal coding standards and leverage specialized libraries. Given the substantial size of these models, traditional fine-tuning approaches can be excessively demanding on computational resources. However, we'll introduce techniques to effectively fine-tune these models on just a single GPU using QLoRA, PEFT, and Bits and Bytes, ensuring a more practical approach for resource-limited environments.