https://github.com/jordandeklerk/opencodeinterpreter-finetune-sql
Fine-tuning coding LLM OpenCodeInterpreter-DS-6.7B for Text-to-SQL Code Generation on a Single A100 GPU in PyTorch
https://github.com/jordandeklerk/opencodeinterpreter-finetune-sql
artificial-intelligence code-llms finetuning-llms llms machine-learning open-code-interpreter peft qlora
Last synced: 8 months ago
JSON representation
Fine-tuning coding LLM OpenCodeInterpreter-DS-6.7B for Text-to-SQL Code Generation on a Single A100 GPU in PyTorch
- Host: GitHub
- URL: https://github.com/jordandeklerk/opencodeinterpreter-finetune-sql
- Owner: jordandeklerk
- License: mit
- Created: 2024-04-01T15:20:54.000Z (over 1 year ago)
- Default Branch: main
- Last Pushed: 2024-06-06T20:22:35.000Z (over 1 year ago)
- Last Synced: 2024-12-31T11:43:41.554Z (9 months ago)
- Topics: artificial-intelligence, code-llms, finetuning-llms, llms, machine-learning, open-code-interpreter, peft, qlora
- Language: Jupyter Notebook
- Homepage:
- Size: 1.01 MB
- Stars: 1
- Watchers: 1
- Forks: 0
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
- License: LICENSE
Awesome Lists containing this project
README
# Highlights
This project provides a guide to fine-tuning the OpenCodeInterpreter-DS-6.7B coding LLM model for text-to-SQL code generation using the QLoRA+ technique. QLoRA+ is an improvement over the standard LoRA (Low-Rank Adaptation) approach that allows for different learning rates for the adapter matrices, significantly reducing the number of trainable parameters while maintaining model performance and speeding up fine-tuning by up to 2x. The fine-tuned model can generate accurate SQL queries based on natural language questions and database schemas. A Gradio app is created to showcase the model's capabilities, allowing users to interact with it in real-time by providing a schema and asking questions