https://github.com/datvodinh/llm-trainer

Finetuning script for Llama 3 in different tasks, using Unsloth and PEFT
https://github.com/datvodinh/llm-trainer

Last synced: about 18 hours ago
JSON representation

Finetuning script for Llama 3 in different tasks, using Unsloth and PEFT

• Host: GitHub
• URL: https://github.com/datvodinh/llm-trainer
• Owner: datvodinh
• License: mit
• Created: 2024-04-21T05:39:50.000Z (7 months ago)
• Default Branch: master
• Last Pushed: 2024-04-30T02:08:04.000Z (7 months ago)
• Last Synced: 2024-05-01T02:47:13.117Z (7 months ago)
• Language: Python
• Homepage:
• Size: 179 KB
• Stars: 1
• Watchers: 2
• Forks: 1
• Open Issues: 0
• Metadata Files:
  • Readme: README.md
  • License: LICENSE

Awesome Lists containing this project

README

        
# Finetuning script for Llama 3 in different tasks, using Unsloth and PEFT

## Spec

- ~ 9Gb VRAM (Batch size 2, Accumulate Grad 4)

## Setup

### Colab and Local

```bash

source scripts/setup_colab.sh

```

### Kaggle

```bash

source scripts/setup_kaggle.sh

```

## Train

### Function Calling

```bash

source scripts/train_function_calling.sh \

--hf_username  \

--hf_token  \

--wandb_token 

```

### Instruction

- Coming Soon

## Todo

- [ ] Instruction Fine-tuning.

- [ ] DPO Trainer.

- [ ] Custom dataset for different task.