https://github.com/cloneofsimo/fim-llama-deepspeed
https://github.com/cloneofsimo/fim-llama-deepspeed
Last synced: about 1 year ago
JSON representation
- Host: GitHub
- URL: https://github.com/cloneofsimo/fim-llama-deepspeed
- Owner: cloneofsimo
- Created: 2024-01-01T15:20:45.000Z (over 2 years ago)
- Default Branch: master
- Last Pushed: 2024-01-01T15:30:22.000Z (over 2 years ago)
- Last Synced: 2025-04-13T09:47:32.796Z (over 1 year ago)
- Language: Python
- Size: 693 KB
- Stars: 31
- Watchers: 3
- Forks: 0
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
Awesome Lists containing this project
README
# FIM-LLAMA and Deepspeed FIM Trainer
https://github.com/cloneofsimo/fim-llama-deepspeed/assets/35953539/e5f4e48c-c46b-4d33-a57a-05ce9568a795
Link to [Huggingface model](https://huggingface.co/cloneofsimo/fim-llama), fine-tuned [Tulu-2-dpo](https://huggingface.co/allenai/tulu-2-dpo-70b) for FIM capability based on this code.
---
Train llama to have Fill-in-the-middle capability, with pure deepspeed.
It is mostly copy-paste from [deepspeed-examples](https://github.com/microsoft/DeepSpeedExamples/blob/master/applications/DeepSpeed-Chat/training/step1_supervised_finetuning/main.py).
## Installation
```bash
deepspeed --num_gpus 8 main.py
```
## How to prompt FIM-LLAMA?
The model is trained with new tokens :
```python
new_tokens = ["<|SUFFIX|>", "<|PREFIX|>", "<|order|>", "<|STARTFIM|>","<|ENDMIDDLE|>", "<|MIDDLE_0|>", "<|MIDDLE_1|>", "<|MIDDLE_2|>", "<|MIDDLE_3|>", "<|MIDDLE_4|>", "<|MIDDLE_5|>", "<|MIDDLE_6|>"],
```
Here is how you would prompt it:
```python
instruction = f"<|user|> Complete the following text. <|assistant|><|STARTFIM|><|SUFFIX|>{suffix}<|PREFIX|>{prefix}<|MIDDLE|><|MIDDLE_N|>"
```
* $N$ is order of magnitude of the middle you want to fill in.
* Document is in form of prefix + MIDDLE_YOU_WANT + suffix.