https://github.com/fisheeped/creek
生成模型 tokenizer训练，模型初始化，模型预训练，指令微调。llama，creek
https://github.com/fisheeped/creek
Last synced: about 1 month ago
JSON representation
生成模型 tokenizer训练，模型初始化，模型预训练，指令微调。llama，creek
- Host: GitHub
- URL: https://github.com/fisheeped/creek
- Owner: fisheeped
- Created: 2024-04-06T16:40:02.000Z (about 1 year ago)
- Default Branch: main
- Last Pushed: 2024-12-13T01:53:38.000Z (5 months ago)
- Last Synced: 2024-12-13T02:30:44.068Z (5 months ago)
- Language: Python
- Size: 804 KB
- Stars: 2
- Watchers: 2
- Forks: 0
- Open Issues: 0
-
Metadata Files:
- Readme: readme.md
Awesome Lists containing this project
- awesome-hacking-lists - fisheeped/creek - 生成模型 tokenizer训练，模型初始化，模型预训练，指令微调。llama，creek (Python)
README
🤗https://huggingface.co/maheer/creek
# 从零创建生成大模型
下面代码中的参数，资源占用：预训练A800*8，每张卡显存占用70G+。
可以调小batch_size,max_length，最少单卡12G显存应该能完成下面步骤。
**train tokenizer**
```shell
python model_init/tokenization/train_eval_tokenizer.py
```
**init model**
```shell
python model_init/model_init.py
```
**pretrain**
```shell
bash pretrain.sh
```
**finetune**
```shell
bash sft.sh
```