https://github.com/buaadreamer/mllm-finetuning-demo
使用LLaMA-Factory微调多模态大语言模型的示例代码 Demo of Finetuning Multimodal LLM with LLaMA-Factory
https://github.com/buaadreamer/mllm-finetuning-demo
finetune-llm huggingface-datasets llama-factory llava lora mllm paligemma pretraining supervised-finetuning transformers yi-vl
Last synced: over 1 year ago
JSON representation
使用LLaMA-Factory微调多模态大语言模型的示例代码 Demo of Finetuning Multimodal LLM with LLaMA-Factory
- Host: GitHub
- URL: https://github.com/buaadreamer/mllm-finetuning-demo
- Owner: BUAADreamer
- Created: 2024-05-25T07:19:29.000Z (about 2 years ago)
- Default Branch: master
- Last Pushed: 2024-09-08T17:15:28.000Z (almost 2 years ago)
- Last Synced: 2025-04-11T00:02:40.015Z (over 1 year ago)
- Topics: finetune-llm, huggingface-datasets, llama-factory, llava, lora, mllm, paligemma, pretraining, supervised-finetuning, transformers, yi-vl
- Language: Python
- Homepage:
- Size: 61.5 KB
- Stars: 32
- Watchers: 1
- Forks: 2
- Open Issues: 2
-
Metadata Files:
- Readme: README.md
Awesome Lists containing this project
README
# MLLM-Finetuning-Demo
## 安装LLaMA-Factory
```shell
git clone https://github.com/hiyouga/LLaMA-Factory.git
cd LLaMA-Factory
pip install -e .[torch,metrics]
cd .. # 回到项目根目录
```
## 预训练
LLaVA中的特征对齐，冻结`language_model`和`vision_tower`，只微调`multi_modal_projector`。
```shell
CUDA_VISIBLE_DEVICES=0 llamafactory-cli train config/llava_pt.yaml
```
## 微调
```shell
CUDA_VISIBLE_DEVICES=0 llamafactory-cli train config/llava_lora_sft.yaml
```
## 网页聊天
```shell
CUDA_VISIBLE_DEVICES=0 llamafactory-cli webchat \
--model_name_or_path llava-hf/llava-1.5-7b-hf \
--adapter_name_or_path saves/llava1_5-7b/lora/sft \
--template llava
```
## 上传数据集到Huggingface
请在 `upload_dataset.py` 中替换您自己的key.
```shell
python3 upload_dataset.py
```
## 导出和上传模型到huggingface
请在 `config/llava_lora_sft_export.yaml` 中替换您自己的 `export_hub_model_id` 和 `hf_hub_token`.
```shell
CUDA_VISIBLE_DEVICES=0 llamafactory-cli export config/llava_lora_sft_export.yaml
```