Ecosyste.ms: Awesome
An open API service indexing awesome lists of open source software.
https://github.com/next-gpt/next-gpt
Code and models for NExT-GPT: Any-to-Any Multimodal Large Language Model
https://github.com/next-gpt/next-gpt
chatgpt foundation-models gpt-4 instruction-tuning large-language-models llm multi-modal-chatgpt multimodal visual-language-learning
Last synced: 24 days ago
JSON representation
Code and models for NExT-GPT: Any-to-Any Multimodal Large Language Model
- Host: GitHub
- URL: https://github.com/next-gpt/next-gpt
- Owner: NExT-GPT
- License: bsd-3-clause
- Created: 2023-08-30T03:34:11.000Z (about 1 year ago)
- Default Branch: main
- Last Pushed: 2024-10-12T15:49:10.000Z (27 days ago)
- Last Synced: 2024-10-15T06:21:38.866Z (24 days ago)
- Topics: chatgpt, foundation-models, gpt-4, instruction-tuning, large-language-models, llm, multi-modal-chatgpt, multimodal, visual-language-learning
- Language: Python
- Homepage: https://next-gpt.github.io/
- Size: 126 MB
- Stars: 3,250
- Watchers: 57
- Forks: 320
- Open Issues: 67
-
Metadata Files:
- Readme: README.md
- License: LICENSE.md
Awesome Lists containing this project
- awesome-ChatGPT-repositories - NExT-GPT - Code and models for NExT-GPT: Any-to-Any Multimodal Large Language Model (NLP)
README
# NExT-GPT: Any-to-Any Multimodal LLM
[Shengqiong Wu](https://chocowu.github.io/), [Hao Fei](http://haofei.vip/)*, [Leigang Qu](#), [Wei Ji](https://jiwei0523.github.io/), and [Tat-Seng Chua](https://www.chuatatseng.com/).
(*Correspondence )**[NExT++](https://www.nextcenter.org/), School of Computing, National University of Singapore**
-----
![License](https://img.shields.io/badge/License-BSD-blue.svg)
[![YouTube](https://badges.aleen42.com/src/youtube.svg)](https://www.youtube.com/watch?v=aqw2SCWeWD0)This repository hosts the code, data and model weight of **NExT-GPT**, the first end-to-end MM-LLM that perceives input and generates output in arbitrary combinations (any-to-any) of text, image, video, and audio and beyond.
-----------
## π News
- [x] [2023.09.15] ππ Release the code of NExT-GPT in version `7b_tiva_v0`.
- [x] [2023.09.27] π¨π§© Added modality-blended batch sampler.
- [x] [2023.10.01] π’π’ Release the T2M instruction dataset.
- [x] [2023.10.04] ππ Release the checkpoint of NExT-GPT in version [7b_tiva_v0](https://huggingface.co/ChocoWu/nextgpt_7b_tiva_v0) .
- [x] [2023.10.15] π¨π Update of NExT-GPT in version [7b_tiva_v0](https://huggingface.co/ChocoWu/nextgpt_7b_tiva_v0) .## π TODO
- [ ] Release MosIT data.
- [ ] Updating NExT-GPT in more types&sizes of LLMs.
- [ ] Empowering NExT-GPT with more modalities of inputs&outputs.
- [ ] ...-----------
## Example Demos
Here we showcase examples generated from NExT-GPT.
For more examples, kindly visit the [webpage](https://next-gpt.github.io/), or the online live [demo](https://acc414b22d6839d28f.gradio.live).https://github.com/NExT-GPT/NExT-GPT/assets/18722770/0c2b3d88-a533-4899-ab44-65580fe54538
https://github.com/NExT-GPT/NExT-GPT/assets/18722770/eb1319a6-38aa-4546-a96e-163207e7de93
https://github.com/NExT-GPT/NExT-GPT/assets/18722770/36bec0ad-9bad-4bcf-bc37-92b028f1bc6a
## Brief Introduction
NExt-GPT is built on top of existing pre-trained LLM, multimodal encoder and SoTA diffusion models, with sufficient end-to-end instruction tuning.
- **Multimodal Encoding Stage.** Leveraging established encoders to encode inputs in various modalities, where these representations are projected into language-like representations comprehensible to the LLM through a projection layer.
- **LLM Understanding and Reasoning Stage.** Harnessing an existing open-sourced LLM as the core to process input information for semantic understanding and reasoning. The LLM not only directly generates text tokens but also produces unique βmodality signalβ tokens that serve as instructions to dictate the decoding layers whether & what modal content to output correspondingly.
- **Multimodal Generation Stage.** Receiving the multimodal signals with specific instructions from LLM (if any), the Transformer-based output projection layers map the signal token representations into the ones that are understandable to following multimodal decoders.For more technical details, kindly refer to the [paper](https://arxiv.org/pdf/2309.05519.pdf).
-----------
## Getting Started
### Table of Contents:
* 1. Code Structure
* 2. Environment Preparation
* 3. Training/Adapting NExt-GPT on Your Own
* 3.1. Preparing Pre-trained Checkpoint
* 3.2. Preparing Dataset
* 3.3. Precomputing Embeddings
* 3.4. Training NExT-GPT
* 4. Running NExT-GPT System
* 4.1. Preparing checkpoints
* 4.2. Deploying Demo System****
### 1. Code Structure
```
βββ figures
βββ data
β βββ T-X_pair_data
β β βββ audiocap # text-autio pairs data
β β β βββ audios # audio files
β β β βββ audiocap.json # the audio captions
β β βββ cc3m # text-image paris data
β β β βββ images # image files
β β β βββ cc3m.json # the image captions
β β βββ webvid # text-video pairs data
β β β βββ videos # video files
β β β βββ webvid.json # the video captions
β βββ IT_data # instruction data
β β βββ T+X-T_data # text+[image/audio/video] to text instruction data
β β β βββ alpaca # textual instruction data
β β β βββ llava # visual instruction data
β β βββ T-T+X # synthesized text to text+[image/audio/video] instruction data
β β βββ MosIT # Modality-switching Instruction Tuning instruction data
βββ code
β βββ config
β β βββ base.yaml # the model configuration
β β βββ stage_1.yaml # enc-side alignment training configuration
β β βββ stage_2.yaml # dec-side alignment training configuration
β β βββ stage_3.yaml # instruction-tuning configuration
β βββ dsconfig
β β βββ stage_1.json # deepspeed configuration for enc-side alignment training
β β βββ stage_2.json # deepspeed configuration for dec-side alignment training
β β βββ stage_3.json # deepspeed configuration for instruction-tuning training
β βββ datast
β β βββ base_dataset.py
β β βββ catalog.py # the catalog information of the dataset
β β βββ cc3m_datast.py # process and load text-image pair dataset
β β βββ audiocap_datast.py # process and load text-audio pair dataset
β β βββ webvid_dataset.py # process and load text-video pair dataset
β β βββ T+X-T_instruction_dataset.py # process and load text+x-to-text instruction dataset
β β βββ T-T+X_instruction_dataset.py # process and load text-to-text+x instruction dataset
β β βββ concat_dataset.py # process and load multiple dataset
β βββ model
β β βββ ImageBind # the code from ImageBind Model
β β βββ common
β β βββ anyToImageVideoAudio.py # the main model file
β β βββ agent.py
β β βββ modeling_llama.py
β β βββ custom_ad.py # the audio diffusion
β β βββ custom_sd.py # the image diffusion
β β βββ custom_vd.py # the video diffusion
β β βββ layers.py # the output projection layers
β β βββ ...
β βββ scripts
β β βββ train.sh # training NExT-GPT script
β β βββ app.sh # deploying demo script
β βββ header.py
β βββ process_embeddings.py # precompute the captions embeddings
β βββ train.py # training
β βββ inference.py # inference
β βββ demo_app.py # deploy Gradio demonstration
β βββ ...
βββ ckpt
β βββ delta_ckpt # tunable NExT-GPT params
β β βββ nextgpt
β β β βββ 7b_tiva_v0 # the directory to save the log file
β β β β βββ log # the logs
β βββ ...
β βββ pretrained_ckpt # frozen params of pretrained modules
β β βββ imagebind_ckpt
β β β βββhuge # version
β β β β βββimagebind_huge.pth
β β βββ vicuna_ckpt
β β β βββ 7b_v0 # version
β β β β βββ config.json
β β β β βββ pytorch_model-00001-of-00002.bin
β β β β βββ tokenizer.model
β β β β βββ ...
βββ LICENCE.md
βββ README.md
βββ requirements.txt
```
### 2. Environment Preparation [Back to Top]
Please first clone the repo and install the required environment, which can be done by running the following commands:
```
conda env create -n nextgpt python=3.8conda activate nextgpt
# CUDA 11.6
conda install pytorch==1.13.1 torchvision==0.14.1 torchaudio==0.13.1 pytorch-cuda=11.6 -c pytorch -c nvidiagit clone https://github.com/NExT-GPT/NExT-GPT.git
cd NExT-GPTpip install -r requirements.txt
```
### 3. Training/Adapting NExt-GPT on Your Own
####
#### 3.1. Preparing Pre-trained Checkpoint [Back to Top]
NExT-GPT is trained based on following excellent existing models.
Please follow the instructions to prepare the checkpoints.- `ImageBind`
is the unified image/video/audio encoder. The pre-trained checkpoint can be downloaded from [here](https://dl.fbaipublicfiles.com/imagebind/imagebind_huge.pth) with version `huge`. Afterward, put the `imagebind_huge.pth` file at [[./ckpt/pretrained_ckpt/imagebind_ckpt/huge]](ckpt/pretrained_ckpt/imagebind_ckpt/).
- `Vicuna`:
first prepare the LLaMA by following the instructions [[here]](ckpt/pretrained_ckpt/prepare_vicuna.md). Then put the pre-trained model at [[./ckpt/pretrained_ckpt/vicuna_ckpt/]](ckpt/pretrained_ckpt/vicuna_ckpt/).
- `Image Diffusion`
is used to generate images. NExT-GPT uses [Stable Diffusion](https://huggingface.co/runwayml/stable-diffusion-v1-5) with version `
v1-5`. (_will be automatically downloaded_)
- `Audio Diffusion`
for producing audio content. NExT-GPT employs [AudioLDM](https://github.com/haoheliu/AudioLDM) with version `l-full`. (_will be automatically downloaded_)
- `Video Diffusion`
for the video generation. We employ [ZeroScope](https://huggingface.co/cerspense/zeroscope_v2_576w) with version `v2_576w`. (_will be automatically downloaded_)
#### 3.2. Preparing Dataset [Back to Top]
Please download the following datasets used for model training:A) T-X pairs data
- `CC3M` of ***text-image*** pairs, please follow this instruction [[here]](./data/T-X_pair_data/cc3m/prepare.md). Then put the data at [[./data/T-X_pair_data/cc3m]](./data/T-X_pair_data/cc3m).
- `WebVid` of ***text-video*** pairs, see the [[instruction]](./data/T-X_pair_data/webvid/prepare.md). The file should be saved at [[./data/T-X_pair_data/webvid]](./data/T-X_pair_data/webvid).
- `AudioCap` of ***text-audio*** pairs, see the [[instruction]](./data/T-X_pair_data/audiocap/prepare.md). Save the data in [[./data/T-X_pair_data/audiocap]](./data/T-X_pair_data/audiocap).B) Instruction data
- T+X-T
- `LLaVA` of the ***visual instruction data***, download it from [here](https://github.com/haotian-liu/LLaVA/blob/main/docs/Data.md), and then put it at [[./data/IT_data/T+X-T_data/llava]](./data/IT_data/T+X-T_data/llava/).
- `Alpaca` of the ***textual instruction data***, download it from [here](https://github.com/tatsu-lab/stanford_alpaca), and then put it at [[./data/IT_data/T+X-T_data/alpaca/]](data/IT_data/T+X-T_data/alpaca/).
- `VideoChat`, download the ***video instruction data*** [here](https://github.com/OpenGVLab/InternVideo/tree/main/Data/instruction_data), and then put it at [[./data/IT_data/T+X-T_data/videochat/]](data/IT_data/T+X-T_data/videochat/).
Side noteοΌAfter downloading dataset, please run `preprocess_dataset.py` to preprocess the dataset into a unified format.
- T-X+T (T2M)
- The `T-X+T` instruction datasets (T2M) are saved at [[./data/IT_data/T-T+X_data]](./data/IT_data/T-T+X_data).
- MosIT
- Download the file from [here](), put them in [[./data/IT_data/MosIT_data/]](./data/IT_data/MosIT_data/). (_We are in the process of finalizing the data and handling the copyright issue. Will release later._)
#### 3.3. Precomputing Embeddings [Back to Top]
In decoding-side alignment training, we minimize the distance between the representation of signal tokens and captions.
To save costs of time and memory, we precompute the text embeddings for image, audio and video captions using the text encoder within the respective diffusion models.Please run this command before the following training of NExT-GPT, where the produced `embedding` file will be saved at [[./data/embed]](./data/embed).
```angular2html
cd ./code/
python process_embeddings.py ../data/T-X_pair_data/cc3m/cc3m.json image ../data/embed/ runwayml/stable-diffusion-v1-5
```Note of arguments:
- args[1]: path of caption file;
- args[2]: modality, which can be `image`, `video`, and `audio`;
- args[3]: saving path of embedding file;
- args[4]: corresponding pre-trained diffusion model name.
#### 3.4. Training NExT-GPT [Back to Top]
First of all, please refer to the base configuration file [[./code/config/base.yaml]](./code/config/base.yaml) for the basic system setting of overall modules.
Then, the training of NExT-GPT starts with this script:
```angular2html
cd ./code
bash scripts/train.sh
```
Specifying the command:
```angular2html
deepspeed --include localhost:0 --master_addr 127.0.0.1 --master_port 28459 train.py \
--model nextgpt \
--stage 1\
--save_path ../ckpt/delta_ckpt/nextgpt/7b_tiva_v0/\
--log_path ../ckpt/delta_ckpt/nextgpt/7b_tiva_v0/log/
```
where the key arguments are:
- `--include`: `localhost:0` indicating the GPT cuda number `0` of deepspeed.
- `--stage`: training stage.
- `--save_path`: the directory which saves the trained delta weights. This directory will be automatically created.
- `--log_path`: the directory which saves the log file.The whole NExT-GPT training involves 3 steps:
- **Step-1**: Encoding-side LLM-centric Multimodal Alignment. This stage trains the ***input projection layer*** while freezing the ImageBind, LLM, output projection layer.
Just run the above `train.sh` script by setting: `--stage 1`
Also refer to the running config file [[./code/config/stage_1.yaml]](./code/config/stage_1.yaml) and deepspeed config file [[./code/dsconfig/stage_1.yaml]](./code/dsconfig/stage_1.yaml) for more step-wise configurations.Note that the dataset used for training in this step is included `dataset_name_list` and the dataset name must precisely match the definition in [[./code/dataset/catalog.py]](./code/dataset/catalog.py)
- **Step-2**: Decoding-side Instruction-following Alignment. This stage trains the ***output projection layers*** while freezing the ImageBind, LLM, input projection layers.
Just run the above `train.sh` script by setting: `--stage 2`
Also refer to the running config file [[./code/config/stage_2.yaml]](./code/config/stage_2.yaml) and deepspeed config file [[./code/dsconfig/stage_2.yaml]](./code/dsconfig/stage_2.yaml) for more step-wise configurations.
- **Step-3**: Instruction Tuning. This stage instruction-tune 1) the ***LLM*** via LoRA, 2) ***input projection layer*** and 3) ***output projection layer*** on the instruction dataset.
Just run the above `train.sh` script by setting: `--stage 3`
Also refer to the running config file [[./code/config/stage_3.yaml]](./code/config/stage_3.yaml) and deepspeed config file [[./code/dsconfig/stage_3.yaml]](./code/dsconfig/stage_3.yaml) for more step-wise configurations.
## 4. Running NExT-GPT System [Back to Top]
#### 4.1. Preparing Checkpoints
First, loading the pre-trained NExT-GPT system.
- **Step-1**: load `Frozen parameters`. Please refer to 3.1 Preparing Pre-trained Checkpoint.- **Step-2**: load `Tunable parameters`. Please put the NExT-GPT system at [[./ckpt/delta_ckpt/nextgpt/7b_tiva_v0]](./ckpt/delta_ckpt/nextgpt/7b_tiva_v0). You may either 1) use the params trained yourselves, or 2) download our checkpoints from [Huggingface](https://huggingface.co/ChocoWu/nextgpt_7b_tiva_v0).
#### 4.2. Deploying Gradio Demo
Upon completion of the checkpoint loading, you can run the demo locally via:
```angular2html
cd ./code
bash scripts/app.sh
```
Specifying the key arguments as:
- `--nextgpt_ckpt_path`: the path of pre-trained NExT-GPT params.---------
## Contact
For any questions or feedback, feel free to contact [Shengqiong Wu](mailto:[email protected]) and [Hao Fei](mailto:[email protected]).
## Citation
If you find NextGPT useful in your research or applications, please kindly cite:
```
@articles{wu2023nextgpt,
title={NExT-GPT: Any-to-Any Multimodal LLM},
author={Shengqiong Wu and Hao Fei and Leigang Qu and Wei Ji and Tat-Seng Chua},
journal = {CoRR},
volume = {abs/2309.05519},
year={2023}
}
```## Acknowledgements
You may refer to related work that serves as foundations for our framework and code repository,
[Vicuna](https://github.com/lm-sys/FastChat),
[ImageBind](https://github.com/facebookresearch/ImageBind),
[Stable Diffusion](https://huggingface.co/docs/diffusers/api/pipelines/stable_diffusion/text2img),
[AudioLDM](https://github.com/haoheliu/AudioLDM), and
[Zeroscope](https://huggingface.co/cerspense/zeroscope_v2_576w).
We also partially draw inspirations from
[PandaGPT](https://github.com/yxuansu/PandaGPT),
[VPGTrans](https://vpgtrans.github.io/),
[GILL](https://github.com/kohjingyu/gill/),
[CoDi](https://codi-gen.github.io/),
[Video-LLaMA](https://github.com/DAMO-NLP-SG/Video-LLaMA),
and [MiniGPT-4](https://github.com/Vision-CAIR/MiniGPT-4).
Thanks for their wonderful works.## License Notices
This repository is under [BSD 3-Clause License](LICENSE.txt).
NExT-GPT is a research project intended for non-commercial use only.
One must NOT use the code of NExT-GPT for any illegal, harmful, violent, racist, or sexual purposes.
One is strictly prohibited from engaging in any activity that will potentially violate these guidelines.
Any potential commercial use of this code should be approved by the authors.