{"id":13347273,"url":"https://github.com/NExT-GPT/NExT-GPT","last_synced_at":"2025-03-12T08:31:26.753Z","repository":{"id":191645508,"uuid":"684881857","full_name":"NExT-GPT/NExT-GPT","owner":"NExT-GPT","description":"Code and models for NExT-GPT: Any-to-Any Multimodal Large Language Model","archived":false,"fork":false,"pushed_at":"2024-10-17T09:36:09.000Z","size":132642,"stargazers_count":3254,"open_issues_count":68,"forks_count":321,"subscribers_count":57,"default_branch":"main","last_synced_at":"2024-10-19T13:03:20.764Z","etag":null,"topics":["chatgpt","foundation-models","gpt-4","instruction-tuning","large-language-models","llm","multi-modal-chatgpt","multimodal","visual-language-learning"],"latest_commit_sha":null,"homepage":"https://next-gpt.github.io/","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"bsd-3-clause","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/NExT-GPT.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE.md","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2023-08-30T03:34:11.000Z","updated_at":"2024-10-18T12:41:39.000Z","dependencies_parsed_at":"2023-08-30T22:59:40.499Z","dependency_job_id":"5242cbd2-eded-44c5-8632-e771b32e6b9d","html_url":"https://github.com/NExT-GPT/NExT-GPT","commit_stats":null,"previous_names":["next-gpt/next-gpt"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/NExT-GPT%2FNExT-GPT","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/NExT-GPT%2FNExT-GPT/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/NExT-GPT%2FNExT-GPT/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/NExT-GPT%2FNExT-GPT/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/NExT-GPT","download_url":"https://codeload.github.com/NExT-GPT/NExT-GPT/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":221279984,"owners_count":16790563,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["chatgpt","foundation-models","gpt-4","instruction-tuning","large-language-models","llm","multi-modal-chatgpt","multimodal","visual-language-learning"],"created_at":"2024-07-29T20:01:35.920Z","updated_at":"2024-10-24T05:30:58.137Z","avatar_url":"https://github.com/NExT-GPT.png","language":"Python","funding_links":[],"categories":["Python","Project List","多模态大模型","Repos","SDK, Libraries, Frameworks","📦 Legacy \u0026 Inactive Projects"],"sub_categories":["\u003cspan id=\"tool\"\u003eLLM (LLM \u0026 Tool)\u003c/span\u003e","网络服务_其他","Python library, sdk or frameworks"],"readme":"# \u003cimg src=\"code/nextgpt.png\" style=\"width: 5%\"\u003e NExT-GPT: Any-to-Any Multimodal LLM\n[Shengqiong Wu](https://chocowu.github.io/), [Hao Fei](http://haofei.vip/)*, [Leigang Qu](#), [Wei Ji](https://jiwei0523.github.io/), and [Tat-Seng Chua](https://www.chuatatseng.com/).\n(*Correspondence )\n\n**[NExT++](https://www.nextcenter.org/), School of Computing, National University of Singapore**\n\n-----\n\n\u003ca href='https://next-gpt.github.io/'\u003e\u003cimg src='https://img.shields.io/badge/Project-Page-Green'\u003e\u003c/a\u003e\n\u003ca href='#'\u003e\u003cimg src='https://img.shields.io/badge/Demo-Page-purple'\u003e\u003c/a\u003e \n\u003ca href='https://arxiv.org/pdf/2309.05519'\u003e\u003cimg src='https://img.shields.io/badge/Paper-PDF-orange'\u003e\u003c/a\u003e \n![License](https://img.shields.io/badge/License-BSD-blue.svg)\n[![YouTube](https://badges.aleen42.com/src/youtube.svg)](https://www.youtube.com/watch?v=aqw2SCWeWD0)\n\n\nThis repository hosts the code, data and model weight of **NExT-GPT**, the first end-to-end MM-LLM that perceives input and generates output in arbitrary combinations (any-to-any) of text, image, video, and audio and beyond.\n\n\n\n-----------\n\n## 🎉 News \n\n- [x] [2023.09.15] 🚀🚀 Release the code of NExT-GPT in version `7b_tiva_v0`.\n- [x] [2023.09.27] 🔨🧩 Added modality-blended batch sampler .\n- [x] [2023.10.01] 📢📢 Release the T2M instruction dataset.\n- [x] [2023.10.04] 👏👏 Release the checkpoint of NExT-GPT in version [7b_tiva_v0](https://huggingface.co/ChocoWu/nextgpt_7b_tiva_v0) .\n- [x] [2023.10.15] 🔨🚀 Update of NExT-GPT in version [7b_tiva_v0](https://huggingface.co/ChocoWu/nextgpt_7b_tiva_v0) .\n\n\n## 👉 TODO \n- [ ] Release MosIT data.\n- [ ] Updating NExT-GPT in more types\u0026sizes of LLMs.\n- [ ] Empowering NExT-GPT with more modalities of inputs\u0026outputs.\n- [ ] ...\n\n\n\n-----------\n\n## Example Demos\nHere we showcase examples generated from NExT-GPT.\nFor more examples, kindly visit the [webpage](https://next-gpt.github.io/), or the online live [demo](https://acc414b22d6839d28f.gradio.live). \n\n\nhttps://github.com/NExT-GPT/NExT-GPT/assets/18722770/0c2b3d88-a533-4899-ab44-65580fe54538\n\n\nhttps://github.com/NExT-GPT/NExT-GPT/assets/18722770/eb1319a6-38aa-4546-a96e-163207e7de93\n\n\nhttps://github.com/NExT-GPT/NExT-GPT/assets/18722770/36bec0ad-9bad-4bcf-bc37-92b028f1bc6a\n\n\n\n\u003cspan id='introduction'/\u003e\n\n## Brief Introduction \n\n\nNExt-GPT is built on top of existing pre-trained LLM, multimodal encoder and SoTA diffusion models, with sufficient end-to-end instruction tuning.\n\n\u003cp align=\"center\" width=\"100%\"\u003e\n\u003ca target=\"_blank\"\u003e\u003cimg src=\"figures/framework.png\" alt=\"Video-LLaMA\" style=\"width: 90%; min-width: 200px; display: block; margin: auto;\"\u003e\u003c/a\u003e\n\u003c/p\u003e\n\n- **Multimodal Encoding Stage.** Leveraging established encoders to encode inputs in various modalities, where these representations are projected into language-like representations comprehensible to the LLM through a projection layer.\n- **LLM Understanding and Reasoning Stage.** Harnessing an existing open-sourced LLM as the core to process input information for semantic understanding and reasoning. The LLM not only directly generates text tokens but also produces unique “modality signal” tokens that serve as instructions to dictate the decoding layers whether \u0026 what modal content to output correspondingly.\n- **Multimodal Generation Stage.** Receiving the multimodal signals with specific instructions from LLM (if any), the Transformer-based output projection layers map the signal token representations into the ones that are understandable to following multimodal decoders.\n\n\nFor more technical details, kindly refer to the [paper](https://arxiv.org/pdf/2309.05519.pdf). \n\n\n-----------\n\n\n\u003cspan id='Usage'/\u003e\n\n## Getting Started\n\n\n\n\u003cspan id='all_catelogue'/\u003e\n\n### Table of Contents:\n* \u003ca href='#Code Structure'\u003e1. Code Structure\u003c/a\u003e\n* \u003ca href='#Environment Preparation'\u003e2. Environment Preparation \u003c/a\u003e\n* \u003ca href='#Training on Your Own'\u003e3. Training/Adapting NExt-GPT on Your Own\u003c/a\u003e\n  * \u003ca href='#Prepare Pre-trained Checkpoint'\u003e3.1. Preparing Pre-trained Checkpoint\u003c/a\u003e\n  * \u003ca href='#Prepare Dataset'\u003e3.2. Preparing Dataset \u003c/a\u003e\n  * \u003ca href='#Precompute Embeddings'\u003e3.3. Precomputing Embeddings\u003c/a\u003e\n  * \u003ca href='#Train NExT-GPT'\u003e3.4. Training NExT-GPT\u003c/a\u003e\n* \u003ca href='#Run NExT-GPT System'\u003e4. Running NExT-GPT System\u003c/a\u003e\n  * \u003ca href='#Prepare checkpoints'\u003e4.1. Preparing checkpoints\u003c/a\u003e\n  * \u003ca href='#Deploy Demo System'\u003e4.2. Deploying Demo System\u003c/a\u003e\n\n****\n\n\n\n\n\n\u003cspan id='Code Structure'/\u003e\n\n### 1. Code Structure \n\n```\n├── figures\n├── data\n│   ├── T-X_pair_data  \n│   │   ├── audiocap                      # text-autio pairs data\n│   │   │   ├── audios                    # audio files\n│   │   │   └── audiocap.json             # the audio captions\n│   │   ├── cc3m                          # text-image paris data\n│   │   │   ├── images                    # image files\n│   │   │   └── cc3m.json                 # the image captions\n│   │   └── webvid                        # text-video pairs data\n│   │   │   ├── videos                    # video files\n│   │   │   └── webvid.json               # the video captions\n│   ├── IT_data                           # instruction data\n│   │   ├── T+X-T_data                    # text+[image/audio/video] to text instruction data\n│   │   │   ├── alpaca                    # textual instruction data\n│   │   │   ├── llava                     # visual instruction data\n│   │   ├── T-T+X                         # synthesized text to text+[image/audio/video] instruction data\n│   │   └── MosIT                         # Modality-switching Instruction Tuning instruction data\n├── code\n│   ├── config\n│   │   ├── base.yaml                     # the model configuration \n│   │   ├── stage_1.yaml                  # enc-side alignment training configuration\n│   │   ├── stage_2.yaml                  # dec-side alignment training configuration\n│   │   └── stage_3.yaml                  # instruction-tuning configuration\n│   ├── dsconfig\n│   │   ├── stage_1.json                  # deepspeed configuration for enc-side alignment training\n│   │   ├── stage_2.json                  # deepspeed configuration for dec-side alignment training\n│   │   └── stage_3.json                  # deepspeed configuration for instruction-tuning training\n│   ├── datast\n│   │   ├── base_dataset.py\n│   │   ├── catalog.py                    # the catalog information of the dataset\n│   │   ├── cc3m_datast.py                # process and load text-image pair dataset\n│   │   ├── audiocap_datast.py            # process and load text-audio pair dataset\n│   │   ├── webvid_dataset.py             # process and load text-video pair dataset\n│   │   ├── T+X-T_instruction_dataset.py  # process and load text+x-to-text instruction dataset\n│   │   ├── T-T+X_instruction_dataset.py  # process and load text-to-text+x instruction dataset\n│   │   └── concat_dataset.py             # process and load multiple dataset\n│   ├── model                     \n│   │   ├── ImageBind                     # the code from ImageBind Model\n│   │   ├── common\n│   │   ├── anyToImageVideoAudio.py       # the main model file\n│   │   ├── agent.py\n│   │   ├── modeling_llama.py\n│   │   ├── custom_ad.py                  # the audio diffusion \n│   │   ├── custom_sd.py                  # the image diffusion\n│   │   ├── custom_vd.py                  # the video diffusion\n│   │   ├── layers.py                     # the output projection layers\n│   │   └── ...  \n│   ├── scripts\n│   │   ├── train.sh                      # training NExT-GPT script\n│   │   └── app.sh                        # deploying demo script\n│   ├── header.py\n│   ├── process_embeddings.py             # precompute the captions embeddings\n│   ├── train.py                          # training\n│   ├── inference.py                      # inference\n│   ├── demo_app.py                       # deploy Gradio demonstration \n│   └── ...\n├── ckpt                           \n│   ├── delta_ckpt                        # tunable NExT-GPT params\n│   │   ├── nextgpt         \n│   │   │   ├── 7b_tiva_v0                # the directory to save the log file\n│   │   │   │   ├── log                   # the logs\n│   └── ...       \n│   ├── pretrained_ckpt                   # frozen params of pretrained modules\n│   │   ├── imagebind_ckpt\n│   │   │   ├──huge                       # version\n│   │   │   │   └──imagebind_huge.pth\n│   │   ├── vicuna_ckpt\n│   │   │   ├── 7b_v0                     # version\n│   │   │   │   ├── config.json\n│   │   │   │   ├── pytorch_model-00001-of-00002.bin\n│   │   │   │   ├── tokenizer.model\n│   │   │   │   └── ...\n├── LICENCE.md\n├── README.md\n└── requirements.txt\n```\n\n\n\u003cspan id='Environment Preparation'/\u003e\n\n\n### 2. Environment Preparation  \u003ca href='#all_catelogue'\u003e[Back to Top]\u003c/a\u003e\nPlease first clone the repo and install the required environment, which can be done by running the following commands:\n```\nconda env create -n nextgpt python=3.8\n\nconda activate nextgpt\n\n# CUDA 11.6\nconda install pytorch==1.13.1 torchvision==0.14.1 torchaudio==0.13.1 pytorch-cuda=11.6 -c pytorch -c nvidia\n\ngit clone https://github.com/NExT-GPT/NExT-GPT.git\ncd NExT-GPT\n\npip install -r requirements.txt\n```\n\n\u003cspan id='Training on Your Own'/\u003e\n\n### 3. Training/Adapting NExt-GPT on Your Own \n\n####\n\n\n\n\u003cspan id='Prepare Pre-trained Checkpoint'/\u003e\n\n#### 3.1. Preparing Pre-trained Checkpoint  \u003ca href='#all_catelogue'\u003e[Back to Top]\u003c/a\u003e\nNExT-GPT is trained based on following excellent existing models.\nPlease follow the instructions to prepare the checkpoints.\n\n- `ImageBind`\nis the unified image/video/audio encoder. The pre-trained checkpoint can be downloaded from [here](https://dl.fbaipublicfiles.com/imagebind/imagebind_huge.pth) with version `huge`. Afterward, put the `imagebind_huge.pth` file at [[./ckpt/pretrained_ckpt/imagebind_ckpt/huge]](ckpt/pretrained_ckpt/imagebind_ckpt/). \n- `Vicuna`:\nfirst prepare the LLaMA by following the instructions [[here]](ckpt/pretrained_ckpt/prepare_vicuna.md). Then put the pre-trained model at [[./ckpt/pretrained_ckpt/vicuna_ckpt/]](ckpt/pretrained_ckpt/vicuna_ckpt/). \n- `Image Diffusion`\nis used to generate images. NExT-GPT uses [Stable Diffusion](https://huggingface.co/runwayml/stable-diffusion-v1-5) with version `\nv1-5`. (_will be automatically downloaded_)\n- `Audio Diffusion`\nfor producing audio content. NExT-GPT employs [AudioLDM](https://github.com/haoheliu/AudioLDM) with version `l-full`. (_will be automatically downloaded_)\n- `Video Diffusion`\nfor the video generation. We employ [ZeroScope](https://huggingface.co/cerspense/zeroscope_v2_576w) with version `v2_576w`. (_will be automatically downloaded_)\n\n\n\n\u003cspan id='Prepare Dataset'/\u003e\n\n#### 3.2. Preparing Dataset  \u003ca href='#all_catelogue'\u003e[Back to Top]\u003c/a\u003e\nPlease download the following datasets used for model training:\n\nA) T-X pairs data\n  - `CC3M` of ***text-image*** pairs, please follow this instruction [[here]](./data/T-X_pair_data/cc3m/prepare.md). Then put the data at [[./data/T-X_pair_data/cc3m]](./data/T-X_pair_data/cc3m).\n  - `WebVid` of ***text-video*** pairs, see the [[instruction]](./data/T-X_pair_data/webvid/prepare.md). The file should be saved at [[./data/T-X_pair_data/webvid]](./data/T-X_pair_data/webvid).\n  - `AudioCap` of ***text-audio*** pairs, see the [[instruction]](./data/T-X_pair_data/audiocap/prepare.md). Save the data in [[./data/T-X_pair_data/audiocap]](./data/T-X_pair_data/audiocap).\n\nB) Instruction data\n  - T+X-T\n    - `LLaVA` of the ***visual instruction data***, download it from [here](https://github.com/haotian-liu/LLaVA/blob/main/docs/Data.md), and then put it at [[./data/IT_data/T+X-T_data/llava]](./data/IT_data/T+X-T_data/llava/).\n    - `Alpaca` of the ***textual instruction data***, download it from [here](https://github.com/tatsu-lab/stanford_alpaca), and then put it at [[./data/IT_data/T+X-T_data/alpaca/]](data/IT_data/T+X-T_data/alpaca/).\n    - `VideoChat`, download the ***video instruction data*** [here](https://github.com/OpenGVLab/InternVideo/tree/main/Data/instruction_data), and then put it at [[./data/IT_data/T+X-T_data/videochat/]](data/IT_data/T+X-T_data/videochat/).\n    \n    Side note：After downloading dataset, please run `preprocess_dataset.py` to preprocess the dataset into a unified format.\n  - T-X+T (T2M)\n    - The `T-X+T` instruction datasets (T2M) are saved at [[./data/IT_data/T-T+X_data]](./data/IT_data/T-T+X_data).\n   \n  - MosIT\n    - Download the file from [here](), put them in [[./data/IT_data/MosIT_data/]](./data/IT_data/MosIT_data/). (_We are in the process of finalizing the data and handling the copyright issue. Will release later._) \n\n\n\u003cspan id='Precompute Embeddings'/\u003e\n\n#### 3.3. Precomputing Embeddings \u003ca href='#all_catelogue'\u003e[Back to Top]\u003c/a\u003e\nIn decoding-side alignment training, we minimize the distance between the representation of signal tokens and captions. \nTo save costs of time and memory, we precompute the text embeddings for image, audio and video captions using the text encoder within the respective diffusion models.  \n\nPlease run this command before the following training of NExT-GPT, where the produced `embedding` file will be saved at [[./data/embed]](./data/embed).\n```angular2html\ncd ./code/\npython process_embeddings.py ../data/T-X_pair_data/cc3m/cc3m.json image ../data/embed/ runwayml/stable-diffusion-v1-5\n```\n\nNote of arguments:\n- args[1]: path of caption file;\n- args[2]: modality, which can be `image`, `video`, and `audio`;\n- args[3]: saving path of embedding file;\n- args[4]: corresponding pre-trained diffusion model name.\n\n\n\n\u003cspan id='Train NExT-GPT'/\u003e\n\n#### 3.4. Training NExT-GPT  \u003ca href='#all_catelogue'\u003e[Back to Top]\u003c/a\u003e\n\nFirst of all, please refer to the base configuration file [[./code/config/base.yaml]](./code/config/base.yaml) for the basic system setting of overall modules.\n\nThen, the training of NExT-GPT starts with this script:\n```angular2html\ncd ./code\nbash scripts/train.sh\n```\nSpecifying the command:\n```angular2html\ndeepspeed --include localhost:0 --master_addr 127.0.0.1 --master_port 28459 train.py \\\n    --model nextgpt \\\n    --stage 1\\\n    --save_path  ../ckpt/delta_ckpt/nextgpt/7b_tiva_v0/\\\n    --log_path ../ckpt/delta_ckpt/nextgpt/7b_tiva_v0/log/\n```\nwhere the key arguments are:\n- `--include`: `localhost:0` indicating the GPT cuda number `0` of deepspeed.\n- `--stage`: training stage.\n- `--save_path`: the directory which saves the trained delta weights. This directory will be automatically created.\n- `--log_path`: the directory which saves the log file.\n\n\n\n\n\n\nThe whole NExT-GPT training involves 3 steps:\n\n- **Step-1**: Encoding-side LLM-centric Multimodal Alignment. This stage trains the ***input projection layer*** while freezing the ImageBind, LLM, output projection layer.\n  \n  Just run the above `train.sh` script by setting: `--stage 1`\n  \n  Also refer to the running config file [[./code/config/stage_1.yaml]](./code/config/stage_1.yaml) and deepspeed config file [[./code/dsconfig/stage_1.yaml]](./code/dsconfig/stage_1.yaml) for more step-wise configurations.\n\n  Note that the dataset used for training in this step is included `dataset_name_list` and the dataset name must precisely match the definition in [[./code/dataset/catalog.py]](./code/dataset/catalog.py)  \n\n\n\n- **Step-2**: Decoding-side Instruction-following Alignment. This stage trains the ***output projection layers*** while freezing the ImageBind, LLM, input projection layers.\n\n  Just run the above `train.sh` script by setting: `--stage 2`\n\n  Also refer to the running config file [[./code/config/stage_2.yaml]](./code/config/stage_2.yaml) and deepspeed config file [[./code/dsconfig/stage_2.yaml]](./code/dsconfig/stage_2.yaml) for more step-wise configurations.\n\n\n\n\n\n- **Step-3**: Instruction Tuning. This stage instruction-tune 1) the ***LLM*** via LoRA, 2) ***input projection layer*** and 3) ***output projection layer*** on the instruction dataset.\n\n  Just run the above `train.sh` script by setting: `--stage 3`\n\n  Also refer to the running config file [[./code/config/stage_3.yaml]](./code/config/stage_3.yaml) and deepspeed config file [[./code/dsconfig/stage_3.yaml]](./code/dsconfig/stage_3.yaml) for more step-wise configurations.\n\n\n\n\n\u003cspan id='Run NExT-GPT System'/\u003e\n\n## 4. Running NExT-GPT System \u003ca href='#all_catelogue'\u003e[Back to Top]\u003c/a\u003e\n\n\n\u003cspan id='Prepare checkpoints'/\u003e\n\n\n#### 4.1. Preparing Checkpoints\n\nFirst, loading the pre-trained NExT-GPT system.\n- **Step-1**: load `Frozen parameters`. Please refer to \u003ca href='#Prepare Pre-trained Checkpoint'\u003e3.1 Preparing Pre-trained Checkpoint\u003c/a\u003e.\n\n- **Step-2**: load `Tunable parameters`. Please put the NExT-GPT system at [[./ckpt/delta_ckpt/nextgpt/7b_tiva_v0]](./ckpt/delta_ckpt/nextgpt/7b_tiva_v0). You may either 1) use the params trained yourselves, or 2) download our checkpoints from [Huggingface](https://huggingface.co/ChocoWu/nextgpt_7b_tiva_v0). \n\n\n\u003cspan id='Deploy Demo System'/\u003e\n\n\n#### 4.2. Deploying Gradio Demo\nUpon completion of the checkpoint loading, you can run the demo locally via:\n```angular2html\ncd ./code\nbash scripts/app.sh\n```\nSpecifying the key arguments as:\n- `--nextgpt_ckpt_path`: the path of pre-trained NExT-GPT params.\n\n---------\n\n\n## Contact\n\nFor any questions or feedback, feel free to contact [Shengqiong Wu](mailto:swu@u.nus.edu) and [Hao Fei](mailto:haofei37@nus.edu.sg).\n\n\n## Citation\n\nIf you find NextGPT useful in your research or applications, please kindly cite:\n```\n@articles{wu2023nextgpt,\n  title={NExT-GPT: Any-to-Any Multimodal LLM},\n  author={Shengqiong Wu and Hao Fei and Leigang Qu and Wei Ji and Tat-Seng Chua},\n  journal = {CoRR},\n  volume = {abs/2309.05519},\n  year={2023}\n}\n```\n\n\n\n\n\n## Acknowledgements\nYou may refer to related work that serves as foundations for our framework and code repository, \n[Vicuna](https://github.com/lm-sys/FastChat), \n[ImageBind](https://github.com/facebookresearch/ImageBind), \n[Stable Diffusion](https://huggingface.co/docs/diffusers/api/pipelines/stable_diffusion/text2img), \n[AudioLDM](https://github.com/haoheliu/AudioLDM), and\n[Zeroscope](https://huggingface.co/cerspense/zeroscope_v2_576w).\nWe also partially draw inspirations from \n[PandaGPT](https://github.com/yxuansu/PandaGPT), \n[VPGTrans](https://vpgtrans.github.io/), \n[GILL](https://github.com/kohjingyu/gill/), \n[CoDi](https://codi-gen.github.io/),\n[Video-LLaMA](https://github.com/DAMO-NLP-SG/Video-LLaMA),\nand [MiniGPT-4](https://github.com/Vision-CAIR/MiniGPT-4).\nThanks for their wonderful works.\n\n\n\n\n## License Notices\nThis repository is under [BSD 3-Clause License](LICENSE.txt).\nNExT-GPT is a research project intended for non-commercial use only. \nOne must NOT use the code of NExT-GPT for any illegal, harmful, violent, racist, or sexual purposes. \nOne is strictly prohibited from engaging in any activity that will potentially violate these guidelines.\nAny potential commercial use of this code should be approved by the authors.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FNExT-GPT%2FNExT-GPT","html_url":"https://awesome.ecosyste.ms/projects/github.com%2FNExT-GPT%2FNExT-GPT","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FNExT-GPT%2FNExT-GPT/lists"}