https://github.com/jy0205/LaVIT
LaVIT: Empower the Large Language Model to Understand and Generate Visual Content
https://github.com/jy0205/LaVIT
Last synced: 11 months ago
JSON representation
LaVIT: Empower the Large Language Model to Understand and Generate Visual Content
- Host: GitHub
- URL: https://github.com/jy0205/LaVIT
- Owner: jy0205
- License: other
- Created: 2023-09-09T02:21:27.000Z (over 2 years ago)
- Default Branch: main
- Last Pushed: 2024-10-06T15:53:07.000Z (over 1 year ago)
- Last Synced: 2024-11-25T15:52:30.967Z (over 1 year ago)
- Language: Jupyter Notebook
- Homepage:
- Size: 83.5 MB
- Stars: 541
- Watchers: 14
- Forks: 29
- Open Issues: 8
-
Metadata Files:
- Readme: README.md
- License: LICENSE
Awesome Lists containing this project
- StarryDivineSky - jy0205/LaVIT - 语言的统一建模。LaVIT的核心思想是利用视觉标记（Visual Tokens）作为视觉信息的桥梁，让语言模型能够像处理文本一样处理图像。该项目支持多种视觉任务，例如图像描述、视觉问答和图像生成。LaVIT的训练过程包括预训练和微调两个阶段，预训练阶段旨在学习视觉标记的表示，微调阶段则针对特定任务进行优化。项目提供了详细的代码和文档，方便用户进行实验和二次开发。LaVIT的主要优势在于其简单性和可扩展性，它能够轻松地集成到现有的语言模型中，并支持多种视觉模态。LaVIT为探索通用视觉-语言模型提供了一个有价值的框架。 (多模态大模型 / 资源传输下载)
README
# LaVIT: Empower the Large Language Model to Understand and Generate Visual Content
This is the official repository for the multi-modal large language models: **LaVIT** and **Video-LaVIT**. The LaVIT project aims to leverage the exceptional capability of LLM to deal with visual content. The proposed pre-training strategy supports visual understanding and generation with one unified framework.
* Unified Language-Vision Pretraining in LLM with Dynamic Discrete Visual Tokenization, ICLR 2024, [[`arXiv`](https://arxiv.org/abs/2309.04669)] [[`BibTeX`](#Citing)]
* Video-LaVIT: Unified Video-Language Pre-training with Decoupled Visual-Motional Tokenization, ICML 2024 Oral, [[`arXiv`](https://arxiv.org/abs/2402.03161)] [[`Project`](https://video-lavit.github.io)] [[`BibTeX`](#Citing)]
## News and Updates
* ```2024.06.01``` 👏👏👏 Video-LaVIT has been accepted by ICML 2024 as an Oral presentation!
* ```2024.04.21``` 🚀🚀🚀 We have released the pre-trained weight for **Video-LaVIT** on the HuggingFace and provide the inference code.
* ```2024.02.05``` 🌟🌟🌟 We have proposed the **Video-LaVIT**: an effective multimodal pre-training approach that empowers LLMs to comprehend and generate video content in a unified framework.
* ```2024.01.15``` 👏👏👏 LaVIT has been accepted by ICLR 2024!
* ```2023.10.17``` 🚀🚀🚀 We release the pre-trained weight for **LaVIT** on the HuggingFace and provide the inference code of using it for both multi-modal understanding and generation.
## Introduction
The **LaVIT** and **Video-LaVIT** are general-purpose multi-modal foundation models that inherit the successful learning paradigm of LLM: predicting the next visual/textual token in an auto-regressive manner. The core design of the LaVIT series works includes a **visual tokenizer** and a **detokenizer**. The visual tokenizer aims to translate the non-linguistic visual content (e.g., image, video) into a sequence of discrete tokens like a foreign language that LLM can read. The detokenizer recovers the generated discrete tokens from LLM to the continuous visual signals.
LaVIT Pipeline
Video-LaVIT Pipeline
After pre-training, LaVIT and Video-LaVIT can support
* Read image and video content, generate the captions, and answer the questions.
* Text-to-image, Text-to-Video and Image-to-Video generation.
* Generation via Multi-modal Prompt.
## Citation
Consider giving this repository a star and cite LaVIT in your publications if it helps your research.
```
@inproceedings{jin2024unified,
title={Unified Language-Vision Pretraining in LLM with Dynamic Discrete Visual Tokenization},
author={Jin, Yang and Xu, Kun and Xu, Kun and Chen, Liwei and Liao, Chao and Tan, Jianchao and Mu, Yadong and others},
booktitle={International Conference on Learning Representations},
year={2024}
}
@inproceedings{jin2024video,
title={Video-LaVIT: Unified Video-Language Pre-training with Decoupled Visual-Motional Tokenization},
author={Jin, Yang and Sun, Zhicheng and Xu, Kun and Chen, Liwei and Jiang, Hao and Huang, Quzhe and Song, Chengru and Liu, Yuliang and Zhang, Di and Song, Yang and Gai, Kun and Mu, Yadong},
booktitle={International Conference on Machine Learning},
pages={22185--22209},
year={2024}
}