Ecosyste.ms: Awesome
An open API service indexing awesome lists of open source software.
https://github.com/ToonCrafter/ToonCrafter
a research paper for generative cartoon interpolation
https://github.com/ToonCrafter/ToonCrafter
Last synced: about 2 months ago
JSON representation
a research paper for generative cartoon interpolation
- Host: GitHub
- URL: https://github.com/ToonCrafter/ToonCrafter
- Owner: ToonCrafter
- License: apache-2.0
- Created: 2024-05-28T19:17:40.000Z (4 months ago)
- Default Branch: main
- Last Pushed: 2024-06-01T08:05:45.000Z (4 months ago)
- Last Synced: 2024-07-06T09:03:06.346Z (2 months ago)
- Language: Python
- Homepage:
- Size: 67.1 MB
- Stars: 4,733
- Watchers: 52
- Forks: 401
- Open Issues: 34
-
Metadata Files:
- Readme: README.md
- License: LICENSE
Awesome Lists containing this project
- ai-game-devtools - ToonCrafter
- awesome-acg - ToonCrafter - Generative Cartoon Interpolation. [English] (Image Generation)
- awesome-llm-projects - ToonCrafter - trained image-to-video diffusion priors. (Projects / 🌄 Image)
- AiTreasureBox - ToonCrafter/ToonCrafter - 09-13_5119_1](https://img.shields.io/github/stars/ToonCrafter/ToonCrafter.svg)|a research paper for generative cartoon interpolation| (Repos)
README
## ___***ToonCrafter: Generative Cartoon Interpolation***___
## 🔆 Introduction
⚠️ Please check our [disclaimer](#disc) first.
🤗 ToonCrafter can interpolate two cartoon images by leveraging the pre-trained image-to-video diffusion priors. Please check our project page and paper for more information.
### 1.1 Showcases (512x320)
Input starting frame
Input ending frame
Generated video
### 1.2 Sparse sketch guidance
Input starting frame
Input ending frame
Input sketch guidance
Generated video
### 2. Applications
#### 2.1 Cartoon Sketch Interpolation (see project page for more details)
Input starting frame
Input ending frame
Generated video
#### 2.2 Reference-based Sketch Colorization
Input sketch
Input reference
Colorization results
## 📝 Changelog
- [ ] Add sketch control and colorization function.
- __[2024.05.29]__: 🔥🔥 Release code and model weights.
- __[2024.05.28]__: Launch the project page and update the arXiv preprint.
## 🧰 Models
|Model|Resolution|GPU Mem. & Inference Time (A100, ddim 50steps)|Checkpoint|
|:---------|:---------|:--------|:--------|
|ToonCrafter_512|320x512| TBD (`perframe_ae=True`)|[Hugging Face](https://huggingface.co/Doubiiu/ToonCrafter/blob/main/model.ckpt)|
Currently, our ToonCrafter can support generating videos of up to 16 frames with a resolution of 512x320. The inference time can be reduced by using fewer DDIM steps.
## ⚙️ Setup
### Install Environment via Anaconda (Recommended)
```bash
conda create -n tooncrafter python=3.8.5
conda activate tooncrafter
pip install -r requirements.txt
```
## 💫 Inference
### 1. Command line
Download pretrained ToonCrafter_512 and put the `model.ckpt` in `checkpoints/tooncrafter_512_interp_v1/model.ckpt`.
```bash
sh scripts/run.sh
```
### 2. Local Gradio demo
Download the pretrained model and put it in the corresponding directory according to the previous guidelines.
```bash
python gradio_app.py
```
## 📢 Disclaimer
Calm down. Our framework opens up the era of generative cartoon interpolation, but due to the variaity of generative video prior, the success rate is not guaranteed.
⚠️This is an open-source research exploration, instead of commercial products. It can't meet all your expectations.
This project strives to impact the domain of AI-driven video generation positively. Users are granted the freedom to create videos using this tool, but they are expected to comply with local laws and utilize it responsibly. The developers do not assume any responsibility for potential misuse by users.
****