{"id":13958420,"url":"https://github.com/williamyang1991/vtoonify","last_synced_at":"2025-05-15T00:08:42.171Z","repository":{"id":59743686,"uuid":"534480768","full_name":"williamyang1991/VToonify","owner":"williamyang1991","description":"[SIGGRAPH Asia 2022] VToonify: Controllable High-Resolution Portrait Video Style Transfer","archived":false,"fork":false,"pushed_at":"2023-10-25T00:46:55.000Z","size":21654,"stargazers_count":3576,"open_issues_count":28,"forks_count":448,"subscribers_count":61,"default_branch":"main","last_synced_at":"2025-04-13T21:33:58.524Z","etag":null,"topics":["face","siggraph-asia","style-transfer","stylegan2","toonify","video-style-transfer"],"latest_commit_sha":null,"homepage":"","language":"Jupyter Notebook","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"other","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/williamyang1991.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE.md","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2022-09-09T03:15:37.000Z","updated_at":"2025-04-11T14:31:44.000Z","dependencies_parsed_at":"2024-11-29T03:02:33.316Z","dependency_job_id":"1554ac22-0206-4650-b010-f2a1fc9fd724","html_url":"https://github.com/williamyang1991/VToonify","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/williamyang1991%2FVToonify","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/williamyang1991%2FVToonify/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/williamyang1991%2FVToonify/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/williamyang1991%2FVToonify/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/williamyang1991","download_url":"https://codeload.github.com/williamyang1991/VToonify/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":254249204,"owners_count":22039029,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["face","siggraph-asia","style-transfer","stylegan2","toonify","video-style-transfer"],"created_at":"2024-08-08T13:01:33.471Z","updated_at":"2025-05-15T00:08:37.154Z","avatar_url":"https://github.com/williamyang1991.png","language":"Jupyter Notebook","funding_links":[],"categories":["图像风格"],"sub_categories":["网络服务_其他"],"readme":"# VToonify - Official PyTorch Implementation\n\nhttps://user-images.githubusercontent.com/18130694/189483939-0fc4a358-fb34-43cc-811a-b22adb820d57.mp4\n\nThis repository provides the official PyTorch implementation for the following paper:\n\n**VToonify: Controllable High-Resolution Portrait Video Style Transfer**\u003cbr\u003e\n[Shuai Yang](https://williamyang1991.github.io/), [Liming Jiang](https://liming-jiang.com/), [Ziwei Liu](https://liuziwei7.github.io/) and [Chen Change Loy](https://www.mmlab-ntu.com/person/ccloy/)\u003cbr\u003e\nIn ACM TOG (Proceedings of SIGGRAPH Asia), 2022.\u003cbr\u003e\n[**Project Page**](https://www.mmlab-ntu.com/project/vtoonify/) | [**Paper**](https://arxiv.org/abs/2209.11224) | [**Supplementary Video**](https://youtu.be/0_OmVhDgYuY) | [**Input Data and Video Results**](https://drive.google.com/file/d/1A2gC2PW1ZmU6VWQRvMN98njqRxfLjqbk/view?usp=sharing) \u003cbr\u003e\n\n\u003ca href=\"http://colab.research.google.com/github/williamyang1991/VToonify/blob/master/notebooks/inference_playground.ipynb\"\u003e\u003cimg src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"google colab logo\"\u003e\u003c/a\u003e \n[![Hugging Face Spaces](https://img.shields.io/badge/%F0%9F%A4%97%20Hugging%20Face-Spaces-blue)](https://huggingface.co/spaces/PKUWilliamYang/VToonify)\n[![Deque](https://img.shields.io/badge/Deque-Notebook-blue)](https://deque.ai/experience/notebook/dbabc82a-ace4-11ed-84b5-0242ac110002)\n![visitors](https://visitor-badge.laobi.icu/badge?page_id=williamyang1991/VToonify)\n\u003c!--![visitors](https://visitor-badge.glitch.me/badge?page_id=williamyang1991/VToonify)--\u003e\n\n\n\u003e **Abstract:** *Generating high-quality artistic portrait videos is an important and desirable task in computer graphics and vision.\nAlthough a series of successful portrait image toonification models built upon the powerful StyleGAN have been proposed,\nthese image-oriented methods have obvious limitations when applied to videos, such as the fixed frame size, the requirement of face alignment, missing non-facial details and temporal inconsistency.\nIn this work, we investigate the challenging controllable high-resolution portrait video style transfer by introducing a novel **VToonify** framework.\nSpecifically, VToonify leverages the mid- and high-resolution layers of StyleGAN to render high-quality artistic portraits based on the multi-scale content features extracted by an encoder to better preserve the frame details. The resulting fully convolutional architecture accepts non-aligned faces in videos of variable size as input, contributing to complete face regions with natural motions in the output.\nOur framework is compatible with existing StyleGAN-based image toonification models to extend them to video toonification, and inherits appealing features of these models for flexible style control on color and intensity.\nThis work presents two instantiations of VToonify built upon Toonify and DualStyleGAN for collection-based and exemplar-based portrait video style transfer, respectively.\nExtensive experimental results demonstrate the effectiveness of our proposed VToonify framework over existing methods in generating high-quality and temporally-coherent artistic portrait videos with flexible style controls.*\n\n**Features**:\u003cbr\u003e \n**High-Resolution Video** (\u003e1024, support unaligned faces) | **Data-Friendly** (no real training data) | **Style Control**\n\n![overview](https://user-images.githubusercontent.com/18130694/189509940-91c5e1e2-83a8-491e-962e-64775e56d7f6.jpg)\n\n\n## Updates\n\n- [02/2023] Integrated to [Deque Notebook](https://deque.ai/experience/notebook/dbabc82a-ace4-11ed-84b5-0242ac110002).\n- [10/2022] Integrate [Gradio](https://gradio.app/) interface into [Colab notebook](http://colab.research.google.com/github/williamyang1991/VToonify/blob/master/notebooks/inference_playground.ipynb). Enjoy the web demo!\n- [10/2022] Integrated to 🤗 [Hugging Face](https://huggingface.co/spaces/PKUWilliamYang/VToonify). Enjoy the web demo!\n- [09/2022] Input videos and video results are released.\n- [09/2022] Paper is released.\n- [09/2022] Code is released.\n- [09/2022] This website is created.\n\n## Web Demo\n\nIntegrated into [Huggingface Spaces 🤗](https://huggingface.co/spaces) using [Gradio](https://github.com/gradio-app/gradio). Try out the Web Demo [![Hugging Face Spaces](https://img.shields.io/badge/%F0%9F%A4%97%20Hugging%20Face-Spaces-blue)](https://huggingface.co/spaces/PKUWilliamYang/VToonify)\n\n\n## Installation\n**Clone this repo:**\n```bash\ngit clone https://github.com/williamyang1991/VToonify.git\ncd VToonify\n```\n**Dependencies:**\n\nWe have tested on:\n- CUDA 10.1\n- PyTorch 1.7.0\n- Pillow 8.3.1; Matplotlib 3.3.4; opencv-python 4.5.3; Faiss 1.7.1; tqdm 4.61.2; Ninja 1.10.2\n\nAll dependencies for defining the environment are provided in `environment/vtoonify_env.yaml`.\nWe recommend running this repository using [Anaconda](https://docs.anaconda.com/anaconda/install/) (you may need to modify `vtoonify_env.yaml` to install PyTorch that matches your own CUDA version following [https://pytorch.org/](https://pytorch.org/)):\n```bash\nconda env create -f ./environment/vtoonify_env.yaml\n```\n\n☞ Install on Windows: https://github.com/williamyang1991/VToonify/issues/50#issuecomment-1443061101 and https://github.com/williamyang1991/VToonify/issues/38#issuecomment-1442146800\n\n☞ If you have a problem regarding the cpp extention (fused and upfirdn2d), or no GPU is available, you may refer to [CPU compatible version](./model/stylegan/op_cpu#readme).\n\n\u003cbr/\u003e\n\n## (1) Inference for Image/Video Toonification\n\n### Inference Notebook \n\u003ca href=\"http://colab.research.google.com/github/williamyang1991/VToonify/blob/master/notebooks/inference_playground.ipynb\"\u003e\u003cimg src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"google colab logo\"\u003e\u003c/a\u003e \n[![Deque](https://img.shields.io/badge/Deque-Notebook-blue)](https://deque.ai/experience/notebook/dbabc82a-ace4-11ed-84b5-0242ac110002)\n\nTo help users get started, we provide a Jupyter notebook found in `./notebooks/inference_playground.ipynb` that allows one to visualize the performance of VToonify.\nThe notebook will download the necessary pretrained models and run inference on the images found in `./data/`.\n\n### Pre-trained Models\n\nPre-trained models can be downloaded from [Google Drive](https://drive.google.com/drive/folders/1Nmbz9zBM78I1nRVokhHLuBKHleDmjDxv?usp=sharing), [Baidu Cloud](https://pan.baidu.com/s/1Io3PKNV1wD7ttxaVz-MPww?pwd=sigg) (access code: sigg) or [Hugging Face](https://huggingface.co/PKUWilliamYang/VToonify/tree/main/models):\n\n\u003ctable\u003e\n    \u003ctr\u003e\n        \u003cth\u003eBackbone\u003c/th\u003e\u003cth\u003eModel\u003c/th\u003e\u003cth\u003eDescription\u003c/th\u003e\n    \u003c/tr\u003e\n    \u003ctr\u003e\n        \u003ctd rowspan=\"6\"\u003eDualStyleGAN\u003c/td\u003e\u003ctd\u003e\u003ca href=\"https://drive.google.com/drive/folders/1DuZfXt6b_xhTAQSN0D8m7N1np0Web0Ky\"\u003ecartoon\u003c/a\u003e\u003c/td\u003e\u003ctd\u003epre-trained VToonify-D models and 317 cartoon style codes\u003c/td\u003e\n    \u003c/tr\u003e\n    \u003ctr\u003e\n        \u003ctd\u003e\u003ca href=\"https://drive.google.com/drive/folders/12TzTQqwBedsYX3kE_420mdTbWl9lwv4Y\"\u003ecaricature\u003c/a\u003e\u003c/td\u003e\u003ctd\u003epre-trained VToonify-D models and 199 caricature style codes\u003c/td\u003e\n    \u003c/tr\u003e\n    \u003ctr\u003e\n        \u003ctd\u003e\u003ca href=\"https://drive.google.com/drive/folders/1MpEqS26Q1ngTPeex_4MN9qOJxfXKH-k-\"\u003earcane\u003c/a\u003e\u003c/td\u003e\u003ctd\u003epre-trained VToonify-D models and 100 arcane style codes\u003c/td\u003e\n    \u003c/tr\u003e \n    \u003ctr\u003e\n        \u003ctd\u003e\u003ca href=\"https://drive.google.com/drive/folders/15mxb7DxTzEBrKtx5aJ_I5WGDjSWBmcUi\"\u003ecomic\u003c/a\u003e\u003c/td\u003e\u003ctd\u003epre-trained VToonify-D models and 101 comic style codes\u003c/td\u003e\n    \u003c/tr\u003e   \n    \u003ctr\u003e\n        \u003ctd\u003e\u003ca href=\"https://drive.google.com/drive/folders/1Hld7OeZqYBrg6r35IA_x4sNtt1abHUMU\"\u003epixar\u003c/a\u003e\u003c/td\u003e\u003ctd\u003epre-trained VToonify-D models and 122 pixar style codes\u003c/td\u003e\n    \u003c/tr\u003e \n    \u003ctr\u003e\n        \u003ctd\u003e\u003ca href=\"https://drive.google.com/drive/folders/1LQGNMDEHM70nOhm3-xY228YpJNlPnf_s\"\u003eillustration\u003c/a\u003e\u003c/td\u003e\u003ctd\u003epre-trained VToonify-D models and 156 illustration style codes\u003c/td\u003e\n    \u003c/tr\u003e     \n    \u003ctr\u003e\n        \u003ctd rowspan=\"5\"\u003eToonify\u003c/td\u003e\u003ctd\u003e\u003ca href=\"https://drive.google.com/drive/folders/1FFtTVgiDKZ_InnwUJLDuA1wfghZp41nX\"\u003ecartoon\u003c/a\u003e\u003c/td\u003e\u003ctd\u003epre-trained VToonify-T model\u003c/td\u003e\n    \u003c/tr\u003e\n    \u003ctr\u003e\n        \u003ctd\u003e\u003ca href=\"https://drive.google.com/drive/folders/1ReRxttV-macgV3epC61qg4TQ3FGAhGqG\"\u003ecaricature\u003c/a\u003e\u003c/td\u003e\u003ctd\u003epre-trained VToonify-T model\u003c/td\u003e\n    \u003c/tr\u003e\n    \u003ctr\u003e\n        \u003ctd\u003e\u003ca href=\"https://drive.google.com/drive/folders/1OXU95BOCCT0f6pGbwQ4yQ1EHb2LPd2yb\"\u003earcane\u003c/td\u003e\u003c/a\u003e\u003ctd\u003epre-trained VToonify-T model\u003c/td\u003e\n    \u003c/tr\u003e \n    \u003ctr\u003e\n        \u003ctd\u003e\u003ca href=\"https://drive.google.com/drive/folders/1KvawsOXzKgwDM3Z27sagO_KGE_Kc5GZS\"\u003ecomic\u003c/td\u003e\u003c/a\u003e\u003ctd\u003epre-trained VToonify-T model\u003c/td\u003e\n    \u003c/tr\u003e   \n    \u003ctr\u003e\n        \u003ctd\u003e\u003ca href=\"https://drive.google.com/drive/folders/19N4ddcTXhXbTEayTbrFc533EktbhOXMz\"\u003epixar\u003c/td\u003e\u003c/a\u003e\u003ctd\u003epre-trained VToonify-T model\u003c/td\u003e\n    \u003c/tr\u003e   \n    \u003ctr\u003e\n        \u003cth colspan=\"2\"\u003eSupporting model\u003c/th\u003e\u003cth\u003e \u003c/th\u003e\n    \u003c/tr\u003e\n    \u003ctr\u003e\n        \u003ctd colspan=\"2\"\u003e\u003ca href=\"https://drive.google.com/file/d/1NgI4mPkboYvYw3MWcdUaQhkr0OWgs9ej/view?usp=sharing\"\u003eencoder.pt\u003c/a\u003e\u003c/td\u003e\u003ctd\u003ePixel2style2pixel encoder to map real faces into Z+ space of StyleGAN\u003c/td\u003e\n    \u003c/tr\u003e  \n    \u003ctr\u003e\n        \u003ctd colspan=\"2\"\u003e\u003ca href=\"https://drive.google.com/file/d/1jY0mTjVB8njDh6e0LP_2UxuRK3MnjoIR/view\"\u003efaceparsing.pth\u003c/a\u003e\u003c/td\u003e\u003ctd\u003eBiSeNet for face parsing from \u003ca href=\"https://github.com/zllrunning/face-parsing.PyTorch\"\u003eface-parsing.PyTorch\u003c/a\u003e\u003c/td\u003e\n    \u003c/tr\u003e      \n\u003c/table\u003e\n\nThe downloaded models are suggested to be arranged in [this folder structure](./checkpoint/).\n\nThe VToonify-D models are named with suffixes to indicate the settings, where\n- `_sXXX`: supports only one fixed style with `XXX` the index of this style.\n    - `_s` without `XXX` means the model supports examplar-based style transfer\n- `_dXXX`: supports only a fixed style degree of `XXX`.\n    - `_d` without `XXX` means the model supports style degrees ranging from 0 to 1\n- `_c`: supports color transfer.\n\n### Style Transfer with VToonify-D\n\n**✔ A quick start [HERE](./output#readme)**\n\nTransfer a default cartoon style onto a default face image `./data/077436.jpg`:\n```python\npython style_transfer.py --scale_image\n```\nThe results are saved in the folder `./output/`, where `077436_input.jpg` is the rescaled input image to fit VToonify (this image can serve as the input without `--scale_image`) and `077436_vtoonify_d.jpg` is the result.\n\n![077436_overview](https://user-images.githubusercontent.com/18130694/189530937-eb468f96-ac02-4f33-8621-03cb93d17e73.jpg)\n\nSpecify the content image and the model, control the style with the following options:\n- `--content`: path to the target face image or video\n- `--style_id`: the index of the style image (find the mapping between index and the style image [here](https://github.com/williamyang1991/DualStyleGAN/tree/main/doc_images)). \n- `--style_degree` (default: 0.5): adjust the degree of style.\n- `--color_transfer`(default: False): perform color transfer if loading a VToonify-Dsdc model.\n- `--ckpt`: path of the VToonify-D model. By default, a VToonify-Dsd trained on cartoon style is loaded.\n- `--exstyle_path`: path of the extrinsic style code. By default, codes in the same directory as `--ckpt` are loaded.\n- `--scale_image`: rescale the input image/video to fit VToonify (highly recommend).\n- `--padding` (default: 200, 200, 200, 200): left, right, top, bottom paddings to the eye center.\n\nHere is an example of arcane style transfer:\n```python\npython style_transfer.py --content ./data/038648.jpg \\\n       --scale_image --style_id 77 --style_degree 0.5 \\\n       --ckpt ./checkpoint/vtoonify_d_arcane/vtoonify_s_d.pt \\\n       --padding 600 600 600 600     # use large padding to avoid cropping the image\n```\n![arcane](https://user-images.githubusercontent.com/18130694/189533139-94c3d086-7fe9-49f9-b31f-dbd2a4798e9f.jpg)\n\n\nSpecify `--video` to perform video toonification:\n```python\npython style_transfer.py --scale_image --content ./data/YOUR_VIDEO.mp4 --video\n```\nThe above style control options (`--style_id`, `--style_degree`, `--color_transfer`) also work for videos.\n\n\n\n### Style Transfer with VToonify-T\n\nSpecify `--backbone` as ''toonify'' to load and use a VToonify-T model.\n```python\npython style_transfer.py --content ./data/038648.jpg \\\n       --scale_image --backbone toonify \\\n       --ckpt ./checkpoint/vtoonify_t_arcane/vtoonify.pt \\\n       --padding 600 600 600 600     # use large padding to avoid cropping the image\n```\n![arcane2](https://user-images.githubusercontent.com/18130694/189540365-d04ffb2a-d72f-4ada-a2a8-89b8ac9ea441.jpg)\n\nIn VToonify-T, `--style_id`, `--style_degree`, `--color_transfer`, `--exstyle_path` are not used.\n\nAs with VToonify-D, specify `--video` to perform video toonification.\n\n\u003cbr/\u003e\n\n## (2) Training VToonify\n\nDownload the supporting models to the `./checkpoint/` folder and arrange them in [this folder structure](./checkpoint/):\n\n| Model | Description |\n| :--- | :--- |\n| [stylegan2-ffhq-config-f.pt](https://drive.google.com/file/d/1EM87UquaoQmk17Q8d5kYIAHqu0dkYqdT/view) | StyleGAN model trained on FFHQ taken from [rosinality](https://github.com/rosinality/stylegan2-pytorch) |\n| [encoder.pt](https://drive.google.com/file/d/1NgI4mPkboYvYw3MWcdUaQhkr0OWgs9ej/view) | Pixel2style2pixel encoder that embeds FFHQ images into StyleGAN2 Z+ latent code |\n| [faceparsing.pth](https://drive.google.com/file/d/1jY0mTjVB8njDh6e0LP_2UxuRK3MnjoIR/view) | BiSeNet for face parsing from [face-parsing.PyTorch](https://github.com/zllrunning/face-parsing.PyTorch) |\n| [directions.npy](https://drive.google.com/file/d/1HbjmOIOfxqTAVScZOI2m7_tPgMPnc0uM/view) | Editing vectors taken from [LowRankGAN](https://github.com/zhujiapeng/LowRankGAN) for editing face attributes |\n| [Toonify](https://drive.google.com/drive/folders/1GZQ6Gs5AzJq9lUL-ldIQexi0JYPKNy8b) \\| [DualStyleGAN](https://drive.google.com/drive/folders/1GZQ6Gs5AzJq9lUL-ldIQexi0JYPKNy8b) | pre-trained stylegan-based toonification models|\n\nTo customize your own style, you may need to train a new Toonify/DualStyleGAN model following [here](https://github.com/williamyang1991/DualStyleGAN#3-training-dualstylegan).\n\n### Train VToonify-D\n\nGiven the supporting models arranged in the [default folder structure](./checkpoint/), we can simply pre-train the encoder and train the whole VToonify-D by running\n```python\n# for pre-training the encoder\npython -m torch.distributed.launch --nproc_per_node=N_GPU --master_port=PORT train_vtoonify_d.py \\\n       --iter ITERATIONS --stylegan_path DUALSTYLEGAN_PATH --exstyle_path EXSTYLE_CODE_PATH \\\n       --batch BATCH_SIZE --name SAVE_NAME --pretrain\n# for training VToonify-D given the pre-trained encoder\npython -m torch.distributed.launch --nproc_per_node=N_GPU --master_port=PORT train_vtoonify_d.py \\\n       --iter ITERATIONS --stylegan_path DUALSTYLEGAN_PATH --exstyle_path EXSTYLE_CODE_PATH \\\n       --batch BATCH_SIZE --name SAVE_NAME                  # + ADDITIONAL STYLE CONTROL OPTIONS\n```\nThe models and the intermediate results are saved in `./checkpoint/SAVE_NAME/` and `./log/SAVE_NAME/`, respectively. \n\nVToonify-D provides the following STYLE CONTROL OPTIONS:\n- `--fix_degree`: if specified, model is trained with a fixed style degree (no degree adjustment)\n- `--fix_style`: if specified, model is trained with a fixed style image (no examplar-based style transfer)\n- `--fix_color`: if specified, model is trained with color preservation (no color transfer)\n- `--style_id`: the index of the style image (find the mapping between index and the style image [here](https://github.com/williamyang1991/DualStyleGAN/tree/main/doc_images)). \n- `--style_degree` (default: 0.5): the degree of style.\n\nHere is an example to reproduce the VToonify-Dsd on Cartoon style and the VToonify-D specialized for a mild toonification on the 26th cartoon style:\n```python\npython -m torch.distributed.launch --nproc_per_node=8 --master_port=8765 train_vtoonify_d.py \\\n       --iter 30000 --stylegan_path ./checkpoint/cartoon/generator.pt --exstyle_path ./checkpoint/cartoon/refined_exstyle_code.npy \\\n       --batch 1 --name vtoonify_d_cartoon --pretrain      \npython -m torch.distributed.launch --nproc_per_node=8 --master_port=8765 train_vtoonify_d.py \\\n       --iter 2000 --stylegan_path ./checkpoint/cartoon/generator.pt --exstyle_path ./checkpoint/cartoon/refined_exstyle_code.npy \\\n       --batch 4 --name vtoonify_d_cartoon --fix_color \npython -m torch.distributed.launch --nproc_per_node=8 --master_port=8765 train_vtoonify_d.py \\\n       --iter 2000 --stylegan_path ./checkpoint/cartoon/generator.pt --exstyle_path ./checkpoint/cartoon/refined_exstyle_code.npy \\\n       --batch 4 --name vtoonify_d_cartoon --fix_color --fix_degree --style_degree 0.5 --fix_style --style_id 26\n```\nNote that the pre-trained encoder is shared by different STYLE CONTROL OPTIONS. VToonify-D only needs to pre-train the encoder once for each DualStyleGAN model.\nEight GPUs are not necessary, one can train the model with a single GPU with larger `--iter`.\n\n**Tips**: [how to find an ideal model] we can first train a versatile model VToonify-Dsd, \nand navigate around different styles and degrees. After finding the ideal setting, we can then train the model specialized in that setting for high-quality stylization.\n\n### Train VToonify-T\n\nThe training of VToonify-T is similar to VToonify-D,\n```python\n# for pre-training the encoder\npython -m torch.distributed.launch --nproc_per_node=N_GPU --master_port=PORT train_vtoonify_t.py \\\n       --iter ITERATIONS --finetunegan_path FINETUNED_MODEL_PATH \\\n       --batch BATCH_SIZE --name SAVE_NAME --pretrain       # + ADDITIONAL STYLE CONTROL OPTION\n# for training VToonify-T given the pre-trained encoder\npython -m torch.distributed.launch --nproc_per_node=N_GPU --master_port=PORT train_vtoonify_t.py \\\n       --iter ITERATIONS --finetunegan_path FINETUNED_MODEL_PATH \\\n       --batch BATCH_SIZE --name SAVE_NAME                  # + ADDITIONAL STYLE CONTROL OPTION\n```\nVToonify-T only has one STYLE CONTROL OPTION:\n - `--weight` (default: 1,1,1,1,1,1,1,1,1,0,0,0,0,0,0,0,0,0): 18 numbers indicate how the 18 layers of the ffhq stylegan model and the finetuned model are blended to obtain the final Toonify model. [Here](https://github.com/williamyang1991/VToonify/blob/edfd68e96eb0c0ab4c31628feef1b667e890a2cd/train_vtoonify_t.py#L30) is the `--weight` we use in the paper for different styles. Please refer to [toonify](https://github.com/justinpinkney/toonify) for the details.\n\nHere is an example to reproduce the VToonify-T model on Arcane style:\n```python\npython -m torch.distributed.launch --nproc_per_node=8 --master_port=8765 train_vtoonify_t.py \\\n       --iter 30000 --finetunegan_path ./checkpoint/arcane/finetune-000600.pt \\\n       --batch 1 --name vtoonify_t_arcane --pretrain --weight 0.5 0.5 0.5 0.5 0.5 0.5 0.5 1 1 1 1 1 1 1 1 1 1 1\npython -m torch.distributed.launch --nproc_per_node=8 --master_port=8765 train_vtoonify_t.py \\\n       --iter 2000 --finetunegan_path ./checkpoint/arcane/finetune-000600.pt \\\n       --batch 4 --name vtoonify_t_arcane --weight 0.5 0.5 0.5 0.5 0.5 0.5 0.5 1 1 1 1 1 1 1 1 1 1 1\n```\n\n\n\u003cbr/\u003e\n\n## (3) Results\n\nOur framework is compatible with existing StyleGAN-based image toonification models to extend them to video toonification, and inherits their appealing features for flexible style control. With DualStyleGAN as the backbone, our VToonify is able to transfer the style of various reference images and adjust the style degree in one model.\n\nhttps://user-images.githubusercontent.com/18130694/189510094-4378caca-e8d9-48e1-9e5d-c8ec038e4bc5.mp4\n\nHere are the color interpolated results of VToonify-D and VToonify-Dc on Arcane, Pixar and Comic styles.\n\nhttps://user-images.githubusercontent.com/18130694/189510233-b4e3b4f7-5a37-4e0c-9821-a8049ce5f781.mp4\n\n## Citation\n\nIf you find this work useful for your research, please consider citing our paper:\n\n```bibtex\n@article{yang2022Vtoonify,\n  title={VToonify: Controllable High-Resolution Portrait Video Style Transfer},\n  author={Yang, Shuai and Jiang, Liming and Liu, Ziwei and Loy, Chen Change},\n  journal={ACM Transactions on Graphics (TOG)},\n  volume={41},\n  number={6},\n  articleno={203},\n  pages={1--15},\n  year={2022},\n  publisher={ACM New York, NY, USA},\n  doi={10.1145/3550454.3555437},\n}\n```\n\n## Acknowledgments\n\nThe code is mainly developed based on [stylegan2-pytorch](https://github.com/rosinality/stylegan2-pytorch), [pixel2style2pixel](https://github.com/eladrich/pixel2style2pixel) and [DualStyleGAN](https://github.com/williamyang1991/DualStyleGAN).\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fwilliamyang1991%2Fvtoonify","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fwilliamyang1991%2Fvtoonify","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fwilliamyang1991%2Fvtoonify/lists"}