{"id":26229982,"url":"https://github.com/mingtaoguo/relightable-portrait-animation","last_synced_at":"2025-06-13T18:32:52.567Z","repository":{"id":279958245,"uuid":"940556063","full_name":"MingtaoGuo/Relightable-Portrait-Animation","owner":"MingtaoGuo","description":"[CVPR 2025] High-Fidelity Relightable Monocular Portrait Animation with Lighting-Controllable Video Diffusion Model","archived":false,"fork":false,"pushed_at":"2025-06-04T03:18:59.000Z","size":11161,"stargazers_count":41,"open_issues_count":2,"forks_count":3,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-06-04T10:19:39.012Z","etag":null,"topics":["portrait-animation","relighting","video-diffusion-model"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/MingtaoGuo.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2025-02-28T11:38:41.000Z","updated_at":"2025-06-04T03:19:01.000Z","dependencies_parsed_at":"2025-03-24T08:37:49.186Z","dependency_job_id":null,"html_url":"https://github.com/MingtaoGuo/Relightable-Portrait-Animation","commit_stats":null,"previous_names":["mingtaoguo/relightable-portrait-animation"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/MingtaoGuo/Relightable-Portrait-Animation","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/MingtaoGuo%2FRelightable-Portrait-Animation","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/MingtaoGuo%2FRelightable-Portrait-Animation/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/MingtaoGuo%2FRelightable-Portrait-Animation/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/MingtaoGuo%2FRelightable-Portrait-Animation/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/MingtaoGuo","download_url":"https://codeload.github.com/MingtaoGuo/Relightable-Portrait-Animation/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/MingtaoGuo%2FRelightable-Portrait-Animation/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":259698846,"owners_count":22898142,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["portrait-animation","relighting","video-diffusion-model"],"created_at":"2025-03-12T22:18:52.776Z","updated_at":"2025-06-13T18:32:52.557Z","avatar_url":"https://github.com/MingtaoGuo.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"\u003cdiv align=\"center\"\u003e\n\u003ch2\u003eHigh-Fidelity Relightable Monocular Portrait Animation with Lighting-Controllable Video Diffusion Model\u003c/h2\u003e\n\n[Mingtao Guo]()\u003csup\u003e1\u003c/sup\u003e\u0026nbsp;\n[Guanyu Xing]()\u003csup\u003e2\u003c/sup\u003e\u0026nbsp;\n[Yanli Liu]()\u003csup\u003e1,3\u003c/sup\u003e\u0026nbsp;\n\n\n\u003csup\u003e1\u003c/sup\u003e National Key Laboratory of Fundamental Science on Synthetic Vision,\nSichuan University, Chengdu, China \n\n\u003csup\u003e2\u003c/sup\u003e School of Cyber Science and Engineering, \nSichuan University, Chengdu, China \n\n\u003csup\u003e3\u003c/sup\u003e College of Computer Science, Sichuan University, Chengdu, China \n\n\u003ch3 style=\"color:#b22222;\"\u003e To Appear at CVPR 2025 \u003c/h3\u003e\n\n\u003ch4\u003e\n\u003ca href=\"https://arxiv.org/abs/2502.19894\"\u003e📄 arXiv Paper\u003c/a\u003e \u0026nbsp; \n\u003ca href=\"https://mingtaoguo.github.io/\"\u003e🌐 Project Page\u003c/a\u003e \u0026nbsp; \n\u003ca href=\"\"\u003e📺 Video\u003c/a\u003e\n\u003c/h4\u003e\n\n\u003c/div\u003e\n\n\u003cdiv align=\"center\"\u003e\n\u003cimg src=\"assets/intro.png?raw=true\" width=\"100%\"\u003e\n\u003c/div\u003e\n\u003c/div\u003e\n\n## :fire: News\n* **[2025.03.02]** Our pre-trained model is out on [HuggingFace](https://huggingface.co/MartinGuo/Relightable-Portrait-Animation)!\n* **[2025.02.27]** ⭐ Exciting News! Relightable-Portrait-Animation got accepted by CVPR 2025!\n\n## :bookmark_tabs: Todos\nWe are going to make all the following contents available:\n- [x] Model inference code\n- [x] Model checkpoint\n- [x] Training code\n- [x] Data processing code\n\n## Installation\n\n1. Clone this repo locally:\n\n```bash\ngit clone https://github.com/MingtaoGuo/Relightable-Portrait-Animation\ncd Relightable-Portrait-Animation\n```\n2. Install the dependencies:\n\n```bash\nconda create -n relipa python=3.8\nconda activate relipa\n```\n3. Install packages for inference:\n\n```bash\npip install torch==2.2.2 torchvision==0.17.2 torchaudio==2.2.2 --index-url https://download.pytorch.org/whl/cu121\npip install --extra-index-url https://miropsota.github.io/torch_packages_builder pytorch3d==0.7.7+pt2.2.2cu121\n\npip install -r requirements.txt\n```\n## Download weights\n```shell\nmkdir pretrained_weights\nmkdir pretrained_weights/relipa\ngit-lfs install\n\ngit clone https://huggingface.co/MartinGuo/Relightable-Portrait-Animation\nmv Relightable-Portrait-Animation/ref_embedder.pth pretrained_weights/relipa\nmv Relightable-Portrait-Animation/light_embedder.pth pretrained_weights/relipa\nmv Relightable-Portrait-Animation/head_embedder.pth pretrained_weights/relipa\nmv Relightable-Portrait-Animation/unet.pth pretrained_weights/relipa\n\nmv Relightable-Portrait-Animation/data src/decalib\nmv Relightable-Portrait-Animation/u2net_human_seg.pth src/facematting\n\ngit clone https://huggingface.co/stabilityai/stable-video-diffusion-img2vid-xt\nmv stable-video-diffusion-img2vid-xt pretrained_weights\n\ngit clone https://huggingface.co/stabilityai/sd-vae-ft-mse\nmv sd-vae-ft-mse pretrained_weights/stable-video-diffusion-img2vid-xt\n```\nThe weights will be put in the `./pretrained_weights` directory. Heads up! The whole downloading process could take quite a long time.\nFinally, these weights should be orgnized as follows:\n\n```text\n./pretrained_weights/\n|-- relipa\n|   |-- unet.pth\n|   |-- ref_embedder.pth\n|   |-- light_embedder.pth\n|   |-- head_embedder.pth\n|-- stable-video-diffusion-img2vid-xt\n    |-- sd-vae-ft-mse\n    |   |-- config.json\n    |   |-- diffusion_pytorch_model.bin\n    |-- feature_extractor\n    |   |-- preprocessor_config.json\n    |-- scheduler\n    |   |-- scheduler_config.json\n    |-- model_index.json\n    |-- unet\n    |   |-- config.json\n    |   |-- diffusion_pytorch_model.safetensors\n    |   |-- diffusion_pytorch_model.fp16.safetensors\n    |-- image_encoder\n    |   |-- config.json\n    |   |-- model.safetensors\n    |   |-- model.fp16.safetensors\n```\n# 🚀 Training and Inference \n\n## Inference of the Relightable Portrait Animation\n\nHere's the command to run preprocess scripts: Use DECA to extract the pose from the driving video and the mesh from the reference portrait, then render shading hints by combining the driving video's pose, the reference portrait's mesh, and the target lighting.\n\n```shell\npython preprocess.py --video_path resources/WDA_DebbieDingell1_000.mp4 --source_path resources/reference.png --light_path resources/target_lighting1.png --save_path resources/shading.mp4 --motion_align relative\n```\n\nAfter running ```preprocess.py``` you'll get the results: \n\n1. Reference, 2. Mask, 3. Driving image, 4. Landmark, 5. Shading hints \n![](https://github.com/MingtaoGuo/Relightable-Portrait-Animation/blob/main/assets/shading.png)\n\nHere's the command to run inference scripts: Guide our model with the shading hints obtained from preprocessing to generate results where the pose is consistent with that of the driving video, the identity is consistent with the reference image, and the lighting is consistent with the target lighting. \n\n```shell\npython inference.py --pretrained_model_name_or_path pretrained_weights/stable-video-diffusion-img2vid-xt --checkpoint_path pretrained_weights/relipa/ --video_path resources/shading.mp4 --save_path result.mp4 --guidance 4.5 --inference_steps 25 --driving_mode relighting\n```\n\nAfter running ```inference.py``` you'll get the results: \n\n1. Reference, 2. Shading hints, 3. Relighting result, 4. Driving image\n![](https://github.com/MingtaoGuo/Relightable-Portrait-Animation/blob/main/assets/relighting.png)\n## Training of the Relightable Portrait Animation \n```shell\npython train.py --pretrained_model_name_or_path pretrained_weights/stable-video-diffusion-img2vid-xt \\\n                --height 512 --width 512  --num_frames 16 --validation_steps 100 --max_train_steps 30000 \\\n                --gradient_accumulation_steps 8 --gradient_checkpointing True --learning_rate 1e-5 --use_8bit_adam True \\\n                --sample_rate 4 --num_workers 2 --checkpointing_steps 1000 --checkpoints_total_limit 2 \\\n                --data_meta_path TalkingHeadVideo/VFHQ/VFHQ-data-consistent.json\n```\n## Talking Head Video Dataset\n|VFHQ-video|VFHQ-kpmap|VFHQ-mesh|VFHQ-mask|\n|-|-|-|-|\n|![](https://github.com/MingtaoGuo/Relightable-Portrait-Animation/blob/main/assets/video.png)|![](https://github.com/MingtaoGuo/Relightable-Portrait-Animation/blob/main/assets/kpmap.png)|![](https://github.com/MingtaoGuo/Relightable-Portrait-Animation/blob/main/assets/mesh.png)|![](https://github.com/MingtaoGuo/Relightable-Portrait-Animation/blob/main/assets/maskdd.png)|\n\nMaking a training dataset for VFHQ\n```text\n    |-- VFHQ\n        |-- Clip+zZEv-ATOpoY+P0+C2+F3168-3532_10369.mp4\n        ...\n```\n```shell\npython make_dataset.py --video_path ./VFHQ --save_path ./TalkingHeadVideo --dataset VFHQ\n```\nMaking a training dataset for CelebV-HQ\n```text\n    |-- CelebV-HQ\n        |-- __lRwnjxeCg_1.mp4\n        ...\n```\n```shell\npython make_dataset.py --video_path ./CelebV-HQ --save_path ./TalkingHeadVideo --dataset CelebV-HQ\n```\nFinal dataset format\n```text\n./TalkingHeadVideo/\n    |-- VFHQ\n        |-- VFHQ-mask\n            |-- Clip+zZEv-ATOpoY+P0+C2+F3168-3532_10369.mp4\n             ...\n        |-- VFHQ-kpmap\n        |-- VFHQ-video\n        |-- VFHQ-mesh\n        VFHQ-data-consistent.json\n    |-- CelebV-HQ\n        |-- CelebV-HQ-mask\n            |-- __lRwnjxeCg_1.mp4\n             ...\n        |-- CelebV-HQ-kpmap\n        |-- CelebV-HQ-video\n        |-- CelebV-HQ-mesh\n        CelebV-HQ-data-consistent.json\n```\n# Acknowledgements\nWe first thank to the contributors to the [StableVideoDiffusion](https://github.com/Stability-AI/generative-models), [SVD_Xtend](https://github.com/pixeli99/SVD_Xtend), [Moore-AnimateAnyone](https://github.com/MooreThreads/Moore-AnimateAnyone), [Echomimic](https://github.com/antgroup/echomimic) and [MimicMotion](https://github.com/Tencent/MimicMotion) repositories, for their open research and exploration. Furthermore, our repo incorporates some codes from [DECA](https://github.com/yfeng95/DECA), [MediaPipe](https://github.com/google-ai-edge/mediapipe) and [U2Net](https://github.com/xuebinqin/U-2-Net), and we extend our thanks to them as well.\n## Citation\nIf you use this model in your research, please consider citing:\n\n```bibtex\n@InProceedings{Guo_2025_CVPR,\n    author    = {Guo, Mingtao and Xing, Guanyu and Liu, Yanli},\n    title     = {High-Fidelity Relightable Monocular Portrait Animation with Lighting-Controllable Video Diffusion Model},\n    booktitle = {Proceedings of the Computer Vision and Pattern Recognition Conference (CVPR)},\n    month     = {June},\n    year      = {2025},\n    pages     = {228-238}\n}\n```\n## License\n\nThis project is licensed under the MIT License. See the [LICENSE](LICENSE) file for details.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmingtaoguo%2Frelightable-portrait-animation","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fmingtaoguo%2Frelightable-portrait-animation","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmingtaoguo%2Frelightable-portrait-animation/lists"}