{"id":13768330,"url":"https://github.com/harlanhong/CVPR2022-DaGAN","last_synced_at":"2025-05-10T23:31:15.663Z","repository":{"id":37373124,"uuid":"467358744","full_name":"harlanhong/CVPR2022-DaGAN","owner":"harlanhong","description":"Official code for CVPR2022 paper: Depth-Aware Generative Adversarial Network for Talking Head Video Generation","archived":false,"fork":false,"pushed_at":"2023-12-04T12:14:44.000Z","size":6828,"stargazers_count":988,"open_issues_count":33,"forks_count":128,"subscribers_count":24,"default_branch":"master","last_synced_at":"2025-04-12T14:56:02.402Z","etag":null,"topics":["deepfake","depth","face","face-reenactment","gan","image-animation","motion-transfer","talking-head"],"latest_commit_sha":null,"homepage":"https://harlanhong.github.io/publications/dagan.html","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"other","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/harlanhong.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE.md","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null}},"created_at":"2022-03-08T04:08:06.000Z","updated_at":"2025-04-04T06:32:39.000Z","dependencies_parsed_at":"2024-01-06T20:18:10.934Z","dependency_job_id":"d8e96416-c4ef-4f4c-84c0-1ce9f7d8e9f6","html_url":"https://github.com/harlanhong/CVPR2022-DaGAN","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/harlanhong%2FCVPR2022-DaGAN","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/harlanhong%2FCVPR2022-DaGAN/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/harlanhong%2FCVPR2022-DaGAN/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/harlanhong%2FCVPR2022-DaGAN/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/harlanhong","download_url":"https://codeload.github.com/harlanhong/CVPR2022-DaGAN/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":253497296,"owners_count":21917683,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["deepfake","depth","face","face-reenactment","gan","image-animation","motion-transfer","talking-head"],"created_at":"2024-08-03T16:01:19.785Z","updated_at":"2025-05-10T23:31:10.641Z","avatar_url":"https://github.com/harlanhong.png","language":"Python","funding_links":[],"categories":["GitHub projects"],"sub_categories":[],"readme":"\n## :book: Depth-Aware Generative Adversarial Network for Talking Head Video Generation (CVPR 2022)\n\u003cp align=\"center\"\u003e\n  \u003csmall\u003e:fire: If DaGAN is helpful in your photos/projects, please help to :star: it or recommend it to your friends. Thanks:fire:\u003c/small\u003e\n\u003c/p\u003e\n\u003cp align=\"center\"\u003e\n\u003csmall\u003e:fire: Seeking for the collaboration and internship opportunities. :fire:\u003c/small\u003e\n\u003c/p\u003e\n\n\u003e [[Paper](https://arxiv.org/abs/2203.06605)] \u0026emsp; [[Project Page](https://harlanhong.github.io/publications/dagan.html)] \u0026emsp; [[Demo](https://huggingface.co/spaces/HarlanHong/DaGAN)] \u0026emsp; [[Poster Video](https://www.youtube.com/watch?v=nahsJNjWzGo\u0026t=1s)]\u003cbr\u003e\n\u003c!-- \u003e [Fa-Ting Hong](https://harlanhong.github.io), [Longhao Zhang](https://dblp.org/pid/236/7382.html), [Li Shen](https://scholar.google.co.uk/citations?user=ABbCaxsAAAAJ\u0026hl=en), [Dan Xu](https://www.danxurgb.net) \u003cbr\u003e --\u003e\n\u003c!-- \u003e The Hong Kong University of Science and Technology, Alibaba Cloud --\u003e\n\u003e [Fa-Ting Hong](https://harlanhong.github.io), [Longhao Zhang](), [Li Shen](), [Dan Xu](https://www.danxurgb.net) \u003cbr\u003e\n\u003e The Hong Kong University of Science and Technology\u003cbr\u003e\n\u003e Alibaba Cloud\n\n### Cartoon Sample\nhttps://user-images.githubusercontent.com/19970321/162151632-0195292f-30b8-4122-8afd-9b1698f1e4fe.mp4\n\n### Human Sample\nhttps://user-images.githubusercontent.com/19970321/162151327-f2930231-42e3-40f2-bfca-a88529599f0f.mp4\n\n### Voxceleb1 Dataset\n\u003cp align=\"center\"\u003e\n  \u003cimg src=\"assets/visual_vox1.png\"\u003e\n\u003c/p\u003e\n\n:triangular_flag_on_post: **Updates** \n- :fire::fire::white_check_mark: July 20 2023: Our new talking head work **[MCNet](https://harlanhong.github.io/publications/mcnet.html) was accpted by ICCV2023. There's no need to train a facial depth network, which makes it more convenient for users to test and fine-tune.\n- :fire::fire::white_check_mark: July 26, 2022: The normal dataparallel training scripts were released since some researchers informed me they ran into **DistributedDataParallel** problems. Please try to train your own model using this [command](#dataparallel). Also, we deleted the command line \"with torch.autograd.set_detect_anomaly(True)\" to boost the training speed.\n- :fire::fire::white_check_mark: June 26, 2022: The repo of our face depth network is released, please refer to [Face-Depth-Network](https://github.com/harlanhong/Face-Depth-Network) and feel free to email me if you meet any problem.\n- :fire::fire::white_check_mark: June 21, 2022: [Digression] I am looking for research intern/research assistant opportunities in European next year. Please contact me If you think I'm qualified for your position.\n- :fire::fire::white_check_mark: May 19, 2022: The depth face model (50 layers) trained on Voxceleb2 is released! (The corresponding checkpoint of DaGAN will release soon). Click the [LINK](https://hkustconnect-my.sharepoint.com/:f:/g/personal/fhongac_connect_ust_hk/EkxzfH7zbGJNr-WVmPU6fcABWAMq_WJoExAl4SttKK6hBQ?e=fbtGlX)\n\n- :fire::fire::white_check_mark: April 25, 2022: Integrated into Huggingface Spaces 🤗 using Gradio. Try out the web demo: [![Hugging Face Spaces](https://img.shields.io/badge/%F0%9F%A4%97%20Hugging%20Face-Spaces-blue)](https://huggingface.co/spaces/HarlanHong/DaGAN) (GPU version will come soon!)\n- :fire::fire::white_check_mark: Add **[SPADE model](https://hkustconnect-my.sharepoint.com/:f:/g/personal/fhongac_connect_ust_hk/EjfeXuzwo3JMn7s0oOPN_q0B81P5Wgu_kbYJAh7uSAKS2w?e=XNZl3K)**, which produces **more natural** results.\n\n\n## :wrench: Dependencies and Installation\n\n- Python \u003e= 3.7 (Recommend to use [Anaconda](https://www.anaconda.com/download/#linux) or [Miniconda](https://docs.conda.io/en/latest/miniconda.html))\n- [PyTorch \u003e= 1.7](https://pytorch.org/)\n- Option: NVIDIA GPU + [CUDA](https://developer.nvidia.com/cuda-downloads)\n- Option: Linux\n\n### Installation\nWe now provide a *clean* version of DaGAN, which does not require customized CUDA extensions. \u003cbr\u003e\n\n1. Clone repo\n\n    ```bash\n    git clone https://github.com/harlanhong/CVPR2022-DaGAN.git\n    cd CVPR2022-DaGAN\n    ```\n\n2. Install dependent packages\n\n    ```bash\n    pip install -r requirements.txt\n\n    ## Install the Face Alignment lib\n    cd face-alignment\n    pip install -r requirements.txt\n    python setup.py install\n    ```\n## :zap: Quick Inference\n\nWe take the paper version for an example. More models can be found [here](https://hkustconnect-my.sharepoint.com/:f:/g/personal/fhongac_connect_ust_hk/EjfeXuzwo3JMn7s0oOPN_q0B81P5Wgu_kbYJAh7uSAKS2w?e=KaQcPk).\n\n### YAML configs\nSee ```config/vox-adv-256.yaml``` to get description of each parameter.\n\n### Pre-trained checkpoint\nThe pre-trained checkpoint of face depth network and our DaGAN checkpoints can be found under following link: [OneDrive](https://hkustconnect-my.sharepoint.com/:f:/g/personal/fhongac_connect_ust_hk/EjfeXuzwo3JMn7s0oOPN_q0B81P5Wgu_kbYJAh7uSAKS2w?e=KaQcPk).\n\n**Inference!**\nTo run a demo, download checkpoint and run the following command:\n\n```bash\nCUDA_VISIBLE_DEVICES=0 python demo.py  --config config/vox-adv-256.yaml --driving_video path/to/driving --source_image path/to/source --checkpoint path/to/checkpoint --relative --adapt_scale --kp_num 15 --generator DepthAwareGenerator \n```\nThe result will be stored in ```result.mp4```. The driving videos and source images should be cropped before it can be used in our method. To obtain some semi-automatic crop suggestions you can use ```python crop-video.py --inp some_youtube_video.mp4```. It will generate commands for crops using ffmpeg. \n\n\n\n\n## :computer: Training\n\n\n### Datasets\n \n1) **VoxCeleb**. Please follow the instruction from https://github.com/AliaksandrSiarohin/video-preprocessing.\n\n### Train on VoxCeleb\nTo train a model on specific dataset run:\n```\nCUDA_VISIBLE_DEVICES=0,1,2,3,4,5,6,7 python -m torch.distributed.launch --master_addr=\"0.0.0.0\" --master_port=12348 run.py --config config/vox-adv-256.yaml --name DaGAN --rgbd --batchsize 12 --kp_num 15 --generator DepthAwareGenerator\n```\n\u003cdiv id=\"dataparallel\" \u003eOr\u003c/div\u003e\n\n```\nCUDA_VISIBLE_DEVICES=0,1,2,3 python run_dataparallel.py --config config/vox-adv-256.yaml --device_ids 0,1,2,3 --name DaGAN_voxceleb2_depth --rgbd --batchsize 48 --kp_num 15 --generator DepthAwareGenerator\n```\n\n\n\u003c!-- CUDA_VISIBLE_DEVICES=1,2,3,4,5,6,7 python -m torch.distributed.launch --master_addr=\"0.0.0.0\" --master_port=12348 run.py --config config/vox-adv-256.yaml --name SpadeDaGAN --rgbd --batchsize 6 --kp_num 15 --generator SPADEDepthAwareGenerator --\u003e\n\nThe code will create a folder in the log directory (each run will create a new name-specific directory).\nCheckpoints will be saved to this folder.\nTo check the loss values during training see ```log.txt```.\nBy default the batch size is tunned to run on 8 GeForce RTX 3090 gpu (You can obtain the best performance after about 150 epochs). You can change the batch size in the train_params in ```.yaml``` file.\n\n\nAlso, you can watch the training loss by running the following command:\n```bash\ntensorboard --logdir log/DaGAN/log\n```\nWhen you kill your process for some reasons in the middle of training, a zombie process may occur, you can kill it using our provided tool:\n ```bash\npython kill_port.py PORT\n```\n\n### Training on your own dataset\n1) Resize all the videos to the same size e.g 256x256, the videos can be in '.gif', '.mp4' or folder with images.\nWe recommend the later, for each video make a separate folder with all the frames in '.png' format. This format is loss-less, and it has better i/o performance.\n\n2) Create a folder ```data/dataset_name``` with 2 subfolders ```train``` and ```test```, put training videos in the ```train``` and testing in the ```test```.\n\n3) Create a config ```config/dataset_name.yaml```, in dataset_params specify the root dir the ```root_dir:  data/dataset_name```. Also adjust the number of epoch in train_params.\n\n\n\n## :scroll: Acknowledgement\n\n Our DaGAN implementation is inspired by [FOMM](https://github.com/AliaksandrSiarohin/first-order-model). We appreciate the authors of [FOMM](https://github.com/AliaksandrSiarohin/first-order-model) for making their codes available to public.\n\n## :scroll: BibTeX\n\n```\n@inproceedings{hong2022depth,\n            title={Depth-Aware Generative Adversarial Network for Talking Head Video Generation},\n            author={Hong, Fa-Ting and Zhang, Longhao and Shen, Li and Xu, Dan},\n            journal={IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)},\n            year={2022}\n          }\n\n@article{hong2023dagan,\n            title={DaGAN++: Depth-Aware Generative Adversarial Network for Talking Head Video Generation},\n            author={Hong, Fa-Ting and and Shen, Li and Xu, Dan},\n            journal={IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI)},\n            year={2023}\n          }\n```\n\n### :e-mail: Contact\n\nIf you have any question or collaboration need (research purpose or commercial purpose), please email `fhongac@cse.ust.hk`.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fharlanhong%2FCVPR2022-DaGAN","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fharlanhong%2FCVPR2022-DaGAN","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fharlanhong%2FCVPR2022-DaGAN/lists"}