{"id":13768354,"url":"https://github.com/theEricMa/OTAvatar","last_synced_at":"2025-05-10T23:31:15.727Z","repository":{"id":148975541,"uuid":"611590475","full_name":"theEricMa/OTAvatar","owner":"theEricMa","description":"This is the official repository for OTAvatar: One-shot Talking Face Avatar with Controllable Tri-plane Rendering [CVPR2023].","archived":false,"fork":false,"pushed_at":"2024-03-05T06:04:21.000Z","size":12977,"stargazers_count":310,"open_issues_count":7,"forks_count":39,"subscribers_count":11,"default_branch":"main","last_synced_at":"2024-11-17T03:31:28.749Z","etag":null,"topics":["cvpr2023","deep-learning","deepfake","face-animation","face-reenactment","image-animation","motion-transfer","pose-transfer","pytorch","talking-head"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/theEricMa.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"licenses/LICENSE.md","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null}},"created_at":"2023-03-09T06:10:42.000Z","updated_at":"2024-10-11T08:43:22.000Z","dependencies_parsed_at":null,"dependency_job_id":"f0f1537e-9543-4a24-84ea-1749f00086bb","html_url":"https://github.com/theEricMa/OTAvatar","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/theEricMa%2FOTAvatar","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/theEricMa%2FOTAvatar/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/theEricMa%2FOTAvatar/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/theEricMa%2FOTAvatar/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/theEricMa","download_url":"https://codeload.github.com/theEricMa/OTAvatar/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":253497296,"owners_count":21917683,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["cvpr2023","deep-learning","deepfake","face-animation","face-reenactment","image-animation","motion-transfer","pose-transfer","pytorch","talking-head"],"created_at":"2024-08-03T16:01:20.163Z","updated_at":"2025-05-10T23:31:10.712Z","avatar_url":"https://github.com/theEricMa.png","language":"Python","readme":"# OTAvatar : One-shot Talking Face Avatar with Controllable Tri-plane Rendering\n## [Paper](https://arxiv.org/abs/2303.14662)  |  [Demo](https://youtu.be/qpIoMYFr7Aw)\n\n## Update\n\nApril.30: The model weight is released. The dataset is also available in Google Drive, see below for detail.\n\nApril.4: The preprocessed dataset is released, please see the `Data preparation` section. Some missing files are also uploaded.\n\n## Get started\n### Environment Setup\n```\ngit clone git@github.com:theEricMa/OTAvatar.git\ncd OTAvatar\nconda env create -f environment.yml\nconda activate otavatar\n```\n\n### Pre-trained Models\nDownload and copy EG3D FFHQ model `ffhqrebalanced512-64.pth` [[Baidu Netdisk](https://pan.baidu.com/s/1_iEqB7qbJBK7DsjlxxW8MA?pwd=CBSR)][[Google Drive](https://drive.google.com/file/d/18RWTfeydaG_qm__rgYKPdxKcS0D_f2mr/view?usp=share_link)] to the `pretrained` directory. It is the `ffhqrebalanced512-64.pkl` file obtained from [webpage](https://catalog.ngc.nvidia.com/orgs/nvidia/teams/research/models/eg3d), and converted to `.pth` format using the [pkl2pth](https://github.com/oneThousand1000/EG3D-projector/blob/master/eg3d/convert_pkl_2_pth.py) script.\n\nDownload [`arcface_resnet18.pth`](https://github.com/ronghuaiyang/arcface-pytorch) and save to the `pretrained` directory.\n\n### Data preparation \nWe upload the processed dataset `hdtf_lmdb_inv` in [[Baidu Netdisk](https://pan.baidu.com/s/1yXvLR2WvwdamGrWFnNefZw?pwd=CBSR)][[Google Drive](https://drive.google.com/drive/folders/1kDIhBU_Cz_HyfYHAuETcDquCuZBDfqeu?usp=share_link)]. In the root directory,\n\n```\nmkdir datasets\nmv \u003cyour hdtf_lmdb_inv path\u003e datasets/\n```\n\nGenerally the processing scripts is a mixture of that in [PIRenderer](https://github.com/RenYurui/PIRender) and [ADNeRF](https://github.com/YudongGuo/AD-NeRF). We plan to further open a new repo to upload our revised preocessing script.\n\n### Face Animation\nCreate the folder `result/otavatar`if it does not exist. Place the model downloaded from [[Baidu Netdisk](https://pan.baidu.com/s/1bLlXMUT4r76MQc4vywLreg?pwd=CBSR)][[Google Drive](https://drive.google.com/file/d/1JW1ieAgeu5qugC41O7zjn7IA49Jfnytt/view?usp=share_link)] under this directory. Run,\n```\nexport CUDA_VISIBLE_DEVICES=0\npython -m torch.distributed.launch --nproc_per_node=1 --master_port 12345 inference_refine_1D_cam.py \\\n--config ./config/otavatar.yaml \\\n--name otavatar \\\n--no_resume \\\n--which_iter 2000 \\\n--image_size 512 \\\n--ws_plus \\\n--cross_id \\\n--cross_id_target WRA_EricCantor_000 \\\n--output_dir ./result/otavatar/evaluation/cross_ws_plus_WRA_EricCantor_000\n```\nTo animate each identity given the motion from `WRA_EricCantor_000`.\n\nOr simply run,\n```\nsh scripts/inference.sh\n```\n\n### Start Training\nRun,\n```\nexport CUDA_VISIBLE_DEVICES=0,1,2,3\npython -m torch.distributed.launch --nproc_per_node=4 --master_port 12346 train_inversion.py \\\n--config ./config/otavatar.yaml \\\n--name otavatar\n```\n\nOr simply run,\n```\nsh scripts/train.sh\n```\n\n### Acknowledgement\nWe appreciate the model or code from [EG3D](https://github.com/NVlabs/eg3d), [PIRenderer](https://github.com/RenYurui/PIRender), [StyleHEAT](https://github.com/FeiiYin/StyleHEAT), [EG3D-projector](https://github.com/oneThousand1000/EG3D-projector).\n\n### Citation\nIf you find this work helpful, please cite:\n```\n@article{ma2023otavatar,\n  title={OTAvatar: One-shot Talking Face Avatar with Controllable Tri-plane Rendering},\n  author={Ma, Zhiyuan and Zhu, Xiangyu and Qi, Guojun and Lei, Zhen and Zhang, Lei},\n  journal={arXiv preprint arXiv:2303.14662},\n  year={2023}\n}\n```\n\n","funding_links":[],"categories":["GitHub projects"],"sub_categories":[],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FtheEricMa%2FOTAvatar","html_url":"https://awesome.ecosyste.ms/projects/github.com%2FtheEricMa%2FOTAvatar","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FtheEricMa%2FOTAvatar/lists"}