{"id":13768408,"url":"https://github.com/shrubb/latent-pose-reenactment","last_synced_at":"2025-09-19T08:32:31.082Z","repository":{"id":53574181,"uuid":"302642567","full_name":"shrubb/latent-pose-reenactment","owner":"shrubb","description":"The authors' implementation of the \"Neural Head Reenactment with Latent Pose Descriptors\" (CVPR 2020) paper.","archived":false,"fork":false,"pushed_at":"2023-03-14T10:56:39.000Z","size":1662,"stargazers_count":181,"open_issues_count":6,"forks_count":34,"subscribers_count":5,"default_branch":"master","last_synced_at":"2024-12-27T14:39:00.688Z","etag":null,"topics":["avatar","deep-learning","face-reenactment","facial-landmarks","generative-model","head-avatar","head-reenactment","landmark-detection","pose-estimation","pytorch","self-supervised-learning","talking-head","voxceleb","voxceleb2"],"latest_commit_sha":null,"homepage":"https://shrubb.github.io/research/latent-pose-reenactment/","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/shrubb.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE.txt","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null}},"created_at":"2020-10-09T13:01:44.000Z","updated_at":"2024-09-27T06:16:10.000Z","dependencies_parsed_at":"2024-01-06T20:18:35.618Z","dependency_job_id":"504e8519-081c-4783-ad8e-cab47e2ce0a3","html_url":"https://github.com/shrubb/latent-pose-reenactment","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/shrubb%2Flatent-pose-reenactment","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/shrubb%2Flatent-pose-reenactment/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/shrubb%2Flatent-pose-reenactment/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/shrubb%2Flatent-pose-reenactment/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/shrubb","download_url":"https://codeload.github.com/shrubb/latent-pose-reenactment/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":233561338,"owners_count":18694487,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["avatar","deep-learning","face-reenactment","facial-landmarks","generative-model","head-avatar","head-reenactment","landmark-detection","pose-estimation","pytorch","self-supervised-learning","talking-head","voxceleb","voxceleb2"],"created_at":"2024-08-03T16:01:21.064Z","updated_at":"2025-09-19T08:32:25.595Z","avatar_url":"https://github.com/shrubb.png","language":"Python","funding_links":[],"categories":["GitHub projects"],"sub_categories":[],"readme":"# Neural Head Reenactment with Latent Pose Descriptors\n\n![](https://user-images.githubusercontent.com/9570420/94962966-0a8bb900-0500-11eb-90ee-3315368019b8.png)\n\nBurkov, E., Pasechnik, I., Grigorev, A., \u0026 Lempitsky V. (2020, June). **Neural Head Reenactment with Latent Pose Descriptors**. *IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)*.\n\nSee the [project page](https://shrubb.github.io/research/latent-pose-reenactment/) for an overview.\n\n## Prerequisites\n\nFor fine-tuning a pre-trained model, you'll need an NVIDIA GPU, preferably with 8+ GB VRAM. To train from scratch, we recommend a total of 40+ GB VRAM.\n\nSet up your environment as described [here](INSTALL.md).\n\n## Running the pretrained model\n\n* Collect images of the person to reenact.\n* Run [`utils/preprocess_dataset.sh`](utils/preprocess_dataset.sh) to preprocess them. Read inside for instructions.\n* Download the meta-model [checkpoint](https://drive.google.com/file/d/14-FYaz6YhTX5M_P3-rm2ITcxGljmWl-F/view?usp=share_link).\n* Run the below to fine-tune the meta-model to your person, first setting the top variables. If you want, also launch a TensorBoard at `\"$OUTPUT_PATH\"` to view progress, preferably with the [`--samples_per_plugin \"scalars=1000,images=100\"`](https://stackoverflow.com/questions/57669234/how-to-display-more-than-10-images-in-tensorboard) option; mainly check the \"images\" tab to find out at which iteration the identity gap becomes small enough.\n\n```bash\n# in this example, your images should be \"$DATASET_ROOT/images-cropped/$IDENTITY_NAME/*.jpg\"\nDATASET_ROOT=\"/where/is/your/data\"\nIDENTITY_NAME=\"identity/name\"\nMAX_BATCH_SIZE=8             # pick the largest possible, start with 8 and decrease until it fits in VRAM\nCHECKPOINT_PATH=\"/where/is/checkpoint.pth\"\nOUTPUT_PATH=\"outputs/\"       # a directory for outputs, will be created\nRUN_NAME=\"tony_hawk_take_1\"  # give your run a name if you want\n\n# Important. See the note below\nTARGET_NUM_ITERATIONS=230\n\n# Don't change these\nNUM_IMAGES=`ls -1 \"$DATASET_ROOT/images-cropped/$IDENTITY_NAME\" | wc -l`\nBATCH_SIZE=$((NUM_IMAGES\u003cMAX_BATCH_SIZE ? NUM_IMAGES : MAX_BATCH_SIZE))\nITERATIONS_IN_EPOCH=$(( NUM_IMAGES / BATCH_SIZE ))\n\nmkdir -p $OUTPUT_PATH\n\npython3 train.py \\\n    --config finetuning-base                 \\\n    --checkpoint_path \"$CHECKPOINT_PATH\"     \\\n    --data_root \"$DATASET_ROOT\"              \\\n    --train_split_path \"$IDENTITY_NAME\"      \\\n    --batch_size $BATCH_SIZE                 \\\n    --num_epochs $(( (TARGET_NUM_ITERATIONS + ITERATIONS_IN_EPOCH - 1) / ITERATIONS_IN_EPOCH )) \\\n    --experiments_dir \"$OUTPUT_PATH\"         \\\n    --experiment_name \"$RUN_NAME\"\n```\n\n**Note**. `TARGET_NUM_ITERATIONS` is important, make sure to tune it. Pick too low, underfit and get an identity gap; pick too high, overfit and get poor mimics. I suggest that you start with **125 when `NUM_IMAGES=1`** and increase with more images, say, to **230 when `NUM_IMAGES\u003e30`**. But your concrete case may be different. If you have a lot of disk space, pass a flag to save checkpoints every so often (e.g. `--save_frequency 4` will save a checkpoint every `4 * NUM_IMAGES` iterations), then drive (see below how) each of them and thus find the iteration where the best tradeoff happens for your avatar.\n\n* Take your driving video and crop it with `python3 utils/crop_as_in_dataset.py`. Run with `--help` to learn how. Or, equivalently, just reuse [`utils/preprocess_dataset.sh`](utils/preprocess_dataset.sh) with `COMPUTE_SEGMENTATION=false`.\n* Organize the cropped images from the previous step as `\"\u003cdata_root\u003e/images-cropped/\u003cimages_path\u003e/*.jpg\"`.\n* Use them to drive your fine-tuned model (the checkpoint is at `\"$OUTPUT_PATH/$RUN_NAME/checkpoints\"`) with `python3 drive.py`. Run with `--help` to learn how.\n\n## Training (meta-learning) your own model\n\nYou'll need a training configuration (aka config) file. Start with `\"configs/default.yaml\"` or just edit that. These files specify various training options which you can find in code as `argparse` parameters. Any of these options can be specified both in the config file and on the command line (e.g. `--batch_size=7`), and are resolved as follows (any source here overrides all the preceding ones):\n\n* `argparse` defaults — these are specified in the code directly;\n* those saved in a loaded checkpoint (if starting from a checkpoint);\n* your `--config` file;\n* command line.\n\nThe command is\n\n```bash\npython3 train.py --config=config_name [any extra arguments ...]\n```\n\nOr, with multiple GPUs,\n\n```bash\npython3 -um torch.distributed.launch --nproc_per_node=\u003cnumber of GPUs\u003e train.py --config=config_name [any extra arguments ...]\n```\n\n## Reference\n\nConsider citing us if you use the code:\n\n```bibtex\n@InProceedings{Burkov_2020_CVPR,\nauthor = {Burkov, Egor and Pasechnik, Igor and Grigorev, Artur and Lempitsky, Victor},\ntitle = {Neural Head Reenactment with Latent Pose Descriptors},\nbooktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)},\nmonth = {June},\nyear = {2020}\n}\n```\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fshrubb%2Flatent-pose-reenactment","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fshrubb%2Flatent-pose-reenactment","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fshrubb%2Flatent-pose-reenactment/lists"}