{"id":21352220,"url":"https://github.com/yfeng95/delta","last_synced_at":"2025-04-03T01:13:44.317Z","repository":{"id":194430141,"uuid":"677871822","full_name":"yfeng95/DELTA","owner":"yfeng95","description":"Learning Disentangled Avatars with Hybrid 3D Representations. (Face, Body, Hair and Clothing)","archived":false,"fork":false,"pushed_at":"2023-09-27T22:19:33.000Z","size":34495,"stargazers_count":253,"open_issues_count":10,"forks_count":15,"subscribers_count":14,"default_branch":"main","last_synced_at":"2025-03-24T08:05:17.139Z","etag":null,"topics":["avatar","body","clothing","face","hair","modeling","reconstruction","try-on","virtual"],"latest_commit_sha":null,"homepage":"https://yfeng95.github.io/delta/","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"other","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/yfeng95.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2023-08-12T23:30:34.000Z","updated_at":"2025-02-28T06:20:06.000Z","dependencies_parsed_at":null,"dependency_job_id":"a9b9ec61-bb40-4078-af70-b1ec3cd6d352","html_url":"https://github.com/yfeng95/DELTA","commit_stats":{"total_commits":9,"total_committers":1,"mean_commits":9.0,"dds":0.0,"last_synced_commit":"9736386b5f6dd5fd81d79001e3edd5ef75209edf"},"previous_names":["yfeng95/delta"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/yfeng95%2FDELTA","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/yfeng95%2FDELTA/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/yfeng95%2FDELTA/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/yfeng95%2FDELTA/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/yfeng95","download_url":"https://codeload.github.com/yfeng95/DELTA/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":246916761,"owners_count":20854514,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["avatar","body","clothing","face","hair","modeling","reconstruction","try-on","virtual"],"created_at":"2024-11-22T03:13:00.812Z","updated_at":"2025-04-03T01:13:44.297Z","avatar_url":"https://github.com/yfeng95.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"\u003c!-- PROJECT LOGO --\u003e\n\n\u003cp align=\"center\"\u003e\n  \u003ch1 align=\"center\"\u003eDELTA: Learning Disentangled Avatars with Hybrid 3D Representations\n \u003c/h1\u003e\n\u003c!--  \u003cp align=\"center\"\u003e\n    \u003ca href=\"https://ps.is.tuebingen.mpg.de/person/yxiu\"\u003e\u003cstrong\u003eYao Feng\u003c/strong\u003e\u003c/a\u003e\n    ·\n    \u003ca href=\"https://ps.is.tuebingen.mpg.de/person/jyang\"\u003e\u003cstrong\u003eJinlong Yang\u003c/strong\u003e\u003c/a\u003e\n    ·\n    \u003ca href=\"https://ps.is.tuebingen.mpg.de/person/black\"\u003e\u003cstrong\u003eMichael J. Black\u003c/strong\u003e\u003c/a\u003e\n    .\n    \u003ca href=\"https://people.inf.ethz.ch/pomarc/\"\u003e\u003cstrong\u003eMarc Pollefeys\u003c/strong\u003e\u003c/a\u003e\n    .\n    \u003ca href=\"https://ps.is.mpg.de/person/tbolkart\"\u003e\u003cstrong\u003eTimo Bolkart\u003c/strong\u003e\u003c/a\u003e\n  \u003c/p\u003e\n  \u003ch2 align=\"center\"\u003eSIGGRAPH Asia 2022 conference \u003c/h2\u003e\n  --\u003e\n  \u003cdiv align=\"center\"\u003e\n    \u003cimg src=\"Doc/images/teaser_delta.png\" alt=\"teaser\" width=\"100%\"\u003e\n  \u003c/div\u003e\n\u003c/p\u003e \n\nThis is the Pytorch implementation of DELTA. More details please check our [Project](https://yfeng95.github.io/delta) page.  \n\nDELTA leans a compositional avatar using explicit Mesh and implicit NeRF represetntaions.  \nDELTA allows us to synthesize new views of the reconstructed avatar, and to animate the avatar with SMPL-X identity shape and pose control.  \nThe disentanglement of the body and hair/clothing further enables us to transfer hairstyle/clothing between subjects for virtual try-on applications.  \nThe key features:\n1. animate the avatar by changing body poses (including hand articulation and facial expressions),  \n2. synthesize novel views of the avatar, and  \n3. transfer hair/clothing between avatars for virtual try-on applications.  \n\nWe have examples for reconstructing avatars for face/upper-body videos and full-body videos.   \nFor **full-body** video, please check [SCARF](https://github.com/yfeng95/SCARF).  \nFor generating compositional avatar from **text**, please check [TECA](https://yfeng95.github.io/teca).\n  \nWe also provide code for data processing [here](./process_data/README.md), which includes **fitting face, neck and shoulder** to a single image or a monocular video. \n\n## Getting Started\nClone the repo:\n  ```bash\n  git clone https://github.com/yfeng95/DELTA\n  cd DELTA\n  ```  \n### Requirements\n```bash\nbash install_conda.sh\n``` \nIf you have problems when installing [pytorch3d](https://github.com/facebookresearch/pytorch3d/blob/main/INSTALL.md), please follow their instructions.\n\n### Download data \n```\nbash fetch_data.sh\n```\n\n## Play with trained avatars\n* check training frames: \n```\npython main_demo.py --expdir exps --exp_name  person_0004 --visualize capture \n```\n\u003cp align=\"center\"\u003e\n  \u003cdiv align=\"center\"\u003e\n    \u003cimg src=\"Doc/exps/person_0004_capture.gif\" alt=\"capture\" width=\"40%\"\u003e\n  \u003c/div\u003e\n\u003c/p\u003e \n\n* **novel view** synthesis of given frame id: \n```bash\npython main_demo.py --expdir exps --exp_name  person_0004 --visualize novel_view --frame_id 0\n```\n\u003cp align=\"center\"\u003e\n  \u003cdiv align=\"center\"\u003e\n    \u003cimg src=\"Doc/exps/person_0004_000000_novel_view.gif\" alt=\"novel view\" width=\"25%\"\u003e\n  \u003c/div\u003e\n\u003c/p\u003e \n\n* extract **mesh** and visualize \n```bash\npython main_demo.py --expdir exps --exp_name  person_0004 --visualize extract_mesh --frame_id 0\n```\nThis will also save mesh objects (body only and body with hair) that you can open with Meshlab, green is for extracted hair geometry. \n\u003cp align=\"center\"\u003e\n  \u003cdiv align=\"center\"\u003e\n    \u003cimg src=\"Doc/exps/person_0004_mesh_speedup.gif\" alt=\"change shape\" width=\"25%\"\u003e\n  \u003c/div\u003e\n\u003c/p\u003e \n\n* **animate** with given animation sequences\n```bash\npython main_demo.py --expdir exps --exp_name  person_0004 --visualize animate\n```\n* change body shape\n```bash\npython main_demo.py --expdir exps --exp_name  person_0004 --visualize change_shape\n```\n\u003cp align=\"center\"\u003e\n  \u003cdiv align=\"center\"\u003e\n    \u003cimg src=\"Doc/exps/person_0004_000000_change_shape.gif\" alt=\"change shape\" width=\"25%\"\u003e\n  \u003c/div\u003e\n\u003c/p\u003e \n\n* **Hairstyle transfer**\nTransfer hairstyle and visualize: \n```bash\npython main_demo.py --expdir exps --exp_name person_2_train     --body_model_path exps/released_version/person_0004/model.tar  --visualize novel_view --max_yaw 20 \n```\n\u003cp align=\"center\"\u003e \n\u003c!-- \u003cimg src=\"Doc/exps/person_0004_000000_hair_transfer.gif\" alt=\"change shape\" width=\"25%\"\u003e --\u003e\n\u003cimg src=\"Doc/exps/1_hair.gif\" alt=\"change shape\" width=\"40%\"\u003e\n\u003c/p\u003e\n\n## Training \n* training DELTA\n```bash\npython main_train.py --expdir exps --group training_hybrid \\\n     --exp_cfg configs/exp/face/hybrid_ngp.yml \\\n     --data_cfg exps/released_version/person_0004/data_config.yaml \n```\n* training NeRF only\n```bash\npython main_train.py --expdir exps --group training_nerf \\\n     --exp_cfg configs/exp/face/nerf_ngp.yml \\\n     --data_cfg exps/released_version/person_0004/data_config.yaml \n```\n* training with your own data  \n\ncheck [here](./process_data/README.md) to **prepare data with your own videos**, the processed data will be like this: \n\u003cdiv align=\"center\"\u003e\n  \u003cimg src=\"Doc/images/data_teaser.png\" alt=\"visualize data\" width=\"60%\"\u003e\n    \u003c!-- \u003ch5 align=\"center\"\u003ecropped image, subject segmentation, clothing segmentation, SMPL-X estimation \u003c/h5\u003e --\u003e\n\u003c/div\u003e \n\u003c!-- \u003cdiv align=\"center\" dis\u003e\n    \u003ctable class=\"images\" width=\"100%\"  style=\"border:0px solid white; width:100%;\"\u003e\n        \u003ctr style=\"border: 0px;\"\u003e\n            \u003ctd style=\"border: 0px;\"\u003e\u003cimg src=\"Doc/dataset/fit_person_0004.gif\" width=\"128\"\u003e\u003c/a\u003e\u003c/td\u003e\n            \u003ctd style=\"border: 0px;\"\u003e\u003cimg src=\"Doc/dataset/fit_MVI_1810.gif\" width=\"128\"\u003e\u003c/a\u003e\u003c/td\u003e   \n        \u003c/tr\u003e\n    \u003c/table\u003e\n\u003c/div\u003e --\u003e\n\nTo process full body video, please check [here](https://github.com/yfeng95/SCARF/blob/main/process_data/README.md).  \n\nThen run: \n```bash\npython main_train.py --exp_cfg configs/exp/face/hybrid_ngp.yml \\\n     --data_cfg [data cfg file]\n```\n\n\n## Citation\n```bibtex\n@inproceedings{Feng2022scarf,\n    author = {Feng, Yao and Yang, Jinlong and Pollefeys, Marc and Black, Michael J. and Bolkart, Timo},\n    title = {Capturing and Animation of Body and Clothing from Monocular Video},\n    year = {2022},\n    booktitle = {SIGGRAPH Asia 2022 Conference Papers},\n    articleno = {45},\n    numpages = {9},\n    location = {Daegu, Republic of Korea},\n    series = {SA '22}\n} \n@article{Feng2023DELTA,\n    author = {Feng, Yao and Liu, Weiyang and Bolkart, Timo and Yang, Jinlong and Pollefeys, Marc and Black, Michael J.},\n    title = {Learning Disentangled Avatars with Hybrid 3D Representations},\n    journal={arXiv},\n    year = {2023}\n} \n```\n## Acknowledgments \nHere are some great resources we benefit from:\n- [face alignment](https://github.com/1adrianb/face-alignment) to detect face keypoints\n- [MediaPipe](hhttps://github.com/patlevin/face-detection-tflite) to detect iris\n- [MODNet](https://github.com/ZHKKKe/MODNet) to segment subject\n- [face-parsing](https://github.com/zllrunning/face-parsing.PyTorch) to segment hair\n- [RobustVideoMatting](https://github.com/PeterL1n/RobustVideoMatting) for background segmentation\n- [cloth-segmentation](https://github.com/levindabhi/cloth-segmentation) for clothing segmentation\n- [PIXIE](https://github.com/yfeng95/PIXIE) for SMPL-X parameters estimation  \n- [smplx](https://github.com/vchoutas/smplx) for body models \n- [PyTorch3D](https://github.com/facebookresearch/pytorch3d) for Differential Rendering\n- [NerfAcc](https://github.com/KAIR-BAIR/nerfacc) \n- [DECA](https://github.com/yfeng95/DECA) \n- [Neural Head Avatars](https://github.com/philgras/neural-head-avatars)  \nSome functions are based on other repositories, we acknowledge the origin individually in each file.\n\n## License\n\nThis code and model are available for non-commercial scientific research purposes as defined in the [LICENSE](LICENSE) file. By downloading and using the code and model you agree to the terms in the [LICENSE](LICENSE).\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fyfeng95%2Fdelta","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fyfeng95%2Fdelta","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fyfeng95%2Fdelta/lists"}