{"id":15026366,"url":"https://github.com/yuliangxiu/icon","last_synced_at":"2025-05-15T18:05:44.508Z","repository":{"id":37409416,"uuid":"433191004","full_name":"YuliangXiu/ICON","owner":"YuliangXiu","description":"[CVPR'22] ICON: Implicit Clothed humans Obtained from Normals","archived":false,"fork":false,"pushed_at":"2023-11-23T16:44:57.000Z","size":159315,"stargazers_count":1631,"open_issues_count":49,"forks_count":218,"subscribers_count":40,"default_branch":"master","last_synced_at":"2025-04-08T10:13:29.884Z","etag":null,"topics":["3d-reconstruction","animation","avatar-generator","cloth-simulation","computer-graphics","computer-vision","human-pose-estimation","implicit-functions","mesh-deformation","metaverse","normal-maps","pifu","pifuhd","pose-estimation","pytorch","smpl","smpl-body","smpl-model","smplx","virtual-humans"],"latest_commit_sha":null,"homepage":"https://icon.is.tue.mpg.de","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"other","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/YuliangXiu.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null}},"created_at":"2021-11-29T20:42:47.000Z","updated_at":"2025-04-08T02:18:23.000Z","dependencies_parsed_at":"2023-11-23T17:47:39.423Z","dependency_job_id":null,"html_url":"https://github.com/YuliangXiu/ICON","commit_stats":null,"previous_names":[],"tags_count":4,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/YuliangXiu%2FICON","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/YuliangXiu%2FICON/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/YuliangXiu%2FICON/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/YuliangXiu%2FICON/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/YuliangXiu","download_url":"https://codeload.github.com/YuliangXiu/ICON/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":247819933,"owners_count":21001394,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["3d-reconstruction","animation","avatar-generator","cloth-simulation","computer-graphics","computer-vision","human-pose-estimation","implicit-functions","mesh-deformation","metaverse","normal-maps","pifu","pifuhd","pose-estimation","pytorch","smpl","smpl-body","smpl-model","smplx","virtual-humans"],"created_at":"2024-09-24T20:04:22.010Z","updated_at":"2025-04-08T10:13:58.321Z","avatar_url":"https://github.com/YuliangXiu.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"\u003c!-- PROJECT LOGO --\u003e\n\n\u003cp align=\"center\"\u003e\n\n  \u003ch1 align=\"center\"\u003eICON: Implicit Clothed humans Obtained from Normals\u003c/h1\u003e\n  \u003cp align=\"center\"\u003e\n    \u003ca href=\"https://ps.is.tuebingen.mpg.de/person/yxiu\"\u003e\u003cstrong\u003eYuliang Xiu\u003c/strong\u003e\u003c/a\u003e\n    ·\n    \u003ca href=\"https://ps.is.tuebingen.mpg.de/person/jyang\"\u003e\u003cstrong\u003eJinlong Yang\u003c/strong\u003e\u003c/a\u003e\n    ·\n    \u003ca href=\"https://ps.is.mpg.de/~dtzionas\"\u003e\u003cstrong\u003eDimitrios Tzionas\u003c/strong\u003e\u003c/a\u003e\n    ·\n    \u003ca href=\"https://ps.is.tuebingen.mpg.de/person/black\"\u003e\u003cstrong\u003eMichael J. Black\u003c/strong\u003e\u003c/a\u003e\n  \u003c/p\u003e\n  \u003ch2 align=\"center\"\u003eCVPR 2022\u003c/h2\u003e\n  \u003cdiv align=\"center\"\u003e\n    \u003cimg src=\"./assets/teaser.gif\" alt=\"Logo\" width=\"100%\"\u003e\n  \u003c/div\u003e\n\n  \u003cp align=\"center\"\u003e\n  \u003cbr\u003e\n    \u003ca href=\"https://pytorch.org/get-started/locally/\"\u003e\u003cimg alt=\"PyTorch\" src=\"https://img.shields.io/badge/PyTorch-ee4c2c?logo=pytorch\u0026logoColor=white\"\u003e\u003c/a\u003e\n    \u003ca href=\"https://pytorchlightning.ai/\"\u003e\u003cimg alt=\"Lightning\" src=\"https://img.shields.io/badge/-Lightning-792ee5?logo=pytorchlightning\u0026logoColor=white\"\u003e\u003c/a\u003e\n    \u003ca href='https://colab.research.google.com/drive/1-AWeWhPvCTBX0KfMtgtMk10uPU05ihoA?usp=sharing' style='padding-left: 0.5rem;'\u003e\u003cimg src='https://colab.research.google.com/assets/colab-badge.svg' alt='Google Colab'\u003e\u003c/a\u003e\n    \u003ca href=\"https://huggingface.co/spaces/Yuliang/ICON\"  style='padding-left: 0.5rem;'\u003e\u003cimg src='https://img.shields.io/badge/%F0%9F%A4%97%20Hugging%20Face-Spaces-orange'\u003e\u003c/a\u003e\u003cbr\u003e\u003c/br\u003e\n    \u003ca href='https://arxiv.org/abs/2112.09127'\u003e\n      \u003cimg src='https://img.shields.io/badge/Paper-PDF-green?style=for-the-badge\u0026logo=arXiv\u0026logoColor=green' alt='Paper PDF'\u003e\n    \u003c/a\u003e\n    \u003ca href='https://icon.is.tue.mpg.de/' style='padding-left: 0.5rem;'\u003e\n      \u003cimg src='https://img.shields.io/badge/ICON-Page-orange?style=for-the-badge\u0026logo=Google%20chrome\u0026logoColor=orange' alt='Project Page'\u003e\n    \u003ca href=\"https://discord.gg/Vqa7KBGRyk\"\u003e\u003cimg src=\"https://img.shields.io/discord/940240966844035082?color=7289DA\u0026labelColor=4a64bd\u0026logo=discord\u0026logoColor=white\u0026style=for-the-badge\"\u003e\u003c/a\u003e\n    \u003ca href=\"https://youtu.be/hZd6AYin2DE\"\u003e\u003cimg alt=\"youtube views\" title=\"Subscribe to my YouTube channel\" src=\"https://img.shields.io/youtube/views/hZd6AYin2DE?logo=youtube\u0026labelColor=ce4630\u0026style=for-the-badge\"/\u003e\u003c/a\u003e\n  \u003c/p\u003e\n\u003c/p\u003e\n\n\u003cbr /\u003e\n\u003cbr /\u003e\n\n## News :triangular_flag_on_post:\n\n- [2022/12/15] ICON belongs to the past, [ECON](https://github.com/YuliangXiu/ECON) is the future!\n- [2022/09/12] Apply [KeypointNeRF](https://markomih.github.io/KeypointNeRF/) on ICON, quantitative numbers in [evaluation](docs/evaluation.md#benchmark-train-on-thuman20-test-on-cape)\n- [2022/07/30] \u003ca href=\"https://huggingface.co/spaces/Yuliang/ICON\"  style='padding-left: 0.5rem;'\u003e\u003cimg src='https://img.shields.io/badge/%F0%9F%A4%97%20Hugging%20Face-Spaces-orange'\u003e\u003c/a\u003e \u003ca href='https://colab.research.google.com/drive/1-AWeWhPvCTBX0KfMtgtMk10uPU05ihoA?usp=sharing' style='padding-left: 0.5rem;'\u003e\u003cimg src='https://colab.research.google.com/assets/colab-badge.svg' alt='Google Colab'\u003e\u003c/a\u003e are both available\n- [2022/07/26] New cloth-refinement module is released, try `-loop_cloth`\n- [2022/06/13] ETH Zürich students from 3DV course create an add-on for [garment-extraction](docs/garment-extraction.md)\n- [2022/05/16] \u003ca href=\"https://github.com/Arthur151/ROMP\"\u003eBEV\u003c/a\u003e is supported as optional HPS by \u003ca href=\"https://scholar.google.com/citations?hl=en\u0026user=fkGxgrsAAAAJ\"\u003eYu Sun\u003c/a\u003e, see [commit #060e265](https://github.com/YuliangXiu/ICON/commit/060e265bd253c6a34e65c9d0a5288c6d7ffaf68e)\n- [2022/05/15] Training code is released, please check [Training Instruction](docs/training.md)\n- [2022/04/26] \u003ca href=\"https://github.com/Jeff-sjtu/HybrIK\"\u003eHybrIK (SMPL)\u003c/a\u003e is supported as optional HPS by \u003ca href=\"https://jeffli.site/\"\u003eJiefeng Li\u003c/a\u003e, see [commit #3663704](https://github.com/YuliangXiu/ICON/commit/36637046dcbb5667cdfbee3b9c91b934d4c5dd05)\n- [2022/03/05] \u003ca href=\"https://github.com/YadiraF/PIXIE\"\u003ePIXIE (SMPL-X)\u003c/a\u003e, \u003ca href=\"https://github.com/mkocabas/PARE\"\u003ePARE (SMPL)\u003c/a\u003e, \u003ca href=\"https://github.com/HongwenZhang/PyMAF\"\u003ePyMAF (SMPL)\u003c/a\u003e are all supported as optional HPS\n\n\u003cbr\u003e\n\n\u003c!-- TABLE OF CONTENTS --\u003e\n\u003cdetails open=\"open\" style='padding: 10px; border-radius:5px 30px 30px 5px; border-style: solid; border-width: 1px;'\u003e\n  \u003csummary\u003eTable of Contents\u003c/summary\u003e\n  \u003col\u003e\n    \u003cli\u003e\n      \u003ca href=\"#who-needs-ICON\"\u003eWho needs ICON\u003c/a\u003e\n    \u003c/li\u003e\n    \u003cli\u003e\n      \u003ca href=\"#instructions\"\u003eInstructions\u003c/a\u003e\n    \u003c/li\u003e\n    \u003cli\u003e\n      \u003ca href=\"#running-demo\"\u003eRunning Demo\u003c/a\u003e\n    \u003c/li\u003e\n    \u003cli\u003e\n      \u003ca href=\"#citation\"\u003eCitation\u003c/a\u003e\n    \u003c/li\u003e\n  \u003c/ol\u003e\n\u003c/details\u003e\n\u003cbr /\u003e\n\u003cbr /\u003e\n\n## Who needs ICON?\n\n- If you want to **Train \u0026 Evaluate** on **PIFu / PaMIR / ICON** using your own data, please check [dataset.md](./docs/dataset.md) to prepare dataset, [training.md](./docs/training.md) for training, and [evaluation.md](./docs/evaluation.md) for benchmark evaluation.\n\n- Given a raw RGB image, you could get:\n  - image (png):\n    - segmented human RGB\n    - normal maps of body and cloth\n    - pixel-aligned normal-RGB overlap\n  - mesh (obj):\n    - SMPL-(X) body from _PyMAF, PIXIE, PARE, HybrIK, BEV_\n    - 3D clothed human reconstruction\n    - 3D garments (requires 2D mask)\n  - video (mp4):\n    - self-rotated clothed human\n\n|    ![Intermediate Results](assets/intermediate_results.png)     |\n| :-------------------------------------------------------------: |\n|                  _ICON's intermediate results_                  |\n|         ![Iterative Refinement](assets/refinement.gif)          |\n|                  _ICON's SMPL Pose Refinement_                  |\n|             _![Final Results](assets/overlap.gif)_              |\n| _Image -- overlapped normal prediction -- ICON -- refined ICON_ |\n|                ![3D Garment](assets/garment.gif)                |\n|         _3D Garment extracted from ICON using 2D mask_          |\n\n\u003cbr\u003e\n\n## Instructions\n\n- See [docs/installation.md](docs/installation.md) to install all the required packages and setup the models\n- See [docs/dataset.md](docs/dataset.md) to synthesize the train/val/test dataset from THuman2.0\n- See [docs/training.md](docs/training.md) to train your own model using THuman2.0\n- See [docs/evaluation.md](docs/evaluation.md) to benchmark trained models on CAPE testset\n- Add-on: [Garment Extraction from Fashion Images](docs/garment-extraction.md), supported by ETH Zürich students as 3DV course project.\n\n\u003cbr\u003e\n\n## Running Demo\n\n```bash\ncd ICON\n\n# model_type:\n#   \"pifu\"            reimplemented PIFu\n#   \"pamir\"           reimplemented PaMIR\n#   \"icon-filter\"     ICON w/ global encoder (continous local wrinkles)\n#   \"icon-nofilter\"   ICON w/o global encoder (correct global pose)\n#   \"icon-keypoint\"   ICON w/ relative-spatial encoding (insight from KeypointNeRF)\n\npython -m apps.infer -cfg ./configs/icon-filter.yaml -gpu 0 -in_dir ./examples -out_dir ./results -export_video -loop_smpl 100 -loop_cloth 200 -hps_type pixie\n\n```\n\n## More Qualitative Results\n\n|              ![Comparison](assets/compare.gif)               |\n| :----------------------------------------------------------: |\n|       _Comparison with other state-of-the-art methods_       |\n|              ![extreme](assets/normal-pred.png)              |\n| _Predicted normals on in-the-wild images with extreme poses_ |\n\n\u003cbr/\u003e\n\u003cbr/\u003e\n\n## Citation\n\n```bibtex\n@inproceedings{xiu2022icon,\n  title     = {{ICON}: {I}mplicit {C}lothed humans {O}btained from {N}ormals},\n  author    = {Xiu, Yuliang and Yang, Jinlong and Tzionas, Dimitrios and Black, Michael J.},\n  booktitle = {Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)},\n  month     = {June},\n  year      = {2022},\n  pages     = {13296-13306}\n}\n```\n\n## Acknowledgments\n\nWe thank [Yao Feng](https://ps.is.mpg.de/person/yfeng), [Soubhik Sanyal](https://ps.is.mpg.de/person/ssanyal), [Qianli Ma](https://ps.is.mpg.de/person/qma), [Xu Chen](https://ait.ethz.ch/people/xu/), [Hongwei Yi](https://ps.is.mpg.de/person/hyi), [Chun-Hao Paul Huang](https://ps.is.mpg.de/person/chuang2), and [Weiyang Liu](https://wyliu.com/) for their feedback and discussions, [Tsvetelina Alexiadis](https://ps.is.mpg.de/person/talexiadis) for her help with the AMT perceptual study, [Taylor McConnell](https://ps.is.mpg.de/person/tmcconnell) for her voice over, [Benjamin Pellkofer](https://is.mpg.de/person/bpellkofer) for webpage, and [Yuanlu Xu](https://web.cs.ucla.edu/~yuanluxu/)'s help in comparing with ARCH and ARCH++.\n\nSpecial thanks to [Vassilis Choutas](https://ps.is.mpg.de/person/vchoutas) for sharing the code of [bvh-distance-queries](https://github.com/YuliangXiu/bvh-distance-queries)\n\nHere are some great resources we benefit from:\n\n- [MonoPortDataset](https://github.com/Project-Splinter/MonoPortDataset) for Data Processing\n- [PaMIR](https://github.com/ZhengZerong/PaMIR), [PIFu](https://github.com/shunsukesaito/PIFu), [PIFuHD](https://github.com/facebookresearch/pifuhd), and [MonoPort](https://github.com/Project-Splinter/MonoPort) for Benchmark\n- [SCANimate](https://github.com/shunsukesaito/SCANimate) and [AIST++](https://github.com/google/aistplusplus_api) for Animation\n- [rembg](https://github.com/danielgatis/rembg) for Human Segmentation\n- [PyTorch-NICP](https://github.com/wuhaozhe/pytorch-nicp) for normal-based non-rigid refinement\n- [smplx](https://github.com/vchoutas/smplx), [PARE](https://github.com/mkocabas/PARE), [PyMAF](https://github.com/HongwenZhang/PyMAF), [PIXIE](https://github.com/YadiraF/PIXIE), [BEV](https://github.com/Arthur151/ROMP), and [HybrIK](https://github.com/Jeff-sjtu/HybrIK) for Human Pose \u0026 Shape Estimation\n- [CAPE](https://github.com/qianlim/CAPE) and [THuman](https://github.com/ZhengZerong/DeepHuman/tree/master/THUmanDataset) for Dataset\n- [PyTorch3D](https://github.com/facebookresearch/pytorch3d) for Differential Rendering\n\nSome images used in the qualitative examples come from [pinterest.com](https://www.pinterest.com/).\n\nThis project has received funding from the European Union’s Horizon 2020 research and innovation programme under the Marie Skłodowska-Curie grant agreement No.860768 ([CLIPE Project](https://www.clipe-itn.eu)).\n\n## Contributors\n\nKudos to all of our amazing contributors! ICON thrives through open-source. In that spirit, we welcome all kinds of contributions from the community.\n\n\u003ca href=\"https://github.com/yuliangxiu/ICON/graphs/contributors\"\u003e\n  \u003cimg src=\"https://contrib.rocks/image?repo=yuliangxiu/ICON\" /\u003e\n\u003c/a\u003e\n\n_Contributor avatars are randomly shuffled._\n\n---\n\n\u003cbr\u003e\n\n## License\n\nThis code and model are available for non-commercial scientific research purposes as defined in the [LICENSE](LICENSE) file. By downloading and using the code and model you agree to the terms in the [LICENSE](LICENSE).\n\n## Disclosure\n\nMJB has received research gift funds from Adobe, Intel, Nvidia, Meta/Facebook, and Amazon. MJB has financial interests in Amazon, Datagen Technologies, and Meshcapade GmbH. While MJB was a part-time employee of Amazon during this project, his research was performed solely at, and funded solely by, the Max Planck Society.\n\n## Contact\n\nFor more questions, please contact icon@tue.mpg.de\n\nFor commercial licensing, please contact ps-licensing@tue.mpg.de\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fyuliangxiu%2Ficon","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fyuliangxiu%2Ficon","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fyuliangxiu%2Ficon/lists"}