{"id":13958387,"url":"https://github.com/jby1993/SelfReconCode","last_synced_at":"2025-07-20T23:31:21.427Z","repository":{"id":37298813,"uuid":"474694039","full_name":"jby1993/SelfReconCode","owner":"jby1993","description":"This repository contains a pytorch implementation of \"SelfRecon: Self Reconstruction Your Digital Avatar from Monocular Video (CVPR 2022, Oral)\".","archived":false,"fork":false,"pushed_at":"2022-10-17T15:06:03.000Z","size":4727,"stargazers_count":403,"open_issues_count":23,"forks_count":44,"subscribers_count":22,"default_branch":"main","last_synced_at":"2024-08-09T13:18:44.416Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/jby1993.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2022-03-27T16:27:21.000Z","updated_at":"2024-07-22T17:02:39.000Z","dependencies_parsed_at":"2022-07-14T06:40:37.705Z","dependency_job_id":null,"html_url":"https://github.com/jby1993/SelfReconCode","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jby1993%2FSelfReconCode","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jby1993%2FSelfReconCode/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jby1993%2FSelfReconCode/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jby1993%2FSelfReconCode/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/jby1993","download_url":"https://codeload.github.com/jby1993/SelfReconCode/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":226845009,"owners_count":17691140,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-08-08T13:01:31.318Z","updated_at":"2024-11-28T01:31:56.687Z","avatar_url":"https://github.com/jby1993.png","language":"Python","funding_links":[],"categories":["人像\\姿势\\3D人脸"],"sub_categories":["网络服务_其他"],"readme":"# SelfRecon: Self Reconstruction Your Digital Avatar from Monocular Video\n### | [Project Page](https://jby1993.github.io/SelfRecon/) | [Paper](https://arxiv.org/abs/2201.12792) | \nThis repository contains a pytorch implementation of \"[SelfRecon: Self Reconstruction Your Digital Avatar from Monocular Video (CVPR 2022, Oral)](https://arxiv.org/abs/2201.12792)\".\u003cbr/\u003e\nAuthors: Boyi Jiang, [Yang Hong](https://crishy1995.github.io/), [Hujun Bao](http://www.cad.zju.edu.cn/home/bao/), [Juyong Zhang](http://staff.ustc.edu.cn/~juyong/).\n\nThis code is protected under patent, and it can be only used for research purposes. For commercial uses, please send email to \u003cjiangboyi@idr.ai\u003e. \n![](asset/avatars.png)\n## Requirements\n- Python 3\n- Pytorch3d (0.4.0, some compatibility issues may occur in higher versions of pytorch3d) \n\nNote: A GTX 3090 is recommended to run SelfRecon, make sure enough GPU memory if using other cards.\n## Install\n```bash\nconda env create -f environment.yml\nconda activate SelfRecon\nbash install.sh\n```\n\n    \nIt is recommended to install pytorch3d 0.4.0 from source. \n```bash\nwget -O pytorch3d-0.4.0.zip https://github.com/facebookresearch/pytorch3d/archive/refs/tags/v0.4.0.zip\nunzip pytorch3d-0.4.0.zip\ncd pytorch3d-0.4.0 \u0026\u0026 python setup.py install \u0026\u0026 cd ..\n```\n\nTo download the [SMPL](https://smpl.is.tue.mpg.de/) models from [here](https://mailustceducn-my.sharepoint.com/:f:/g/personal/jby1993_mail_ustc_edu_cn/EqosuuD2slZCuZeVI2h4RiABguiaB4HkUBusnn_0qEhWjQ?e=c6r4KS) and move pkls to smpl_pytorch/model.\n\n## Run on PeopleSnapshot Dataset\nThe preprocessing of PeopleSnapshot is described here. If you want to optimize your own data, you can run [VideoAvatar](https://graphics.tu-bs.de/people-snapshot) to get the initial SMPL estimation, then follow the preprocess. Or, you can use your own SMPL initialization and normal prediction method then use SelfRecon to reconstruct.\n### Preprocess\nDownload the [Dataset](https://graphics.tu-bs.de/people-snapshot) and unzip it to some ROOT. Run the following code to extract data for female-3-casual, for example.\n```bash\npython people_snapshot_process.py --root $ROOT/people_snapshot_public/female-3-casual --save_root $ROOT/female-3-casual\n```\n### Extract Normals\nTo enable our normal optimization, you have to install [PIFuHD](https://shunsukesaito.github.io/PIFuHD/) and [Lightweight Openpose](https://github.com/Daniil-Osokin/lightweight-human-pose-estimation.pytorch) in your $ROOT1 and $ROOT2 first. Then copy generate_normals.py and generate_boxs.py to $ROOT1 and $ROOT2 seperately, and run the following code to extract normals before running SelfRecon:\n```bash\ncd $ROOT2\npython generate_boxs.py --data $ROOT/female-3-casual/imgs\ncd $ROOT1\npython generate_normals.py --imgpath $ROOT/female-3-casual/imgs\n```\nThen, run SelfRecon with the following code, this may take one day to finish:\n```bash\nCUDA_VISIBLE_DEVICES=0 python train.py --gpu-ids 0 --conf config.conf --data $ROOT/female-3-casual --save-folder result\n```\nThe results locate in $ROOT/female-3-casual/result\n\n\n\n## Inference\nRun the following code to generate rendered meshes and images.\n```bash\nCUDA_VISIBLE_DEVICES=0 python infer.py --gpu-ids 0 --rec-root $ROOT/female-3-casual/result/ --C\n```\n\n## Texture\nThis repo provides a script to utilize [VideoAvatar](https://graphics.tu-bs.de/people-snapshot) to extract the texture for the reconstructions of SelfRecon. \n\nFirst, You need to install VideoAvatar and copy texture_mesh_extract.py to its repository path.\n\nThen, after performing inference for $ROOT/female-3-casual/result, you need to simplify and parameterize the template mesh tmp.ply yourself, then save the result mesh as $ROOT/female-3-casual/result/template/uvmap.obj. And run the following code to generate the data for texture extraction:\n``` bash\nCUDA_VISIBLE_DEVICES=0 python texture_mesh_prepare.py --gpu-ids 0 --num 120 --rec-root $ROOT/female-3-casual/result/\n```\n\nFinally, go to VideoAvatar path, and run the following code to extract texture:\n```bash\nCUDA_VISIBLE_DEVICES=0 python texture_mesh_extract.py --tmp-root $ROOT/female-3-casual/result/template\n```\n## Dataset\nThe processed dataset, our trained models, some reconstruction results and textured meshes can be downloaded via the [link](https://mailustceducn-my.sharepoint.com/:f:/g/personal/jby1993_mail_ustc_edu_cn/EsSsDtUBYJVLvY21Wk2K_gQBuOWgCKFGGxr2xqheS-0ORw?e=Rda2HX), you can download and unzip some smartphone data, like CHH_female.zip, in $ROOT, and train directly with:\n```bash\nCUDA_VISIBLE_DEVICES=0 python train.py --gpu-ids 0 --conf config.conf --data $ROOT/CHH_female --save-folder result\n```\nAnd you can unzip CHH_female_model.zip in $ROOT/CHH_female and run:\n\n```bash\nCUDA_VISIBLE_DEVICES=0 python infer.py --gpu-ids 0 --rec-root $ROOT/CHH_female/trained/ --C\n```\nto check the results of our trained model in $ROOT/CHH_female/trained.\n\nNote: If you want to train the synthetic data, config_loose.conf is prefered.\n## Acknowledgement\n\nHere are some great resources we benefit or utilize from:\n- [MarchingCubes_CppCuda](https://github.com/WanquanF/MarchingCubes_CppCuda) supplies the GPU MC module (MCGpu)\n- [MonoPort](https://github.com/Project-Splinter/MonoPort) and [ImplicitSegCUDA](https://github.com/Project-Splinter/ImplicitSegCUDA/tree/master/implicit_seg/cuda) constructed our Marching Cubes accelerating (MCAcc)\n- [VideoAvatar](https://graphics.tu-bs.de/people-snapshot) for SMPL initialization and texture extraction\n- [SMPL](https://smpl.is.tue.mpg.de/) for Parametric Body Representation\n- [IDR](https://github.com/lioryariv/idr) for Neural Implicit Reconstruction\n- [PyTorch3D](https://github.com/facebookresearch/pytorch3d) for Differential Explicit Rendering\n\nThis research was supported by National Natural Science Foundation of China (No. 62122071), the Youth Innovation Promotion Association CAS (No. 2018495), ``the Fundamental Research Funds for the Central Universities''(No. WK3470000021).\n\n\n\n  \u003c!-- citing --\u003e\n  \u003cdiv class=\"container\"\u003e\n      \u003cdiv class=\"row \"\u003e\n          \u003cdiv class=\"col-12\"\u003e\n              \u003ch2\u003eCitation\u003c/h2\u003e\n              \u003cpre style=\"background-color: #e9eeef;padding: 1.25em 1.5em\"\u003e\u003ccode\u003e@inproceedings{jiang2022selfrecon,\n  author    = {Boyi Jiang and Yang Hong and Hujun Bao and Juyong Zhang},\n  title     = {SelfRecon: Self Reconstruction Your Digital Avatar from Monocular Video},\n  booktitle = {{IEEE/CVF} Conference on Computer Vision and Pattern Recognition (CVPR)},\n  year      = {2022}\n}\u003c/code\u003e\u003c/pre\u003e\n        \u003c/div\u003e\n      \u003c/div\u003e\n  \u003c/div\u003e\n\n\n## License\nFor non-commercial research use only.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjby1993%2FSelfReconCode","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fjby1993%2FSelfReconCode","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjby1993%2FSelfReconCode/lists"}