{"id":27366404,"url":"https://github.com/hxwork/h2onet_pytorch","last_synced_at":"2025-04-13T05:56:58.117Z","repository":{"id":156271027,"uuid":"617054437","full_name":"hxwork/H2ONet_Pytorch","owner":"hxwork","description":"[CVPR 2023] H2ONet: Hand-Occlusion-and-Orientation-aware Network for Real-time 3D Hand Mesh Reconstruction, Pytorch implementation.","archived":false,"fork":false,"pushed_at":"2023-12-01T04:52:00.000Z","size":11144,"stargazers_count":62,"open_issues_count":3,"forks_count":2,"subscribers_count":4,"default_branch":"main","last_synced_at":"2025-04-13T05:56:51.881Z","etag":null,"topics":["3d-hand-reconstruction","deep-learning","pytorch"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/hxwork.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null}},"created_at":"2023-03-21T15:58:30.000Z","updated_at":"2025-04-01T14:00:13.000Z","dependencies_parsed_at":null,"dependency_job_id":"f3d361ee-0495-4444-9737-4d889662c8e8","html_url":"https://github.com/hxwork/H2ONet_Pytorch","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/hxwork%2FH2ONet_Pytorch","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/hxwork%2FH2ONet_Pytorch/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/hxwork%2FH2ONet_Pytorch/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/hxwork%2FH2ONet_Pytorch/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/hxwork","download_url":"https://codeload.github.com/hxwork/H2ONet_Pytorch/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248670453,"owners_count":21142901,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["3d-hand-reconstruction","deep-learning","pytorch"],"created_at":"2025-04-13T05:56:57.901Z","updated_at":"2025-04-13T05:56:58.107Z","avatar_url":"https://github.com/hxwork.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# [CVPR 2023] H2ONet: Hand-Occlusion-and-Orientation-aware Network for Real-time 3D Hand Mesh Reconstruction\n\n\u003ch4 align = \"center\"\u003eHao Xu\u003csup\u003e1,2\u003c/sup\u003e, Tianyu Wang\u003csup\u003e1\u003c/sup\u003e, Xiao Tang\u003csup\u003e1\u003c/sup\u003e, Chi-Wing Fu\u003csup\u003e1,2,3\u003c/sup\u003e\u003c/h4\u003e\n\u003ch4 align = \"center\"\u003e \u003csup\u003e1\u003c/sup\u003eDepartment of Computer Science and Engineering\u003c/center\u003e\u003c/h4\u003e\n\u003ch4 align = \"center\"\u003e \u003csup\u003e2\u003c/sup\u003eInstitute of Medical Intelligence and XR, \u003csup\u003e3\u003c/sup\u003eShun Hing Institute of Advanced Engineering\u003c/center\u003e\u003c/h4\u003e\n\u003ch4 align = \"center\"\u003e The Chinese University of Hong Kong\u003c/center\u003e\u003c/h4\u003e\n\nThis is the official implementation of our CVPR2023 paper [H2ONet](https://openaccess.thecvf.com/content/CVPR2023/papers/Xu_H2ONet_Hand-Occlusion-and-Orientation-Aware_Network_for_Real-Time_3D_Hand_Mesh_Reconstruction_CVPR_2023_paper.pdf).\n\nOur presentation video: [[Youtube](https://www.youtube.com/watch?v=JN-G8ePC3Mk)].\n\n## Our Poster\n\n![poster](./files/poster.png)\n\n## Todo List\n\n* [X] ~~Single-frame model code~~\n* [X] ~~Occlusion label preparation code for the DexYCB dataset~~\n* [X] ~~Occlusion label preparation code for the HO3D dataset~~\n* [X] ~~Multi-frame model code~~\n* [X] ~~Training config details~~\n* [X] ~~model checkpoints and evaluation code~~\n\n## Install\n\n* Environment\n  ```\n  conda create -n h2onet python=3.8\n  conda activate h2onet\n  ```\n* Requirements\n  ```\n  pip install -r requirements.txt\n  ```\n* Download the pre-trained weights of the backbone `densestack.pth` from Google drive or Baidu cloud, which are both provided by [MobRecon](https://github.com/SeanChenxy/HandMesh). After than, put it into the `checkpoints` folder.\n\n## Data Preparation\n\nWe evaluate different models on the DexYCB and HO3D datasets. The pre-processed ground truths are from [HandOccNet](https://github.com/namepllet/HandOccNet). Please follow its instruction to prepare the data and ground truths like this,\n\n```\n|-- data  \n|   |-- HO3D\n|   |   |-- train\n|   |   |   |-- ABF10\n|   |   |   |-- ......\n|   |   |-- evaluation\n|   |   |-- annotations\n|   |   |   |-- HO3D_train_data.json\n|   |   |   |-- HO3D_evaluation_data.json\n|   |-- DEX_YCB\n|   |   |-- 20200709-subject-01\n|   |   |-- ......\n|   |   |-- annotations\n|   |   |   |--DEX_YCB_s0_train_data.json\n|   |   |   |--DEX_YCB_s0_test_data.json\n```\n\n## Occlusion Label Preparation\n\nOur single-frame model does not need the occlusion prediction. To train the fingle-level occlusion classifier in our multi-frame model, we first prepare the occlusion label from the provided ground truths. Please find more details in `occ_label_preparation`.\n\n## Training\n\nWe adopt a two-stage training strategy: (i) hand shape reconstruction at canonical pose; and (ii) hand orientation regression based on the model trained in (i).\n\nFor training our single-frame model on the DexYCB dataset,\n\n```\n# stage 1\npython3 train.py --model_dir=./experiment/single_frame_dexycb/stage_1\n# stage 2\npython3 train.py --model_dir=./experiment/single_frame_dexycb/stage_2 --resume=./experiment/single_frame_dexycb/stage_1/test_model_best.pth -ow\n```\n\nFor training our multi-frame model on the DexYCB dataset, we first load the pre-trained single-frame model (stage 1).\n\n```\n# stage 1\npython3 train.py --model_dir=./experiment/multi_frame_dexycb/stage_1 --resume=./experiment/single_frame_dexycb/stage_1/test_model_best.pth -ow\n# stage 2\npython3 train.py --model_dir=./experiment/multi_frame_dexycb/stage_2 --resume=./experiment/multi_frame_dexycb/stage_1/test_model_best.pth -ow\n```\n\nFor training our single-frame and multi-frame models on the HO3D-v2 dataset, we follow the same approach and change the dataset name in the scripts. Note that due to the limited scale of the HO3D-v2 dataset, for training our model in stage 2, we first pre-train it on the DexYCB dataset for a few epochs (\u003c5) to avoid unstable training (e.g., NaN in training).\n\n## Testing\n\nTo test our pre-trained model,\n\n```\npython test.py --model_dir=./experiment/multi_frame_dexycb/stage_2 --resume=./experiment/multi_frame_dexycb/stage_2/test_model_best.pth\n```\n\nWe provide the pre-trained multi-frame model on the DexYCB dataset. [[checkpoint](https://drive.google.com/file/d/11VLYLr5bjqCwUgqdihte1hoIcIrkTHt8/view?usp=sharing)]\n\n## Citation\n\n```\n@InProceedings{Xu_2023_CVPR,\n    author    = {Xu, Hao and Wang, Tianyu and Tang, Xiao and Fu, Chi-Wing},\n    title     = {H2ONet: Hand-Occlusion-and-Orientation-Aware Network for Real-Time 3D Hand Mesh Reconstruction},\n    booktitle = {Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)},\n    month     = {June},\n    year      = {2023},\n    pages     = {17048-17058}\n}\n```\n\n## Acknowledgments\n\nIn this project we use (parts of) the official implementations of the following works:\n\n* [MobRecon](https://github.com/SeanChenxy/HandMesh) (Lightweight pipeline)\n* [HandOccNet](https://github.com/namepllet/HandOccNet) (HO3D and DexYCB datasets processing)\n\nWe thank the respective authors for open sourcing their methods.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fhxwork%2Fh2onet_pytorch","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fhxwork%2Fh2onet_pytorch","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fhxwork%2Fh2onet_pytorch/lists"}