{"id":20503107,"url":"https://github.com/idea-research/osx","last_synced_at":"2025-04-12T20:45:37.989Z","repository":{"id":151421209,"uuid":"620198494","full_name":"IDEA-Research/OSX","owner":"IDEA-Research","description":"[CVPR 2023] Official implementation of the paper \"One-Stage 3D Whole-Body Mesh Recovery with Component Aware Transformer\"","archived":false,"fork":false,"pushed_at":"2024-08-26T07:47:17.000Z","size":12532,"stargazers_count":707,"open_issues_count":32,"forks_count":59,"subscribers_count":15,"default_branch":"main","last_synced_at":"2025-04-03T23:11:57.330Z","etag":null,"topics":["3d-body-recovery","cvpr2023","human-pose-estimation","smpl-model","smplx","whole-body-pose-estimation"],"latest_commit_sha":null,"homepage":"https://osx-ubody.github.io/","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/IDEA-Research.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2023-03-28T08:08:43.000Z","updated_at":"2025-04-02T08:55:15.000Z","dependencies_parsed_at":null,"dependency_job_id":"a9a9cdda-2bbf-45ae-ac4e-6c938ba74471","html_url":"https://github.com/IDEA-Research/OSX","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/IDEA-Research%2FOSX","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/IDEA-Research%2FOSX/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/IDEA-Research%2FOSX/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/IDEA-Research%2FOSX/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/IDEA-Research","download_url":"https://codeload.github.com/IDEA-Research/OSX/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248631687,"owners_count":21136556,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["3d-body-recovery","cvpr2023","human-pose-estimation","smpl-model","smplx","whole-body-pose-estimation"],"created_at":"2024-11-15T19:29:15.875Z","updated_at":"2025-04-12T20:45:37.963Z","avatar_url":"https://github.com/IDEA-Research.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# **One-Stage 3D Whole-Body Mesh Recovery with Component Aware Transformer**\n### [Project Page](https://osx-ubody.github.io/) | [Video](https://www.youtube.com/watch?v=s0cG3OVXQUo\u0026t=2s) | [Paper](http://arxiv.org/abs/2303.16160) | [Data](https://docs.google.com/forms/d/e/1FAIpQLSehgBP7wdn_XznGAM2AiJPiPLTqXXHw5uX9l7qeQ1Dh9HoO_A/viewform)\n#### Authors\n\n[Jing Lin](https://jinglin7.github.io), [Ailing Zeng](https://ailingzeng.site/), [Haoqian Wang](https://www.sigs.tsinghua.edu.cn/whq_en/main.htm), [Lei Zhang](https://www.leizhang.org/), [Yu Li](https://yu-li.github.io/)\n\n\u003cp align=\"middle\"\u003e\n\u003cimg src=\"assets/demo_video.gif\" width=\"1000\"\u003e\n\u003cbr\u003e\n\u003cem\u003eThe proposed UBody dataset\u003c/em\u003e\n\u003c/p\u003e\n\n\n\n#### News\n- **2024.08.26 :** Update the implementation of the re-projection from SMPL-X to whole-body 2d keypoints (e.g., to align the 3D-to-2D keypoints), please check **3. Quick demo [Update information]**, Thanks to [Yuhang Yang](https://github.com/yyvhang).\n- **2023.10.12 :** UBody is now supported in [MMPose](https://github.com/open-mmlab/mmpose). Please feel free to use it.  🌟 \n- **2023.07.28 :** UBody can boost 2D whole-body pose estimation and controllable image generation, especially for in-the-wild hand keypoint detection. The training and test code and pre-trained models are released. See [details](https://github.com/IDEA-Research/DWPose). 🥳\n- **2023.05.03 :** UBody-V1 is released. We'll release UBody-V2 later, which have manually annotated bboxes. :man_dancing:\n- **2023.04.17 :** We fix bug of rendering in A100/V100  and support yolov5 as a person detector in demo.py. :rocket: \n- **2023.04.15 :** We merge OSX into [Grounded-SAM](https://github.com/IDEA-Research/Grounded-Segment-Anything)  and support promptable 3D whole-body mesh recovery. 🔥  \n\n\n\u003cp align=\"middle\"\u003e\n\u003cimg src=\"assets/grouned_sam_osx_demo.gif\" width=\"1000\"\u003e\n\u003cbr\u003e\n\u003cem\u003e Demo of Grounded-SAM-OSX.\u003c/em\u003e\n\u003c/p\u003e\n\n\n| ![space-1.jpg](./assets/grounded_sam_osx_output1.jpg) |\n| :---------------------------------------------------: |\n|             *A person with pink clothes*              |\n\n| ![space-1.jpg](./assets/grounded_sam_osx_output2.jpg) |\n| :---------------------------------------------------: |\n|               *A man with a sunglasses*               |\n\n## 1. Introduction  \n\nThis repo is official **[PyTorch](https://pytorch.org)** implementation of [One-Stage 3D Whole-Body Mesh Recovery with Component Aware Transformer (CVPR2023)](https://osx-ubody.github.io/). We propose the first one-stage whole-body mesh recovery method (OSX) and build a large-scale upper-body dataset (UBody). It is the top-1 method on [AGORA benchmark](https://agora-evaluation.is.tuebingen.mpg.de/) SMPL-X Leaderboard (dated March 2023).\n\n## 2. Create Environment  \n\n- [PyTorch \u003e= 1.7](https://pytorch.org/) + [CUDA](https://developer.nvidia.com/cuda-downloads)\n\n  Recommend to install by:\n\n  ```shell\n  pip install torch==1.11.0+cu113 torchvision==0.12.0+cu113 torchaudio==0.11.0 --extra-index-url https://download.pytorch.org/whl/cu113\n  ```\n\n- Python packages:\n\n  ```shell\n  bash install.sh\n  ```\n\n## 3. Quick demo  \n\n* Download the pre-trained OSX from [here](https://drive.google.com/drive/folders/1x7MZbB6eAlrq5PKC9MaeIm4GqkBpokow?usp=share_link).\n* Prepare pre-trained snapshot at `pretrained_models` folder.\n* Prepare `human_model_files` folder following below `Directory` part and place it at `common/utils/human_model_files`.\n* Go to `demo` folders, and run `python demo.py --gpu 0 --img_path IMG_PATH --output_folder OUTPUT_FOLDER `. Please replace `IMG_PATH` and `OUTPUT_FOLDRE` with your own image path and saving folder. For a more efficient inference, you can add `--decoder_setting wo_decoder --pretrained_model_path ../pretrained_models/osx_l_wo_decoder.pth.tar` to use the encoder-only version OSX.\n* [**Update information**] The inference code will output the projected 2d kpts with shape (137,2), please refer to [here](https://github.com/IDEA-Research/OSX/blob/118cf97fb1f144930bf93d88794b525d579b2d0c/common/utils/human_models.py#L71) for details of these 137 key points. The SMPLX version we use has 144 joints, please refer to this [line](https://github.com/IDEA-Research/OSX/blob/118cf97fb1f144930bf93d88794b525d579b2d0c/main/OSX.py#L72). if you want to use COCO format key points, please refer to [here](https://github.com/IDEA-Research/OSX/blob/118cf97fb1f144930bf93d88794b525d579b2d0c/data/MSCOCO/MSCOCO.py#L41). Note: the key points projected onto the image may be misaligned with humans, this is likely due to inaccurate boxes provided by detection models. It is recommended to use more advanced detection models or manually set bounding boxes.\n* If you run this code in ssh environment without display device, do follow:\n```\n1、Install oemesa follow https://pyrender.readthedocs.io/en/latest/install/\n2、Reinstall the specific pyopengl fork: https://github.com/mmatl/pyopengl\n3、Set opengl's backend to egl or osmesa via os.environ[\"PYOPENGL_PLATFORM\"] = \"egl\"\n```\n\n## 4. Directory  \n### (1) Root  \nThe `${ROOT}` is described as below.  \n```  \n${ROOT}  \n|-- data  \n|-- dataset\n|-- demo\n|-- main  \n|-- pretrained_models\n|-- tool\n|-- output  \n|-- common\n|   |-- utils\n|   |   |-- human_model_files\n|   |   |   |-- smpl\n|   |   |   |   |-- SMPL_NEUTRAL.pkl\n|   |   |   |   |-- SMPL_MALE.pkl\n|   |   |   |   |-- SMPL_FEMALE.pkl\n|   |   |   |-- smplx\n|   |   |   |   |-- MANO_SMPLX_vertex_ids.pkl\n|   |   |   |   |-- SMPL-X__FLAME_vertex_ids.npy\n|   |   |   |   |-- SMPLX_NEUTRAL.pkl\n|   |   |   |   |-- SMPLX_to_J14.pkl\n|   |   |   |   |-- SMPLX_NEUTRAL.npz\n|   |   |   |   |-- SMPLX_MALE.npz\n|   |   |   |   |-- SMPLX_FEMALE.npz\n|   |   |   |-- mano\n|   |   |   |   |-- MANO_LEFT.pkl\n|   |   |   |   |-- MANO_RIGHT.pkl\n|   |   |   |-- flame\n|   |   |   |   |-- flame_dynamic_embedding.npy\n|   |   |   |   |-- flame_static_embedding.pkl\n|   |   |   |   |-- FLAME_NEUTRAL.pkl\n```\n* `data` contains data loading codes.  \n* `dataset` contains soft links to images and annotations directories.  \n* `pretrained_models` contains pretrained models.  \n* `demo` contains demo codes.\n* `main` contains high-level codes for training or testing the network.  \n* `tool` contains pre-processing codes of AGORA and pytorch model editing codes.\n* `output` contains log, trained models, visualized outputs, and test result.  \n* `common` contains kernel codes for Hand4Whole.  \n* `human_model_files` contains `smpl`, `smplx`, `mano`, and `flame` 3D model files. Download the files from [[smpl]](https://smpl.is.tue.mpg.de/) [[smplx]](https://smpl-x.is.tue.mpg.de/) [[SMPLX_to_J14.pkl]](https://github.com/vchoutas/expose#preparing-the-data) [[mano]](https://mano.is.tue.mpg.de/) [[flame]](https://flame.is.tue.mpg.de/). We provide the download links for each file [here](https://github.com/IDEA-Research/OSX/tree/main/common/utils/human_model_files).\n### (2) Data  \nYou need to follow directory structure of the `dataset` as below.  \n```  \n${ROOT}  \n|-- dataset  \n|   |-- AGORA\n|   |   |-- data\n|   |   |   |-- AGORA_train.json\n|   |   |   |-- AGORA_validation.json\n|   |   |   |-- AGORA_test_bbox.json\n|   |   |   |-- 1280x720\n|   |   |   |-- 3840x2160\n|   |-- EHF\n|   |   |-- data\n|   |   |   |-- EHF.json\n|   |-- Human36M  \n|   |   |-- images  \n|   |   |-- annotations  \n|   |-- MPII\n|   |   |-- data\n|   |   |   |-- images\n|   |   |   |-- annotations\n|   |-- MPI_INF_3DHP\n|   |   |-- data\n|   |   |   |-- images_1k\n|   |   |   |-- MPI-INF-3DHP_1k.json\n|   |   |   |-- MPI-INF-3DHP_camera_1k.json\n|   |   |   |-- MPI-INF-3DHP_joint_3d.json\n|   |   |   |-- MPI-INF-3DHP_SMPL_NeuralAnnot.json\n|   |-- MSCOCO  \n|   |   |-- images  \n|   |   |   |-- train2017  \n|   |   |   |-- val2017  \n|   |   |-- annotations \n|   |-- PW3D\n|   |   |-- data\n|   |   |   |-- 3DPW_train.json\n|   |   |   |-- 3DPW_validation.json\n|   |   |   |-- 3DPW_test.json\n|   |   |-- imageFiles\n|   |-- UBody\n|   |   |-- images\n|   |   |-- videos\n|   |   |-- annotations\n|   |   |-- splits\n|   |   |   |-- inter_scene_test_list.npy\n|   |   |   |-- intra_scene_test_list.npy\n```\n\n* Download AGORA parsed data [[data](https://drive.google.com/drive/folders/18CWsL28e8v50rqEbYMoU4yHHWoGJdpg_?usp=sharing)][[parsing codes](tool/AGORA)]\n* Download EHF parsed data [[data](https://drive.google.com/file/d/1Ji2PuB2HYQzRpQ016LwSSLguFMezQqOI/view?usp=sharing)]\n* Download Human3.6M parsed data and SMPL-X parameters [[data](https://drive.google.com/drive/folders/1r0B9I3XxIIW_jsXjYinDpL6NFcxTZart?usp=sharing)][[SMPL-X parameters from NeuralAnnot](https://drive.google.com/drive/folders/19ifIQtAB3ll8d37-kerL1eQWp31mOwJM?usp=sharing)]\n* Download MPII parsed data and SMPL-X parameters [[data](https://drive.google.com/drive/folders/1rrL_RxhwQgwhq5BU1iIRPwl285B_KTpU?usp=sharing)][[SMPL-X parameters from NeuralAnnot](https://drive.google.com/file/d/1alkKvhkqQGqriKst83uS-kUG7v6SkM7W/view?usp=sharing)]\n* Download MPI-INF-3DHP parsed data and SMPL-X parameters [[data](https://drive.google.com/drive/folders/1wQbHEXPv-WH1sNOLwdfMVB7OWsiJkq2M?usp=sharing)][[SMPL-X parameters from NeuralAnnot](https://drive.google.com/file/d/1ADOJlaqaBDjZ3IEgrgLTQwNf6iHd-rGH/view?usp=sharing)]\n* Download MSCOCO data and SMPL-X parameters [[data](https://github.com/jin-s13/COCO-WholeBody)][[SMPL-X parameters](https://drive.google.com/file/d/1UVyfqrOtkbhI3MgpBYXd1YXbkD8aJtL9/view?usp=share_link)]\n* Download 3DPW parsed data [[data](https://drive.google.com/drive/folders/1HByTBsdg_A_o-d89qd55glTl44ya3dOs?usp=sharing)]\n* Download UBody dataset from [[data](https://docs.google.com/forms/d/e/1FAIpQLSehgBP7wdn_XznGAM2AiJPiPLTqXXHw5uX9l7qeQ1Dh9HoO_A/viewform)] and run the following commond to convert the videos into images:\n\n  ```\n  cd tool/UBody\n  python video2image.py\n  ```\n* All annotation files follow [MSCOCO format](http://cocodataset.org/#format-data). If you want to add your own dataset, you have to convert it to [MSCOCO format](http://cocodataset.org/#format-data).  \n  \n### (3) Output  \nYou need to follow the directory structure of the `output` folder as below.  \n```  \n${ROOT}  \n|-- output  \n|   |-- log  \n|   |-- model_dump  \n|   |-- result  \n|   |-- vis  \n```\n* Creating `output` folder as soft link form is recommended instead of folder form because it would take large storage capacity.  \n* `log` folder contains training log file.  \n* `model_dump` folder contains saved checkpoints for each epoch.  \n* `result` folder contains final estimation files generated in the testing stage.  \n* `vis` folder contains visualized results.  \n\n\n## 5. Training OSX\n#### (1) Download Pretrained Encoder\n\nDownload pretrained encoder `osx_vit_l.pth` and `osx_vit_b.pth` from [here](https://drive.google.com/drive/folders/1x7MZbB6eAlrq5PKC9MaeIm4GqkBpokow?usp=share_link) and place the pretrained model to `pretrained_models/`.\n\n#### (2) Setting1:  Train on MSCOCO, Human3.6m, MPII and Test on EHF and AGORA-val\n\nIn the `main` folder, run  \n```bash  \npython train.py --gpu 0,1,2,3 --lr 1e-4 --exp_name output/train_setting1 --end_epoch 14 --train_batch_size 16\n```\nAfter training, run the following command to evaluate your pretrained model on EHF and AGORA-val:\n\n```bash  \n# test on EHF\npython test.py --gpu 0,1,2,3 --exp_name output/train_setting1/ --pretrained_model_path ../output/train_setting1/model_dump/snapshot_13.pth.tar --testset EHF\n# test on AGORA-val\npython test.py --gpu 0,1,2,3 --exp_name output/train_setting1/ --pretrained_model_path ../output/train_setting1/model_dump/snapshot_13.pth.tar --testset AGORA\n```\nTo speed up, you can use a light-weight version OSX by change the encoder setting by adding `--encoder_setting osx_b` or change the decoder setting by adding `--decoder_setting wo_face_decoder`. We recommend adding `--decoder_setting wo_face_decoder` as it would obviously speed up and would not lead to significant performance decline. It takes about 20 hours to finish the training with one NVIDIA A100.\n#### (3) Setting2: Train on AGORA and Test on AGORA-test\n\nIn the `main` folder, run  \n\n```bash  \npython train.py --gpu 0,1,2,3 --lr 1e-4 --exp_name output/train_setting2 --end_epoch 140 --train_batch_size 16  --agora_benchmark --decoder_setting wo_decoder\n```\n\nAfter training, run the following command to evaluate your pretrained model on AGORA-test:\n\n```bash  \npython test.py --gpu 0,1,2,3 --exp_name output/train_setting2/ --pretrained_model_path ../output/train_setting2/model_dump/snapshot_139.pth.tar --testset AGORA --agora_benchmark --test_batch_size 64 --decoder_setting wo_decoder\n```\n\nThe reconstruction result will be saved at `output/train_setting2/result/`.\n\nYou can zip the `predictions` folder into `predictions.zip` and submit it to the [AGORA benchmark](https://agora-evaluation.is.tuebingen.mpg.de/) to obtain the evaluation metrics. \n\nYou can use a light-weight version OSX by adding `--encoder_setting osx_b`.\n\n#### (4) Setting3:  Train on MSCOCO, Human3.6m, MPII, UBody-Train and Test on UBody-val\n\nIn the `main` folder, run  \n\n```bash  \npython train.py --gpu 0,1,2,3 --lr 1e-4 --exp_name output/train_setting3 --train_batch_size 16  --ubody_benchmark --decoder_setting wo_decoder\n```\n\nAfter training, run the following command to evaluate your pretrained model on UBody-test:\n\n```bash  \npython test.py --gpu 0,1,2,3 --exp_name output/train_setting3/ --pretrained_model_path ../output/train_setting3/model_dump/snapshot_13.pth --testset UBody --test_batch_size 64 --decoder_setting wo_decoder \n```\n\nThe reconstruction result will be saved at `output/train_setting3/result/`.\n\n## 6. Testing OSX\n\n#### (1) Download Pretrained Models\n\nDownload pretrained models `osx_l.pth.tar` and `osx_l_agora.pth.tar` from [here](https://drive.google.com/drive/folders/1x7MZbB6eAlrq5PKC9MaeIm4GqkBpokow?usp=share_link) and place the pretrained model to `pretrained_models/`.\n\n#### (2) Test on EHF\n\nIn the `main` folder, run  \n\n```bash  \npython test.py --gpu 0,1,2,3 --exp_name output/test_setting1 --pretrained_model_path ../pretrained_models/osx_l.pth.tar --testset EHF\n```\n\n#### (3) Test on AGORA-val\n\nIn the `main` folder, run  \n\n```bash  \npython test.py --gpu 0,1,2,3 --exp_name output/test_setting1 --pretrained_model_path ../pretrained_models/osx_l.pth.tar --testset AGORA\n```\n\n#### (4) Test on AGORA-test\n\nIn the `main` folder, run  \n\n```bash  \npython test.py --gpu 0,1,2,3 --exp_name output/test_setting2  --pretrained_model_path ../pretrained_models/osx_l_agora.pth.tar --testset AGORA --agora_benchmark --test_batch_size 64\n```\n\nThe reconstruction result will be saved at `output/test_setting2/result/`.\n\nYou can zip the `predictions` folder into `predictions.zip` and submit it to the [AGORA benchmark](https://agora-evaluation.is.tuebingen.mpg.de/) to obtain the evaluation metrics. \n\n#### (5) Test on UBody-test\n\nIn the `main` folder, run  \n\n```bash  \npython test.py --gpu 0,1,2,3 --exp_name output/test_setting3  --pretrained_model_path ../pretrained_models/osx_l_wo_decoder.pth.tar --testset UBody --test_batch_size 64\n```\n\nThe reconstruction result will be saved at `output/test_setting3/result/`.\n\n## 7. Results\n\n### (1) AGORA test set\n\n\u003cimg src=\"./assets/agora_test.png\" alt=\"image-20230327202353903\" style=\"zoom: 33%;\" /\u003e\n\n### (2) AGORA-val, EHF, 3DPW\n\n\u003cimg src=\"./assets/ehf_3dpw.png\" alt=\"image-20230327202755593\" style=\"zoom:67%;\" /\u003e\n\n\u003cimg src=\"./assets/agora_val.png\" alt=\"image-20230327204220453\" style=\"zoom:67%;\" /\u003e\n\n\n\n### Troubleshoots\n\n* `RuntimeError: Subtraction, the '-' operator, with a bool tensor is not supported. If you are trying to invert a mask, use the '~' or 'logical_not()' operator instead.`: Go to [here](https://github.com/mks0601/I2L-MeshNet_RELEASE/issues/6#issuecomment-675152527)\n\n* `TypeError: startswith first arg must be bytes or a tuple of bytes, not str.`: Go to [here](https://github.com/mcfletch/pyopengl/issues/27). \n\n### Acknowledgement\n\nThis repo is mainly based on [Hand4Whole](https://github.com/mks0601/Hand4Whole_RELEASE). We thank the well-organized code and patient answers of [Gyeongsik Moon](https://mks0601.github.io/) in the issue!\n\n## Reference  \n\n```  \n@inproceedings{lin2023one,\n  title={One-Stage 3D Whole-Body Mesh Recovery with Component Aware Transformer},\n  author={Lin, Jing and Zeng, Ailing and Wang, Haoqian and Zhang, Lei and Li, Yu},\n  booktitle={Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition},\n  pages={21159--21168},\n  year={2023}\n}\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fidea-research%2Fosx","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fidea-research%2Fosx","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fidea-research%2Fosx/lists"}