{"id":13449290,"url":"https://github.com/statho/ScoreHMR","last_synced_at":"2025-03-22T22:32:46.009Z","repository":{"id":227768068,"uuid":"772336002","full_name":"statho/ScoreHMR","owner":"statho","description":"ScoreHMR: Score-Guided Diffusion for 3D Human Recovery (CVPR 2024)","archived":false,"fork":false,"pushed_at":"2024-05-24T23:47:38.000Z","size":18869,"stargazers_count":372,"open_issues_count":9,"forks_count":25,"subscribers_count":13,"default_branch":"master","last_synced_at":"2024-08-01T06:21:27.931Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/statho.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE.md","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null}},"created_at":"2024-03-15T01:47:27.000Z","updated_at":"2024-07-30T03:36:05.000Z","dependencies_parsed_at":"2024-03-17T23:45:04.444Z","dependency_job_id":null,"html_url":"https://github.com/statho/ScoreHMR","commit_stats":null,"previous_names":["statho/scorehmr"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/statho%2FScoreHMR","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/statho%2FScoreHMR/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/statho%2FScoreHMR/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/statho%2FScoreHMR/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/statho","download_url":"https://codeload.github.com/statho/ScoreHMR/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":221840629,"owners_count":16889834,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-07-31T06:00:35.103Z","updated_at":"2024-10-28T14:30:41.989Z","avatar_url":"https://github.com/statho.png","language":"Python","funding_links":[],"categories":["2D Diffusion with Pretraining","Python"],"sub_categories":["Human and Animal"],"readme":"# ScoreHMR: Score-Guided Human Mesh Recovery\n\nCode repository for the paper:\n**Score-Guided Diffusion for 3D Human Recovery**\\\n[Anastasis Stathopoulos](https://statho.github.io/), [Ligong Han](https://phymhan.github.io/), [Dimitris Metaxas](https://people.cs.rutgers.edu/~dnm/)\n\n[![arXiv](https://img.shields.io/badge/arXiv-2305.20091-00ff00.svg)](https://arxiv.org/abs/2403.09623)  [![Website shields.io](https://img.shields.io/website-up-down-green-red/http/shields.io.svg)](https://statho.github.io/ScoreHMR/)  [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/drive/1WdEWaOyG0XHHZrlg2uX9XwECDKBX5dCn)\n\n![teaser](assets/teaser.jpg)\n\n\n## Installation and Setup\nFirst, clone the repository and submodules. Then, set up a new conda environment and install all dependencies, as follows:\n```bash\ngit clone --recursive https://github.com/statho/ScoreHMR.git\ncd ScoreHMR\nsource install_environment.sh\n```\n\nDownload the pretrained model weights, and annotations for the datasets by running the following:\n```\nsource download_data.sh\n```\nThis will download all necessary data files, and place them in `data/`. Alternatively, you can download them from [here](https://drive.google.com/file/d/1W53UMg8kee3HGRTNd2aNhMUew_kj36OH/view?usp=sharing) and [here](https://drive.google.com/file/d/1f-D3xhQPMC9rwtaCVNoxtD4BQh4oQbY9/view?usp=sharing). Besides these files, you also need to download the *SMPL* model. You will need the [neutral model](http://smplify.is.tue.mpg.de). Please go to the corresponding website and register to get access to the downloads section. Download the model, create a folder `data/smpl`, rename `basicModel_neutral_lbs_10_207_0_v1.0.0.pkl` to `SMPL_NEUTRAL.pkl`, and place it in `data/smpl/`.\n\nFinally, if you wish to run the evaluation and/or training code, you will need to download the images/videos for the datasets. The instructions are mostly common with the description in [here](https://github.com/nkolot/ProHMR/blob/master/dataset_preprocessing/README.md). We provide the annotations for all datasets, so you will only need to download the images/videos. Edit the `IMG_DIR` in `score_hmr/configs/datasets.yml` accordingly.\n\n\n## Run demo on images\nThe following command will run ScoreHMR on top of HMR 2.0, using detected keypoints from ViTPose and bounding boxes from ViTDet, on all images in the specified `--img_folder`. For each image, it will save a rendering of all the reconstructed people together in the front view.\n```bash\npython demo_image.py \\\n    --img_folder example_data/images \\\n    --out_folder demo_out/images\n```\n\n\n## Run demo on videos\nThe following command will first run tracking with 4D-Humans and 2D keypoint detection with ViTPose, and then run temporal model fitting with ScoreHMR on the video specified with `--input_video`. It will create a video rendering of the reconstructed people in the folder specified by `--out_folder`. It will also save intermediate results from 4D-Humans and ViTPose.\n```bash\npython demo_video.py \\\n    --input_video example_data/videos/breakdancing.mp4 \\\n    --out_folder demo_out/videos\n```\n\n\n## Evaluation\nThe evaluation code is contained in `eval/`. We provide evaluation on 3 different settings with the following scripts:\n- `eval_keypoint_fitting.py` is used in single-frame model fitting evaluation as in Tables 1, 2 \u0026 5 of the paper.\n- `eval_multiview.py` is used to evaluate the multi-view refinement as in Table 3 of the paper.\n- `eval_video.py` is used to evaluate ScoreHMR in temporal model fitting as in Table 4 of the paper.\n\nThe evaluation code uses cached HMR 2.0 predictions, which can be downloaded from [here](https://drive.google.com/file/d/1m9lv9uDYosIVZ-u0R3GCy1J1wHYNVUMP/view?usp=sharing) or by running:\n```bash\nsource download_hmr2_preds.sh\n```\nWe also provide example code for saving the HMR 2.0 predictions in the appropriate format in `data_preprocessing/cache_hmr2_preds.py`.\n\nEvaluation code example:\n```bash\npython eval/eval_keypoint_fitting.py --dataset 3DPW-TEST --shuffle --use_default_ckpt\n```\nRunning the above command will compute the MPJPE and Reconstruction Error before and after single-frame model fitting with ScoreHMR on the test set of 3DPW.\n\n\n## Training\nThe training code uses cached image features. First, extract the PARE image features for the training datasets:\n```\npython data_preprocessing/cache_pare_preds.py\n```\nThen, start training using the following command:\n```\npython train.py --name \u003cname_of_experiment\u003e\n```\nCheckpoints and logs will be saved to `logs/`.\n\n\n## Acknowledgements\nParts of the code are taken or adapted from the following repos:\n- [ProHMR](https://github.com/nkolot/ProHMR)\n- [PARE](https://github.com/mkocabas/PARE)\n- [SLAHMR](https://github.com/vye16/slahmr)\n- [4D-Humans](https://github.com/shubham-goel/4D-Humans)\n- [ViTPose](https://github.com/ViTAE-Transformer/ViTPose)\n- [Detectron2](https://github.com/facebookresearch/detectron2)\n- [denoising-diffusion-pytorch](https://github.com/lucidrains/denoising-diffusion-pytorch)\n\n## Citing\nIf you find this code useful for your research, please consider citing the following paper:\n\n```bibtex\n@inproceedings{stathopoulos2024score,\n  title  = {Score-Guided Diffusion for 3D Human Recovery},\n  author = {Stathopoulos, Anastasis and Han, Ligong and Metaxas, Dimitris},\n  booktitle = {CVPR},\n  year = {2024}\n}\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fstatho%2FScoreHMR","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fstatho%2FScoreHMR","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fstatho%2FScoreHMR/lists"}