{"id":15035000,"url":"https://github.com/mkocabas/vibe","last_synced_at":"2025-05-15T03:06:09.983Z","repository":{"id":37617328,"uuid":"227483440","full_name":"mkocabas/VIBE","owner":"mkocabas","description":"Official implementation of CVPR2020 paper \"VIBE: Video Inference for Human Body Pose and Shape Estimation\"","archived":false,"fork":false,"pushed_at":"2023-03-24T23:39:57.000Z","size":15384,"stargazers_count":2998,"open_issues_count":122,"forks_count":563,"subscribers_count":65,"default_branch":"master","last_synced_at":"2025-05-15T03:06:09.294Z","etag":null,"topics":["3d-human-pose","3d-pose-estimation","cvpr","cvpr-2020","cvpr20","cvpr2020","human-pose-estimation","pytorch","smpl","video-pose-estimation"],"latest_commit_sha":null,"homepage":"https://arxiv.org/abs/1912.05656","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"other","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/mkocabas.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null}},"created_at":"2019-12-12T00:01:28.000Z","updated_at":"2025-05-12T06:57:54.000Z","dependencies_parsed_at":"2022-07-12T16:33:54.687Z","dependency_job_id":"1193634d-d811-4145-82ae-87f0d90658c2","html_url":"https://github.com/mkocabas/VIBE","commit_stats":null,"previous_names":[],"tags_count":2,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mkocabas%2FVIBE","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mkocabas%2FVIBE/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mkocabas%2FVIBE/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mkocabas%2FVIBE/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/mkocabas","download_url":"https://codeload.github.com/mkocabas/VIBE/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":254264765,"owners_count":22041793,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["3d-human-pose","3d-pose-estimation","cvpr","cvpr-2020","cvpr20","cvpr2020","human-pose-estimation","pytorch","smpl","video-pose-estimation"],"created_at":"2024-09-24T20:27:08.281Z","updated_at":"2025-05-15T03:06:09.962Z","avatar_url":"https://github.com/mkocabas.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# VIBE: Video Inference for Human Body Pose and Shape Estimation [CVPR-2020]\n[![report](https://img.shields.io/badge/arxiv-report-red)](https://arxiv.org/abs/1912.05656) [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/drive/1dFfwxZ52MN86FA6uFNypMEdFShd2euQA) [![PWC](https://img.shields.io/endpoint.svg?url=https://paperswithcode.com/badge/vibe-video-inference-for-human-body-pose-and/3d-human-pose-estimation-on-3dpw)](https://paperswithcode.com/sota/3d-human-pose-estimation-on-3dpw?p=vibe-video-inference-for-human-body-pose-and)\n\n\u003cp float=\"center\"\u003e\n  \u003cimg src=\"doc/assets/header_1.gif\" width=\"49%\" /\u003e\n  \u003cimg src=\"doc/assets/header_2.gif\" width=\"49%\" /\u003e\n\u003c/p\u003e\n\nCheck our YouTube videos below for more details.\n\n| Paper Video                                                                                                | Qualitative Results                                                                                                |\n|------------------------------------------------------------------------------------------------------------|--------------------------------------------------------------------------------------------------------------------|\n| [![PaperVideo](https://img.youtube.com/vi/rIr-nX63dUA/0.jpg)](https://www.youtube.com/watch?v=rIr-nX63dUA) | [![QualitativeResults](https://img.youtube.com/vi/fW0sIZfQcIs/0.jpg)](https://www.youtube.com/watch?v=fW0sIZfQcIs) |\n\n\u003c!-- \u003csub\u003eSources: left video - [https://www.youtube.com/watch?v=qlPRDVqYO74](https://www.youtube.com/watch?v=qlPRDVqYO74), right video - [https://www.youtube.com/watch?v=Opry3F6aB1I](https://www.youtube.com/watch?v=Opry3F6aB1I)\n\u003c/sub\u003e --\u003e\n\n\u003e [**VIBE: Video Inference for Human Body Pose and Shape Estimation**](https://arxiv.org/abs/1912.05656),            \n\u003e [Muhammed Kocabas](https://ps.is.tuebingen.mpg.de/person/mkocabas), [Nikos Athanasiou](https://ps.is.tuebingen.mpg.de/person/nathanasiou), \n[Michael J. Black](https://ps.is.tuebingen.mpg.de/person/black),        \n\u003e *IEEE Computer Vision and Pattern Recognition, 2020* \n\n## Features\n\n_**V**ideo **I**nference for **B**ody Pose and Shape **E**stimation_ (VIBE) is a video pose and shape estimation method.\nIt predicts the parameters of SMPL body model for each frame of an input video. Pleaser refer to our [arXiv report](https://arxiv.org/abs/1912.05656) for further details.\n\nThis implementation:\n\n- has the demo and training code for VIBE implemented purely in PyTorch,\n- can work on arbitrary videos with multiple people,\n- supports both CPU and GPU inference (though GPU is way faster),\n- is fast, up-to 30 FPS on a RTX2080Ti (see [this table](doc/demo.md#runtime-performance)),\n- achieves SOTA results on 3DPW and MPI-INF-3DHP datasets,\n- includes Temporal SMPLify implementation.\n- includes the training code and detailed instruction on how to train it from scratch.\n- can create an FBX/glTF output to be used with major graphics softwares.\n\n\u003cp float=\"center\"\u003e\n  \u003cimg src=\"doc/assets/method_1.gif\" width=\"49%\" /\u003e\n  \u003cimg src=\"doc/assets/parkour.gif\" width=\"49%\" /\u003e\n\u003c/p\u003e\n\n## Updates\n\n- 05/01/2021: Windows installation tutorial is added thanks to amazing [@carlosedubarreto](https://github.com/carlosedubarreto)\n- 06/10/2020: Support OneEuroFilter smoothing.\n- 14/09/2020: FBX/glTF conversion script is released.\n\n## Getting Started\nVIBE has been implemented and tested on Ubuntu 18.04 with python \u003e= 3.7. It supports both GPU and CPU inference.\nIf you don't have a suitable device, try running our Colab demo. \n\nClone the repo:\n```bash\ngit clone https://github.com/mkocabas/VIBE.git\n```\n\nInstall the requirements using `virtualenv` or `conda`:\n```bash\n# pip\nsource scripts/install_pip.sh\n\n# conda\nsource scripts/install_conda.sh\n```\n\n## Running the Demo\n\nWe have prepared a nice demo code to run VIBE on arbitrary videos. \nFirst, you need download the required data(i.e our trained model and SMPL model parameters). To do this you can just run:\n\n```bash\nsource scripts/prepare_data.sh\n```\n\nThen, running the demo is as simple as:\n\n```bash\n# Run on a local video\npython demo.py --vid_file sample_video.mp4 --output_folder output/ --display\n\n# Run on a YouTube video\npython demo.py --vid_file https://www.youtube.com/watch?v=wPZP8Bwxplo --output_folder output/ --display\n```\n\nRefer to [`doc/demo.md`](doc/demo.md) for more details about the demo code.\n\nSample demo output with the `--sideview` flag:\n\n\u003cp float=\"left\"\u003e\n  \u003cimg src=\"doc/assets/sample_video.gif\" width=\"30%\" /\u003e\n\u003c/p\u003e\n\n### FBX and glTF output (New Feature!)\nWe provide a script to convert VIBE output to standalone FBX/glTF files to be used in 3D graphics tools like\nBlender, Unity etc. You need to follow steps below to be able to run the conversion script.\n\n- You need to download FBX files for SMPL body model\n    - Go to [SMPL website](https://smpl.is.tue.mpg.de/) and create an account.\n    - Download the Unity-compatible FBX file through the [link](https://psfiles.is.tuebingen.mpg.de/downloads/smpl/SMPL_unity_v-1-0-0-zip)\n    - Unzip the contents and locate them `data/SMPL_unity_v.1.0.0`.\n- Install Blender python API\n    - Note that we tested our script with Blender v2.8.0 and v2.8.3.\n- Run the command below to convert VIBE output to FBX:\n```\npython lib/utils/fbx_output.py \\\n    --input output/sample_video/vibe_output.pkl \\\n    --output output/sample_video/fbx_output.fbx \\ # specify the file extension as *.glb for glTF\n    --fps_source 30 \\\n    --fps_target 30 \\\n    --gender \u003cmale or female\u003e \\\n    --person_id \u003ctracklet id from VIBE output\u003e\n\n``` \n### Windows Installation Tutorial\n\nYou can follow the instructions provided by [@carlosedubarreto](https://github.com/carlosedubarreto) to install and run VIBE on a Windows machine:\n\n- VIBE windows installation tutorial: https://youtu.be/3qhs5IRJ1LI\n- FBX conversion: https://youtu.be/w1biKeiQThY\n- Helper github repo: https://github.com/carlosedubarreto/vibe_win_install\n\n## Google Colab\nIf you do not have a suitable environment to run this project then you could give Google Colab a try. \nIt allows you to run the project in the cloud, free of charge. You may try our Colab demo using the notebook we have prepared: \n[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/drive/1dFfwxZ52MN86FA6uFNypMEdFShd2euQA)\n\n\n## Training\nRun the commands below to start training:\n\n```shell script\nsource scripts/prepare_training_data.sh\npython train.py --cfg configs/config.yaml\n```\n\nNote that the training datasets should be downloaded and prepared before running data processing script.\nPlease see [`doc/train.md`](doc/train.md) for details on how to prepare them.\n \n## Evaluation\n\nHere we compare VIBE with recent state-of-the-art methods on 3D pose estimation datasets. Evaluation metric is\nProcrustes Aligned Mean Per Joint Position Error (PA-MPJPE) in mm.\n\n| Models         | 3DPW \u0026#8595; | MPI-INF-3DHP \u0026#8595; | H36M \u0026#8595; |\n|----------------|:----:|:------------:|:----:|\n| SPIN           | 59.2 |     67.5     | **41.1** |\n| Temporal HMR   | 76.7 |     89.8     | 56.8 |\n| VIBE           | 56.5 |     **63.4**     | 41.5 |\n\nSee [`doc/eval.md`](doc/eval.md) to reproduce the results in this table or \nevaluate a pretrained model.\n\n**Correction**: Due to a mistake in dataset preprocessing, VIBE trained with 3DPW results in Table 1 of the original paper are not correct.\nBesides, even though training with 3DPW guarantees better quantitative performance, it does not give good \nqualitative results. ArXiv version will be updated with the corrected results. \n\n## Citation\n\n```bibtex\n@inproceedings{kocabas2019vibe,\n  title={VIBE: Video Inference for Human Body Pose and Shape Estimation},\n  author={Kocabas, Muhammed and Athanasiou, Nikos and Black, Michael J.},\n  booktitle = {The IEEE Conference on Computer Vision and Pattern Recognition (CVPR)},\n  month = {June},\n  year = {2020}\n}\n```\n\n## License\nThis code is available for **non-commercial scientific research purposes** as defined in the [LICENSE file](LICENSE). By downloading and using this code you agree to the terms in the [LICENSE](LICENSE). Third-party datasets and software are subject to their respective licenses.\n\n\n## References\nWe indicate if a function or script is borrowed externally inside each file. Here are some great resources we \nbenefit:\n\n- Pretrained HMR and some functions are borrowed from [SPIN](https://github.com/nkolot/SPIN).\n- SMPL models and layer is from [SMPL-X model](https://github.com/vchoutas/smplx).\n- Some functions are borrowed from [Temporal HMR](https://github.com/akanazawa/human_dynamics).\n- Some functions are borrowed from [HMR-pytorch](https://github.com/MandyMo/pytorch_HMR).\n- Some functions are borrowed from [Kornia](https://github.com/kornia/kornia).\n- Pose tracker is from [STAF](https://github.com/soulslicer/openpose/tree/staf).\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmkocabas%2Fvibe","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fmkocabas%2Fvibe","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmkocabas%2Fvibe/lists"}