Ecosyste.ms: Awesome

An open API service indexing awesome lists of open source software.

Awesome Lists | Featured Topics | Projects

https://github.com/zhec/realtime_multi-person_pose_estimation

Code repo for realtime multi-person pose estimation in CVPR'17 (Oral)
https://github.com/zhec/realtime_multi-person_pose_estimation

caffe computer-vision cpp11 cvpr-2017 deep-learning human-behavior-understanding human-pose-estimation matlab python realtime

Last synced: 5 days ago
JSON representation

Code repo for realtime multi-person pose estimation in CVPR'17 (Oral)

Awesome Lists containing this project

README 

        
# Realtime Multi-Person Pose Estimation

By [Zhe Cao](https://people.eecs.berkeley.edu/~zhecao/), [Tomas Simon](http://www.cs.cmu.edu/~tsimon/), [Shih-En Wei](https://scholar.google.com/citations?user=sFQD3k4AAAAJ&hl=en), [Yaser Sheikh](http://www.cs.cmu.edu/~yaser/).



## Introduction

Code repo for winning 2016 MSCOCO Keypoints Challenge, 2016 ECCV Best Demo Award, and 2017 CVPR Oral paper.  



Watch our video result in [YouTube](https://www.youtube.com/watch?v=pW6nZXeWlGM&t=77s) or [our website](http://posefs1.perception.cs.cmu.edu/Users/ZheCao/humanpose.mp4). 



We present a bottom-up approach for realtime multi-person pose estimation, without using any person detector. For more details, refer to our [CVPR'17 paper](https://arxiv.org/abs/1611.08050), our [oral presentation video recording](https://www.youtube.com/watch?v=OgQLDEAjAZ8&list=PLvsYSxrlO0Cl4J_fgMhj2ElVmGR5UWKpB) at CVPR 2017 or our [presentation slides](http://image-net.org/challenges/talks/2016/Multi-person%20pose%20estimation-CMU.pdf) at ILSVRC and COCO workshop 2016.

















This project is licensed under the terms of the [license](LICENSE).



## Other Implementations

Thank you all for the efforts for the reimplementation! If you have new implementation and want to share with others, feel free to make a pull request or email me! 

- Our new C++ library [OpenPose](https://github.com/CMU-Perceptual-Computing-Lab/openpose) (testing only)

- Tensorflow [[version 1]](https://github.com/ildoonet/tf-openpose) | [[version 2]](https://github.com/michalfaber/keras_Realtime_Multi-Person_Pose_Estimation) | [[version 3]](https://github.com/anatolix/keras_Realtime_Multi-Person_Pose_Estimation) | [[version 4]](https://github.com/raymon-tian/keras_Realtime_Multi-Person_Pose_Estimation) | [[version 5]](https://github.com/tensorlayer/openpose) | [[version 6]](https://github.com/YangZeyu95/unofficial-implement-of-openpose)  | [[version 7 - TF2.1]](https://github.com/MikeOfZen/Yet-Another-Openpose-Implementation) 

- Pytorch [[version 1]](https://github.com/tensorboy/pytorch_Realtime_Multi-Person_Pose_Estimation) | [[version 2]](https://github.com/last-one/Pytorch_Realtime_Multi-Person_Pose_Estimation) | [[version 3]](https://github.com/CVBox/PyTorchCV) 

- Caffe2 [[version 1]](https://github.com/eddieyi/caffe2-pose-estimation)

- Chainer [[version 1]](https://github.com/DeNA/Chainer_Realtime_Multi-Person_Pose_Estimation)

- MXnet [[version 1]](https://github.com/dragonfly90/mxnet_Realtime_Multi-Person_Pose_Estimation)

- MatConvnet [[version 1]](https://github.com/coocoky/matconvnet_Realtime_Multi-Person_Pose_Estimation)

- CNTK [[version 1]](https://github.com/Hzzone/CNTK_Realtime_Multi-Person_Pose_Estimation)



## Contents

1. [Testing](#testing)

2. [Training](#training)

3. [Citation](#citation)



## Testing



### C++ (realtime version, for demo purpose)

- Please use [OpenPose](https://github.com/CMU-Perceptual-Computing-Lab/openpose), now it can run in CPU/ GPU and windows /Ubuntu.

- Three input options: images, video, webcam



### Matlab (slower, for COCO evaluation)

- Compatible with general [Caffe](http://caffe.berkeleyvision.org/). Compile matcaffe. 

- Run `cd testing; get_model.sh` to retrieve our latest MSCOCO model from our web server.

- Change the caffepath in the `config.m` and run `demo.m` for an example usage.



### Python

- `cd testing/python`

- `ipython notebook`

- Open `demo.ipynb` and execute the code



## Training



### Network Architecture

![Teaser?](https://github.com/ZheC/Multi-Person-Pose-Estimation/blob/master/readme/arch.png)



### Training Steps 

- Run `cd training; bash getData.sh` to obtain the COCO images in `dataset/COCO/images/`, keypoints annotations in `dataset/COCO/annotations/` and [COCO official toolbox](https://github.com/pdollar/coco) in `dataset/COCO/coco/`. 

- Run `getANNO.m` in matlab to convert the annotation format from json to mat in `dataset/COCO/mat/`.

- Run `genCOCOMask.m` in matlab to obatin the mask images for unlabeled person. You can use 'parfor' in matlab to speed up the code.

- Run `genJSON('COCO')` to generate a json file in `dataset/COCO/json/` folder. The json files contain raw informations needed for training.

- Run `python genLMDB.py` to generate your LMDB. (You can also download our LMDB for the COCO dataset (189GB file) by: `bash get_lmdb.sh`)

- Download our modified caffe: [caffe_train](https://github.com/CMU-Perceptual-Computing-Lab/caffe_train). Compile pycaffe. It will be merged with caffe_rtpose (for testing) soon.

- Run `python setLayers.py --exp 1` to generate the prototxt and shell file for training.

- Download [VGG-19 model](https://gist.github.com/ksimonyan/3785162f95cd2d5fee77), we use it to initialize the first 10 layers for training.

- Run `bash train_pose.sh 0,1` (generated by setLayers.py) to start the training with two gpus. 



## Citation

Please cite the paper in your publications if it helps your research:



    

    

    @inproceedings{cao2017realtime,

      author = {Zhe Cao and Tomas Simon and Shih-En Wei and Yaser Sheikh},

      booktitle = {CVPR},

      title = {Realtime Multi-Person 2D Pose Estimation using Part Affinity Fields},

      year = {2017}

      }

	  

    @inproceedings{wei2016cpm,

      author = {Shih-En Wei and Varun Ramakrishna and Takeo Kanade and Yaser Sheikh},

      booktitle = {CVPR},

      title = {Convolutional pose machines},

      year = {2016}

      }