https://github.com/tucan9389/poseestimation-tfliteswift
The baseline project for inferencing various Pose Estimation tflite models with TFLiteSwift on iOS
https://github.com/tucan9389/poseestimation-tfliteswift
3d-pose-estimation cpm hourglass ios mobilenetv2 on-device openpose pose-matching posenet simplepose tensorflow tensorflow-lite tflite
Last synced: 6 months ago
JSON representation
The baseline project for inferencing various Pose Estimation tflite models with TFLiteSwift on iOS
- Host: GitHub
- URL: https://github.com/tucan9389/poseestimation-tfliteswift
- Owner: tucan9389
- License: apache-2.0
- Created: 2020-03-13T14:14:27.000Z (over 5 years ago)
- Default Branch: master
- Last Pushed: 2022-09-04T19:17:28.000Z (almost 3 years ago)
- Last Synced: 2024-12-10T12:43:12.722Z (6 months ago)
- Topics: 3d-pose-estimation, cpm, hourglass, ios, mobilenetv2, on-device, openpose, pose-matching, posenet, simplepose, tensorflow, tensorflow-lite, tflite
- Language: Swift
- Homepage: https://github.com/motlabs/awesome-ml-demos-with-ios
- Size: 23.9 MB
- Stars: 151
- Watchers: 10
- Forks: 20
- Open Issues: 23
-
Metadata Files:
- Readme: README.md
- License: LICENSE
Awesome Lists containing this project
README
[](http://makeapullrequest.com)
This project is Pose Estimation on iOS with TensorFlow Lite.
If you are interested in iOS + Machine Learning, visit [here](https://github.com/motlabs/iOS-Proejcts-with-ML-Models) you can see various DEMOs.
| 2D pose estimation in real-time | 3D pose estimation |
| :----------------------------------------------------------: | :----------------------------------------------------------: |
| 
| 
|
## Features
- [x] Support 2D pose estimaiton TFLite models
- [x] Real-time demo with Metal framwork
- [x] Photo album demo
- [x] Support 3D pose estimation TFLite model
- [x] Real-time demo with Metal framwork (but realtime model is not ready yet)
- [x] Real-time pose matching demo
- [x] Photo album demo
- [x] Render the result keypoints of 2D pose estimation in 2D demo page
- [x] Render the result keypoints of 3D pose estimation with SceneKit
- [x] Render the heatmaps of 2D pose estimation output
- [x] Part Confidence Maps for typical heatmap based models
- [x] Part Affinity Fields for OpenPose (2D multi-person)
- [x] Implemented pose-matching with cosine similiarity in 3D pose demo
- [x] Implemented to fix the shoulders' keypoints in 3D pose estimation to pre-process for pose-matching
## Models
Source Images
Name | gif | img-0 | img-1 | img-2
:---: | :---: | :---: | :---: | :---:
Source | - | 
| 
| 
### Joint Samples
Model Names | gif | img-0 | img-1 | img-2
:---: | :---: | :---: | :---: | :---:
PoseNet | - |  |  | 
PEFM CPM | - |  |  | 
PEFM Hourglass | - |  |  | 
OpenPose (multi-person) | - |  |  | 
### Heatmap-ConfidenceMap Samples
Model Names | gif | img-0 | img-1 | img-2
:---: | :---: | :---: | :---: | :---:
PoseNet | - | - | - | - | -
PEFM CPM | - |  |  | 
PEFM Hourglass | - |  |  | 
OpenPose (multi-person) | - |  |  | 
### Heatmap-PAF Samples
Model Names | gif | img-0 | img-1 | img-2
:---: | :---: | :---: | :---: | :---:
OpenPose (PAF x) | - |  |  | 
OpenPose (PAF y) | - |  |  | 
### Meta Data
#### 2D
✅ vs ☑️ | Name | Size | Inference
Time
on iPhone11Pro | Post-process
Time
on iPhone11Pro | PCKh-0.5 | multi person
vs
single person | Model Source | Paper | tflite
download
:---: | --- | --- | --- | --- | --- | :---: | --- | --- | ---
✅ | **PoseNet** | 13.3 MB | - | - | - | single | [tensorflow/tensorflow](https://github.com/tensorflow/examples/blob/master/lite/examples/posenet/ios) | [PersonLab](https://arxiv.org/abs/1803.08225)
✅ | **PEFM CPM** | 2.4 MB | - | - | - | single | [edvardHua/PoseEstimationForMobile](https://github.com/edvardHua/PoseEstimationForMobile) | [Convolutional Pose Machines](https://arxiv.org/abs/1602.00134)
✅ | **PEFM Hourglass v1** | 1.8 MB | - | - | - | single | [edvardHua/PoseEstimationForMobile](https://github.com/edvardHua/PoseEstimationForMobile) | [Stacked Hourglass Networks](https://arxiv.org/abs/1603.06937)
✅ | **PEFM Hourglass v2** | 1.7 MB | - | - | - | single | [edvardHua/PoseEstimationForMobile](https://github.com/edvardHua/PoseEstimationForMobile) | [Stacked Hourglass Networks](https://arxiv.org/abs/1603.06937)
✅ | **OpenPose** | 7.8 MB | - | - | - | multi | [ildoonet/tf-pose-estimation](https://github.com/ildoonet/tf-pose-estimation/issues/355) | [OpenPose](https://arxiv.org/abs/1812.08008)
☑️ | **AlphaPose** | - | - | - | - | single | [osmr/imgclsmob](https://github.com/osmr/imgclsmob) | [RMPE](https://arxiv.org/abs/1612.00137)
☑️ | **SelecSLS** | - | - | - | - | single | [osmr/imgclsmob](https://github.com/osmr/imgclsmob) | -
☑️ | **IBPPose** | - | - | - | - | single | [osmr/imgclsmob](https://github.com/osmr/imgclsmob) | -
☑️ | **Lightweight OpenPose** | - | - | - | - | single | [osmr/imgclsmob](https://github.com/osmr/imgclsmob) | [OpenPose](https://arxiv.org/abs/1812.08008)
#### 3D
✅ vs ☑️ | Name | Size | Inference
Time
on iPhone11Pro | Post-process
Time
on iPhone11Pro | (metric) | Model Source | Paper | tflite
download
:---: | --- | --- | --- | --- | --- | --- | --- | ---
✅ | **Baseline3DPose** | 137.1 MB | 347 ms | 79 ms | - | [mks0601/3DMPPE_POSENET_RELEASE](https://github.com/mks0601/3DMPPE_POSENET_RELEASE) | [Baseline3D](https://arxiv.org/abs/1907.11346) | [download](https://github.com/tucan9389/PoseEstimation-TFLiteSwift/releases/download/v2.0.0/baseline_moon_noS.tflite)
✅ | **LiteBaseline3DPose** | **16.6 MB** | 116 ms
(cpu only) | 19 ms
(cpu only) | | [SangbumChoi/MobileHumanPose](https://github.com/SangbumChoi/MobileHumanPose) | [MHP](https://openaccess.thecvf.com/content/CVPR2021W/MAI/papers/Choi_MobileHumanPose_Toward_Real-Time_3D_Human_Pose_Estimation_in_Mobile_Devices_CVPRW_2021_paper.pdf) | [download](https://github.com/tucan9389/PoseEstimation-TFLiteSwift/releases/download/v2.1.0/lightweight_baseline_choi.tflite)
## Requirements
- Xcode 11.3+
- iOS 11.0+
- Swift 5
- CocoaPods
```shell
gem install cocoapods
```
## Build & Run
1. Install dependencies with cocoapods
```shell
cd ~/{PROJECT_PATH}
pod install
```
2. Open the `PoseEstimation-TFLiteSwift.xcworkspace` and run the project
## See also
- [motlabs/awesome-ml-demos-with-ios](https://github.com/motlabs/awesome-ml-demos-with-ios)
- TensorFlow Lite or Tensorflow models provided by:
- CPM and Hourglass model provided by [edvardHua/PoseEstimationForMobile](https://github.com/edvardHua/PoseEstimationForMobile)
- PoseNet model provided by [tensorflow/examples](https://github.com/tensorflow/examples/blob/master/lite/examples/posenet/ios)
- OpenPose model provided by [ildoonet/tf-pose-estimation](https://github.com/ildoonet/tf-pose-estimation)
- Various model provided by [osmr/imgclsmob](https://github.com/osmr/imgclsmob)
- 3DMPPE PoseNet model provided by [mks0601/3DMPPE_POSENET_RELEASE](https://github.com/mks0601/3DMPPE_POSENET_RELEASE)
- Pose estimation with Core ML - [tucan9389/PoseEstimation-CoreML](https://github.com/tucan9389/PoseEstimation-CoreML)
## License
This repository is licensed under Apache 2.0. Full license text is available in [LICENSE](LICENSE).