{"id":17260345,"url":"https://github.com/mkocabas/epipolarpose","last_synced_at":"2025-04-05T14:08:57.515Z","repository":{"id":39351821,"uuid":"173293097","full_name":"mkocabas/EpipolarPose","owner":"mkocabas","description":"Self-Supervised Learning of 3D Human Pose using Multi-view Geometry (CVPR2019)","archived":false,"fork":false,"pushed_at":"2019-06-26T17:00:01.000Z","size":307,"stargazers_count":601,"open_issues_count":20,"forks_count":98,"subscribers_count":24,"default_branch":"master","last_synced_at":"2025-03-29T13:11:21.550Z","etag":null,"topics":["3d-pose-estimation","computer-vision","cvpr","cvpr-2019","cvpr19","cvpr2019","human-pose-estimation","machine-learning","pytorch","self-supervised-learning"],"latest_commit_sha":null,"homepage":"","language":"Jupyter Notebook","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"other","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/mkocabas.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2019-03-01T11:45:18.000Z","updated_at":"2025-03-04T01:05:58.000Z","dependencies_parsed_at":"2022-08-01T08:19:24.234Z","dependency_job_id":null,"html_url":"https://github.com/mkocabas/EpipolarPose","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mkocabas%2FEpipolarPose","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mkocabas%2FEpipolarPose/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mkocabas%2FEpipolarPose/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mkocabas%2FEpipolarPose/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/mkocabas","download_url":"https://codeload.github.com/mkocabas/EpipolarPose/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":247345854,"owners_count":20924102,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["3d-pose-estimation","computer-vision","cvpr","cvpr-2019","cvpr19","cvpr2019","human-pose-estimation","machine-learning","pytorch","self-supervised-learning"],"created_at":"2024-10-15T07:47:56.864Z","updated_at":"2025-04-05T14:08:57.497Z","avatar_url":"https://github.com/mkocabas.png","language":"Jupyter Notebook","funding_links":[],"categories":[],"sub_categories":[],"readme":"[![PWC](https://img.shields.io/endpoint.svg?url=https://paperswithcode.com/badge/self-supervised-learning-of-3d-human-pose/3d-human-pose-estimation-human36m)](https://paperswithcode.com/sota/3d-human-pose-estimation-human36m?p=self-supervised-learning-of-3d-human-pose)\n\n# Self-Supervised Learning of 3D Human Pose using Multi-view Geometry (CVPR2019) [[project page](https://mkocabas.github.io/epipolarpose.html)]\n\n## Introduction\nThis is a pytorch implementation of \n[*Self-Supervised Learning of 3D Human Pose using Multi-view Geometry*](https://arxiv.org/abs/1903.02330) paper.\n\n\u003e [**Self-Supervised Learning of 3D Human Pose using Multi-view Geometry**](https://arxiv.org/abs/1903.02330),            \n\u003e [Muhammed Kocabas](http://user.ceng.metu.edu.tr/~e2270981/)\\*, [Salih Karagoz](https://salihkaragoz.github.io/)\\*, \n[Emre Akbas](http://user.ceng.metu.edu.tr/~emre/),        \n\u003e *IEEE Computer Vision and Pattern Recognition, 2019* (\\*equal contribution)   \n\nIn this work, we present **_EpipolarPose_**, a self-supervised learning method for \n3D human pose estimation, which does not need any 3D ground-truth data or camera extrinsics.\n\nDuring training, EpipolarPose estimates 2D poses from multi-view images, and then, utilizes epipolar geometry \nto obtain a 3D pose and camera geometry which are subsequently used to train a 3D pose estimator.\n\nIn the test time, it only takes an RGB image to produce a 3D pose result. Check out [`demo.ipynb`](demo.ipynb) to\nrun a simple demo.\n\nHere we show some sample outputs from our model on the Human3.6M dataset. \nFor each set of results we first show the input image, followed by the ground truth, \nfully supervised model and self supervised model outputs.\n\n\u003cp align=\"center\"\u003e\u003cimg src=\"https://i.imgur.com/jZph53h.png\" width=\"100%\" alt=\"\"/\u003e\u003c/p\u003e\n\n### Video Demo\n\u003cp align=\"center\"\u003e\u003ca target=_blank href=\"http://www.youtube.com/watch?v=lkXBiKRfRDw\"\u003e\u003cimg src=\"http://img.youtube.com/vi/lkXBiKRfRDw/0.jpg\" width=\"50%\" alt=\"\" /\u003e\u003c/a\u003e\u003c/p\u003e\n\n\n\n## Overview\n- `scripts/`: includes training and validation scripts.\n- `lib/`: contains data preparation, model definition, and some utility functions.\n- `refiner/`: includes the implementation of _refinement unit_ explained in the paper Section 3.3.\n- `experiments/`: contains `*.yaml` configuration files to run experiments.\n- `sample_images/`: images from Human3.6M dataset to run demo notebook.\n\n\n## Requirements\nThe code is developed using python 3.7.1 on Ubuntu 16.04. NVIDIA GPUs ared needed to train and test. \nSee [`requirements.txt`](requirements.txt) or [`environment.yml`](environment.yml) for other dependencies.\n\n## Quick start\n### Installation\n1. Install pytorch \u003e= v1.0.0 following [official instructions](https://pytorch.org/).\n   _Note that if you use pytorch's version \u003c v1.0.0, you should follow the instructions at \n   \u003chttps://github.com/Microsoft/human-pose-estimation.pytorch\u003e to disable cudnn's implementation of \n   BatchNorm layer. We encourage you to use higher pytorch's version(\u003e=v1.0.0)_\n2. Clone this repo, and we will call the directory that you cloned as `${ROOT}`\n3. Install dependencies using `pip`.\n   ```\n   pip install -r requirements.txt\n   ```\n   or create a new `conda` env:\n   ```\n   conda env create -f environment.yml\n   ```\n4. Download annotation files from [GoogleDrive](https://drive.google.com/open?id=147AlIWRv9QDmp5pGjwG2yMWEG_2-E2ai) \n(150 MB) as a zip file under `${ROOT}` folder. Run below commands to unzip them.\n   ```\n   unzip data.zip\n   rm data.zip\n   ```\n5. Finally prepare your workspace by running:\n    ```bash\n    mkdir output\n    mkdir models\n    ```\n    Optionally you can download pretrained weights using the links in the below table. You can put them under `models`\n    directory. At the end, your directory tree should like this.\n\n   ```\n   ${ROOT}\n   ├── data/\n   ├── experiments/\n   ├── lib/\n   ├── models/\n   ├── output/\n   ├── refiner/\n   ├── sample_images/\n   ├── scripts/\n   ├── demo.ipynb\n   ├── README.md\n   └── requirements.txt\n   ```\n6. Yep, you are ready to run [`demo.ipynb`](demo.ipynb).\n### Data preparation\nYou would need Human3.6M data to train or test our model. **For Human3.6M data**, please download from \n[Human 3.6 M dataset](http://vision.imar.ro/human3.6m/description.php). \nYou would need to create an account to get download permission. After downloading video files, you can run \n[our script](https://gist.github.com/mkocabas/5669c12debec54b172797743a3c0b778) to extract images.\nThen run `ln -s \u003cpath_to_extracted_h36m_images\u003e ${ROOT}/data/h36m/images` to create a soft link to images folder.\nCurrently you can use annotation files we provided in step 4, however we will release the annotation preparation\nscript soon after cleaning and proper testing.    \n\nIf you would like to pretrain an EpipolarPose model on MPII data, \nplease download image files from \n[MPII Human Pose Dataset](https://datasets.d2.mpi-inf.mpg.de/andriluka14cvpr/mpii_human_pose_v1.tar.gz) (12.9 GB).\nExtract it under `${ROOT}/data/mpii` directory. If you already have the MPII dataset, you can create a soft link to images:\n`ln -s \u003cpath_to_mpii_images\u003e ${ROOT}/data/mpii/images`\n\nDuring training, we make use of [synthetic-occlusion](https://github.com/isarandi/synthetic-occlusion). If you want to use it please download the Pascal VOC dataset as instructed in their [repo](https://github.com/isarandi/synthetic-occlusion#getting-started) and update the `VOC` parameter in configuration files.\n\nAfter downloading the datasets your `data` directory tree should look like this: \n```\n${ROOT}\n|── data/\n├───├── mpii/\n|   └───├── annot/\n|       └── images/\n|       \n└───├── h36m/\n    └───├── annot/\n        └── images/\n            ├── S1/\n            └── S5/\n            ...\n```\n### Pretrained Models\n\n#### Human3.6M\nDownload pretrained models using the given links, and put them under indicated paths.\n\n| Model                        | Backbone               | MPJPE on Human3.6M (mm) |    Link   | Directory                                 |\n|------------------------------|:----------------------:|:-----------------------:|:---------:|------------------------------------------|\n| Fully Supervised             | resnet18               | 63.0                    | [model](https://drive.google.com/open?id=1b_1o-gTriiypZlVeZkvoqQfRIqZhpD5T) | `models/h36m/fully_supervised_resnet18.pth.tar`        |\n| Fully Supervised             | resnet34               | 59.6                    | [model](https://drive.google.com/open?id=1pDLqOoUjgC-UI979l_u-AV5hJHcFNaOB) | `models/h36m/fully_supervised_resnet34.pth.tar`        |\n| Fully Supervised             | resnet50               | 51.8                    | [model](https://drive.google.com/open?id=1TCiUCwyNKviO_-_f6WaOJe3b4pI9uPDh) | `models/h36m/fully_supervised.pth.tar`        |\n| Self Supervised R/t          | resnet50               | 76.6                    | [model](https://drive.google.com/open?id=1OVKnQy_TCJXemsAxA9rdEToYVBG4JL4j) | `models/h36m/self_supervised_with_rt.pth.tar` |\n| Self Supervised without R/t  | resnet50               | 78.8 (NMPJPE)           | [model](https://drive.google.com/open?id=1iJT0b5vWgdGyseO-ZNMn9S5ATMuutmac) | `models/h36m/self_supervised_wo_rt.pth.tar`   |\n| Self Supervised (2D GT)      | resnet50               | 55.0                    | [model](https://drive.google.com/open?id=1RZ32GmikfixiQ6C-CTuccq5qfuxAJj8R) | `models/h36m/self_supervised_2d_gt.pth.tar`   |\n| Self Supervised (Subject 1)  | resnet50               | 65.3                    | [model](https://drive.google.com/open?id=1Zo78GY4itUm2fQ85M-v1tYmwoSDkRrW1) | `models/h36m/self_supervised_s1.pth.tar`      |\n| Self Supervised + refinement | MLP-baseline           | 60.5                    | [model](https://drive.google.com/open?id=1oW0PYNVd63G1BkRLmYczqqCsUVEsstPJ) | `models/h36m/refiner.pth.tar`                 |\n\n- **Fully Supervised:** trained using ground truth data.\n- **Self Supervised R/t:** trained using only camera extrinsic parameters.\n- **Self Supervised without R/t:** trained without any ground truth data or camera parameters.\n- **Self Supervised (2D GT):** trained with triangulations from ground truth 2D keypoints provided by the dataset.\n- **Self Supervised (Subject 1):** trained with only ground truth data of Subject #1.\n- **Self Supervised + refinement:** trained with a refinement module. For details of this setting please refer to\n[`refiner/README.md`](refiner/README.md)\n\nCheck out the [paper](https://arxiv.org/abs/1903.02330) for more details about training strategies of each model.\n\n#### MPII\nTo train an EpipolarPose model from scratch, you would need the model pretrained on MPII dataset.\n\n| Model         | Backbone| Mean PCK (%) |    Link   | Directory                                 |\n|---------------|:------------:|:-----------------------:|:---------:|------------------------------------------|\n| MPII Integral | resnet18       | 84.7                    | [model](https://drive.google.com/open?id=1ygdMG5bHcgFSkNCIMJbyDkKhUQebXaoY) | `models/mpii/mpii_integral_r18.pth.tar`|\n| MPII Integral | resnet34       | 86.3                    | [model](https://drive.google.com/open?id=1MFzLysHRq8SbO3V6L8o0ZMTKCIsDDGMW) | `models/mpii/mpii_integral_r34.pth.tar`|\n| MPII Integral | resnet50       | 88.3                    | [model](https://drive.google.com/open?id=19ee09gyyPOyrAzarQgM_YXqpaCthAfED) | `models/mpii/mpii_integral.pth.tar`|\n| MPII heatmap | resnet50       | 88.5                    | [model](https://drive.google.com/open?id=1R4-3uA1o10Gm7zPEemSE2gUBrEF46D1O) | `models/mpii/mpii_heatmap.pth.tar`|\n\n\n### Validation on H36M using pretrained models\nIn order to run validation script with a self supervised model, update the `MODEL.RESUME` field of \n[`experiments/h36m/valid-ss.yaml`](experiments/h36m/valid-ss.yaml) with the path to the pretrained weight and run:\n```\npython scripts/valid.py --cfg experiments/h36m/valid-ss.yaml\n```\nTo run a fully supervised model on validation set, update the `MODEL.RESUME` field of \n[`experiments/h36m/valid.yaml`](experiments/h36m/valid.yaml) with the path to the pretrained weight and run:\n```\npython scripts/valid.py --cfg experiments/h36m/valid.yaml\n```\n\n### Training on H36M\nTo train a self supervised model, try:\n```\npython scripts/train.py --cfg experiments/h36m/train-ss.yaml\n```\nFully supervised model:\n```\npython scripts/train.py --cfg experiments/h36m/train.yaml\n```\n### Citation\nIf this work is useful for your research, please cite our [paper](https://arxiv.org/abs/1903.02330):\n```\n@inproceedings{kocabas2019epipolar,\n    author = {Kocabas, Muhammed and Karagoz, Salih and Akbas, Emre},\n    title = {Self-Supervised Learning of 3D Human Pose using Multi-view Geometry},\n    booktitle = {The IEEE Conference on Computer Vision and Pattern Recognition (CVPR)},\n    month = {June},\n    year = {2019}\n}\n```\n### References\n\n- [Integral Human Pose](https://github.com/JimmySuen/integral-human-pose)\n- [Simple Baselines for Human Pose Estimation](https://github.com/Microsoft/human-pose-estimation.pytorch/)  \n- [3d-pose-baseline](https://github.com/una-dinosauria/3d-pose-baseline)\n- [Synthetic-occlusion](https://github.com/isarandi/synthetic-occlusion)\n\nWe thank to the authors for releasing their codes. Please also consider citing their works.\n\n### License\nThis code is freely available for free non-commercial use, and may be redistributed under these conditions. \nPlease, see the [LICENSE](LICENSE) for further details. \nThird-party datasets and softwares are subject to their respective licenses. \n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmkocabas%2Fepipolarpose","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fmkocabas%2Fepipolarpose","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmkocabas%2Fepipolarpose/lists"}