{"id":13723590,"url":"https://github.com/idea-research/dwpose","last_synced_at":"2025-04-14T20:55:06.065Z","repository":{"id":184400565,"uuid":"670980145","full_name":"IDEA-Research/DWPose","owner":"IDEA-Research","description":"\"Effective Whole-body Pose Estimation with Two-stages Distillation\" (ICCV 2023, CV4Metaverse Workshop)","archived":false,"fork":false,"pushed_at":"2023-12-12T02:54:23.000Z","size":104098,"stargazers_count":2409,"open_issues_count":36,"forks_count":152,"subscribers_count":28,"default_branch":"onnx","last_synced_at":"2025-04-07T16:09:51.779Z","etag":null,"topics":["controlnet","knowledge-distillation","pose-estimation","stable-diffusion-webui"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/IDEA-Research.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null}},"created_at":"2023-07-26T09:12:37.000Z","updated_at":"2025-04-07T11:27:07.000Z","dependencies_parsed_at":"2024-01-18T05:04:36.718Z","dependency_job_id":null,"html_url":"https://github.com/IDEA-Research/DWPose","commit_stats":null,"previous_names":["idea-research/dwpose"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/IDEA-Research%2FDWPose","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/IDEA-Research%2FDWPose/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/IDEA-Research%2FDWPose/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/IDEA-Research%2FDWPose/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/IDEA-Research","download_url":"https://codeload.github.com/IDEA-Research/DWPose/tar.gz/refs/heads/onnx","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248961112,"owners_count":21189991,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["controlnet","knowledge-distillation","pose-estimation","stable-diffusion-webui"],"created_at":"2024-08-03T01:01:43.166Z","updated_at":"2025-04-14T20:55:01.052Z","avatar_url":"https://github.com/IDEA-Research.png","language":"Python","funding_links":[],"categories":["\u003cspan id=\"image\"\u003eImage\u003c/span\u003e"],"sub_categories":["\u003cspan id=\"tool\"\u003eLLM (LLM \u0026 Tool)\u003c/span\u003e"],"readme":"\n\u003cdiv align=\"center\"\u003e\n\u003cp align=\"center\"\u003e \u003cimg src=\"resources/logo.png\" width=\"100px\"\u003e \u003c/p\u003e\n\u003ch2\u003eEffective Whole-body Pose Estimation with Two-stages Distillation \u003c/h2\u003e \n\n\u003ca href='https://arxiv.org/abs/2307.15880'\u003e\u003cimg src='https://img.shields.io/badge/ArXiv-2307.15880-red'\u003e\u003c/a\u003e\n\n[![PWC](https://img.shields.io/endpoint.svg?url=https://paperswithcode.com/badge/effective-whole-body-pose-estimation-with-two/2d-human-pose-estimation-on-coco-wholebody-1)](https://paperswithcode.com/sota/2d-human-pose-estimation-on-coco-wholebody-1?p=effective-whole-body-pose-estimation-with-two)\n\n\n[Zhendong Yang](https://yzd-v.github.io/page/), [Ailing Zeng](https://ailingzeng.site/), [Chun Yuan](https://scholar.google.com/citations?user=fYdxi2sAAAAJ\u0026hl=en\u0026oi=sra), [Yu Li](http://yu-li.github.io/)\n\n\n\u003cimg src=\"resources/lalaland.gif\" style=\"height:200px\" /\u003e   \u003cimg src=\"resources/iron.gif\" style=\"height:200px\" /\u003e  \n\u003cp\u003e\u0026emsp; \u0026emsp; \u0026emsp; \u0026emsp; \u0026emsp; DWPose   \u0026emsp;   \u0026emsp;   \u0026emsp; \u0026emsp;   \u0026emsp; \u0026emsp; \u0026emsp;   \u0026emsp;\u0026emsp; \u0026emsp;   \u0026emsp; DWPose + ControlNet (\u003ci\u003eprompt: Ironman\u003c/i\u003e) \u003c/p\u003e\n\n\u003c/div\u003e\n\n#  💃🏻  DWPose 💃🏻\nThis repository is the official implementation of the [Effective Whole-body Pose Estimation with Two-stages Distillation](https://arxiv.org/abs/2307.15880) (ICCV 2023, CV4Metaverse Workshop). Our code is based on [MMPose](https://github.com/open-mmlab/mmpose/tree/main) and [ControlNet](https://github.com/lllyasviel/ControlNet-v1-1-nightly).\n\n\u003cp align=\"center\"\u003e\u003cimg src=\"resources/architecture.jpg\" width=\"650px\"/\u003e\n\n\n⚔️ We release a series of models named DWPose with different sizes, from tiny to large, for human whole-body pose estimation. Besides, we also replace Openpose with DWPose for ControlNet, obtaining better Generated Images. \n\n## 🔥 News\n- **`2023/12/03`**: DWPose supports [Consistent and Controllable Image-to-Video Synthesis for Character Animation](https://humanaigc.github.io/animate-anyone/).\n- **`2023/08/17`**: Our paper [Effective Whole-body Pose Estimation with Two-stages Distillation](https://arxiv.org/abs/2307.15880) is accepted by ICCV 2023, CV4Metaverse Workshop. 🎉 🎉 🎉\n\n- **`2023/08/09`**: You can try DWPose with [sd-webui-controlnet](https://github.com/Mikubill/sd-webui-controlnet) now! Just update your sd-webui-controlnet \u003e= v1.1237, then choose dw_openpose_full as preprocessor.\n\n- **`2023/08/09`**: We support to run onnx model with cv2. You can avoid installing onnxruntime. See branch [opencv_onnx](https://github.com/IDEA-Research/DWPose/tree/opencv_onnx).\n- **`2023/08/07`**: We upload all DWPose models to [huggingface](https://huggingface.co/yzd-v/DWPose/tree/main). Now, you can download them from baidu drive, google drive and huggingface.\n- **`2023/08/07`**: We release a new DWPose with onnx. You can avoid installing mmcv through this. See branch [onnx](https://github.com/IDEA-Research/DWPose/tree/onnx).\n- **`2023/08/01`**: Thanks to [MMPose](https://github.com/open-mmlab/mmpose/tree/main). You can try our DWPose with this [demo](https://openxlab.org.cn/apps/detail/mmpose/RTMPose) by choosing wholebody!\n\n\u003cp align=\"center\"\u003e\u003cimg src=\"resources/demo.png\" width=\"800px\"\u003e\n\n## 🐟 Installation\nSee [installation instructions](INSTALL.md). This branch uses onnx. You can try DWPose for ControlNet without mmcv.\n\n## 🚀 Results and Models\n### 😎 DWPose on COCO. We release a series of DWPose models.\n\n\u003cp align=\"center\"\u003e\u003cimg src=\"resources/compare.jpg\" width=\"350px\"/\u003e\n\nResults on COCO-WholeBody v1.0 val with detector having human AP of 56.4 on COCO val2017 dataset\n\n| Arch                                    | Input Size | FLOPS (G)| Body AP | Foot AP | Face AP | Hand AP | Whole AP |                   ckpt                   |                   ckpt                   |\n| :-------------------------------------- | :--------: | :--------: | :-----: | :-----: | :-----: | :-----: | :------: | :--------------------------------------: | :-------------------------------------: |\n| [DWPose-t](mmpose/configs/wholebody_2d_keypoint/rtmpose/ubody/rtmpose-t_8xb64-270e_coco-ubody-wholebody-256x192.py) |  256x192   |0.5|  0.585  |  0.465  |  0.735  |  0.357  |  0.485   | [baidu drive](https://pan.baidu.com/s/1X2sVxv4JOZ5WFvOBiwjrNA?pwd=nmvw) | [google drive](https://drive.google.com/file/d/1Csbg56QvB0TtFamJ6pPWNil7h6WziDwl/view?usp=sharing) |\n| [DWPose-s](mmpose/configs/wholebody_2d_keypoint/rtmpose/ubody/rtmpose-s_8xb64-270e_coco-ubody-wholebody-256x192.py) |  256x192   |0.9|  0.633  |  0.533  |  0.776  |  0.427  |  0.538   | [baidu drive](https://pan.baidu.com/s/1k2JxCtJL9dIGU-h31UBQOA?pwd=hcf2) | [google drive](https://drive.google.com/file/d/10TuEeLhArxfd4e6bnE7YgmBI9RFvu9DL/view?usp=sharing) |\n| [DWPose-m](mmpose/configs/wholebody_2d_keypoint/rtmpose/ubody/rtmpose-m_8xb64-270e_coco-ubody-wholebody-256x192.py) |  256x192   |2.2|  0.685  |  0.636  |  0.828  |  0.527  |  0.606   | [baidu drive](https://pan.baidu.com/s/183ovcYHV6I5TQ9Wu1eS-eg?pwd=rcry) | [google drive](https://drive.google.com/file/d/13ZWnGDteGBmjALtErYS8AHhMBBNAN9en/view?usp=sharing) |\n| [DWPose-l](mmpose/configs/wholebody_2d_keypoint/rtmpose/ubody/rtmpose-l_8xb64-270e_coco-ubody-wholebody-256x192.py) |  256x192   |4.5|  0.704  |  0.662  |  0.843  |  0.566  |  0.631   | [baidu drive](https://pan.baidu.com/s/1bWEeiFL5UGoDj9Nkazb98w?pwd=u7ek) | [google drive](https://drive.google.com/file/d/1PHKN3p873dgCSh_YRsYqTZVj-kIbclRS/view?usp=sharing) |\n| [DWPose-l](mmpose/configs/wholebody_2d_keypoint/rtmpose/ubody/rtmpose-l_8xb32-270e_coco-ubody-wholebody-384x288.py) |  384x288   |10.1|  0.722  |  0.704  |  0.887  |  0.621  |  0.665   | [baidu drive](https://pan.baidu.com/s/168T2XGXQDli8j03e_dOJdg?pwd=ajcq) | [google drive](https://drive.google.com/file/d/1Oy9O18cYk8Dk776DbxpCPWmJtJCl-OCm/view?usp=sharing) |\n\n### 🦈 DWPose for ControlNet.\nFirst, you need to download our Pose model dw-ll_ucoco_384.onnx ([baidu](https://pan.baidu.com/s/1nuBjw-KKSxD_BkpmwXUJiw?pwd=28d7), [google](https://drive.google.com/file/d/12L8E2oAgZy4VACGSK9RaZBZrfgx7VTA2/view?usp=sharing)) and Det model yolox_l.onnx ([baidu](https://pan.baidu.com/s/1fpfIVpv5ypo4c1bUlzkMYQ?pwd=mjdn), [google](https://drive.google.com/file/d/1w9pXC8tT0p9ndMN-CArp1__b2GbzewWI/view?usp=sharing)), then put them into ControlNet-v1-1-nightly/annotator/ckpts. Then you can use DWPose to generate the images you like.\n```\ncd ControlNet-v1-1-nightly\npython gradio_dw_open_pose.py\n```\n\n#### Non-cherry-picked test with random seed 12345 (\"spider man\"):\n\u003cp align=\"center\"\u003e\u003cimg src=\"resources/jay_pose.jpg\" width=\"600px\"/\u003e\n\n#### Comparison with OpenPose\n\u003cp align=\"center\"\u003e\u003cimg src=\"resources/generation.jpg\" width=\"600px\"/\u003e\n\n#### Run inference on any images\n\n```\ncd ControlNet-v1-1-nightly\npython dwpose_infer_example.py\n```\nNote: Please change the image path and output path based on your file.\n\n\n## 🚢 Datasets\nPrepare [COCO](https://cocodataset.org/#download) in mmpose/data/coco and [UBody](https://github.com/IDEA-Research/OSX) in mmpose/data/UBody.\n\nUBody needs to be tarnsferred into images. Don't forget.\n```\ncd mmpose\npython video2image.py\n```\nIf you want to evaluate the models on UBody\n```\n# add category into UBody's annotation\ncd mmpose\npython add_cat.py\n```\n## ⭐Train a model\n### Train DWPose with the first stage distillation\n```\ncd mmpose\nbash tools/dist_train.sh configs/distiller/ubody/s1_dis/rtmpose_x_dis_l__coco-ubody-256x192.py 8\n```\n### Train DWPose with the second stage distillation\n```\ncd mmpose\nbash tools/dist_train.sh configs/distiller/ubody/s2_dis/dwpose_l-ll__coco-ubody-256x192.py 8\n```\n### Tansfer the distillation models into regular models\n```\ncd mmpose\n# if first stage distillation\npython pth_transfer.py $dis_ckpt $new_pose_ckpt\n# if second stage distillation\npython pth_transfer.py $dis_ckpt $new_pose_ckpt --two_dis\n```\n## ⭐Test a model\n```\n# test on UBody\nbash tools/dist_test.sh configs/wholebody_2d_keypoint/rtmpose/ubody/rtmpose-l_8xb64-270e_ubody-wholebody-256x192.py $pose_ckpt 8\n\n# test on COCO\nbash tools/dist_test.sh configs/wholebody_2d_keypoint/rtmpose/ubody/rtmpose-l_8xb64-270e_coco-ubody-wholebody-256x192.py $pose_ckpt 8\n\n```\n\n## 🥳 Citation\n```\n@inproceedings{yang2023effective,\n  title={Effective whole-body pose estimation with two-stages distillation},\n  author={Yang, Zhendong and Zeng, Ailing and Yuan, Chun and Li, Yu},\n  booktitle={Proceedings of the IEEE/CVF International Conference on Computer Vision},\n  pages={4210--4220},\n  year={2023}\n}\n```\n\n## 🥂 Acknowledgement\nOur code is based on [MMPose](https://github.com/open-mmlab/mmpose/tree/main) and [ControlNet](https://github.com/lllyasviel/ControlNet-v1-1-nightly).\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fidea-research%2Fdwpose","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fidea-research%2Fdwpose","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fidea-research%2Fdwpose/lists"}