{"id":18387965,"url":"https://github.com/openpifpaf/openpifpaf_posetrack","last_synced_at":"2025-04-07T01:31:21.098Z","repository":{"id":52965525,"uuid":"342624545","full_name":"openpifpaf/openpifpaf_posetrack","owner":"openpifpaf","description":"OpenPifPaf plugin for Posetrack","archived":false,"fork":false,"pushed_at":"2022-05-27T09:37:49.000Z","size":563,"stargazers_count":31,"open_issues_count":2,"forks_count":5,"subscribers_count":5,"default_branch":"main","last_synced_at":"2025-03-22T10:23:54.828Z","etag":null,"topics":["composite-fields","computer-vision","pose-estimation","pose-tracking"],"latest_commit_sha":null,"homepage":"https://arxiv.org/abs/2103.02440","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/openpifpaf.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2021-02-26T15:53:23.000Z","updated_at":"2024-04-01T08:11:29.000Z","dependencies_parsed_at":"2022-08-20T00:41:17.678Z","dependency_job_id":null,"html_url":"https://github.com/openpifpaf/openpifpaf_posetrack","commit_stats":null,"previous_names":["vita-epfl/openpifpaf_posetrack"],"tags_count":1,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/openpifpaf%2Fopenpifpaf_posetrack","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/openpifpaf%2Fopenpifpaf_posetrack/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/openpifpaf%2Fopenpifpaf_posetrack/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/openpifpaf%2Fopenpifpaf_posetrack/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/openpifpaf","download_url":"https://codeload.github.com/openpifpaf/openpifpaf_posetrack/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":247577885,"owners_count":20961192,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["composite-fields","computer-vision","pose-estimation","pose-tracking"],"created_at":"2024-11-06T01:29:46.840Z","updated_at":"2025-04-07T01:31:21.080Z","avatar_url":"https://github.com/openpifpaf.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# The code has been migrated directly into the main [OpenPifPaf](https://github.com/openpifpaf/openpifpaf) repository. The documentation hasn't been moved yet from this repo to the guide in OpenPifPaf so I am keeping this repository for now.\n\n[![Tests](https://github.com/vita-epfl/openpifpaf_posetrack/actions/workflows/tests.yml/badge.svg?branch=main)](https://github.com/vita-epfl/openpifpaf_posetrack/actions/workflows/tests.yml)\n\nThis is the tracking plugin for [OpenPifPaf](https://github.com/vita-epfl/openpifpaf).\u003cbr /\u003e\n[__New__ 2021 paper](https://arxiv.org/abs/2103.02440):\n\n\u003e __OpenPifPaf: Composite Fields for Semantic Keypoint Detection and Spatio-Temporal Association__\u003cbr /\u003e\n\u003e _[Sven Kreiss](https://www.svenkreiss.com), [Lorenzo Bertoni](https://scholar.google.com/citations?user=f-4YHeMAAAAJ\u0026hl=en), [Alexandre Alahi](https://scholar.google.com/citations?user=UIhXQ64AAAAJ\u0026hl=en)_, 2021.\n\u003e\n\u003e Many image-based perception tasks can be formulated as detecting, associating\n\u003e and tracking semantic keypoints, e.g., human body pose estimation and tracking.\n\u003e In this work, we present a general framework that jointly detects and forms\n\u003e spatio-temporal keypoint associations in a single stage, making this the first\n\u003e real-time pose detection and tracking algorithm. We present a generic neural\n\u003e network architecture that uses Composite Fields to detect and construct a\n\u003e spatio-temporal pose which is a single, connected graph whose nodes are the\n\u003e semantic keypoints (e.g., a person's body joints) in multiple frames. For the\n\u003e temporal associations, we introduce the Temporal Composite Association Field\n\u003e (TCAF) which requires an extended network architecture and training method\n\u003e beyond previous Composite Fields. Our experiments show competitive accuracy\n\u003e while being an order of magnitude faster on multiple publicly available datasets\n\u003e such as COCO, CrowdPose and the PoseTrack 2017 and 2018 datasets. We also show\n\u003e that our method generalizes to any class of semantic keypoints such as car and\n\u003e animal parts to provide a holistic perception framework that is well suited for\n\u003e urban mobility such as self-driving cars and delivery robots.\n\nPrevious [CVPR 2019 paper](http://openaccess.thecvf.com/content_CVPR_2019/html/Kreiss_PifPaf_Composite_Fields_for_Human_Pose_Estimation_CVPR_2019_paper.html).\n\n\n# Install\n\n```sh\npip install 'openpifpaf_posetrack[test,train]'\n\n# from source:\npip install --editable '.[test,train]'\n```\n\n\n# Prediction\n\nThe standard `openpifpaf.video` still works exactly the same way. With this\nplugin installed, you can use `--checkpoint=tshufflenetv2k16` (with a `t` at the\nbeginning). This model can be decoded in multiple ways and you should pick one\ndecoder. To get started, we recommend `--decoder=trackingpose:0`.\nPutting it all together, an example command to process a video stream from a\ncamera is:\n\n```sh\nMPLBACKEND=macosx python3 -m openpifpaf.video --show --long-edge=321 --checkpoint=tshufflenetv2k16 --decoder=trackingpose:0 --source 0 --horizontal-flip\n```\n\n\n# Posetrack Dataset\n\nData. Follow the Posetrack instructions to download and untar the images.\nLabels:\n\n```sh\nmkdir data-posetrack\ncd data-posetrack\nwget https://posetrack.net/posetrack18-data/posetrack18_v0.45_public_labels.tar.gz\ntar -xvf posetrack18_v0.45_public_labels.tar.gz\nmv posetrack_data/* .\nrm -r posetrack_data\n```\n\nGenerate PoseTrack2017 json data of the ground truth.\nUsage of `octave` instead of `matlab` is not documented, but this seems to work:\n\n```sh\ncd matlab\noctave --no-gui --eval \"addpath('./external/jsonlab'); mat2json('your_relative_path/data-posetrack2017/annotations/val/'); quit\"\n```\n\nThis takes a long time. It is faster on the test set:\n\n```sh\noctave --no-gui --eval \"addpath('./external/jsonlab'); mat2json('your_relative_path/data-posetrack2017/annotations/test/'); quit\"\n```\n\nThe Posetrack poses look like these:\n\n![poses](docs/skeleton_overview.png)\n\nCreated with `python -m openpifpaf_posetrack.draw_poses`.\n\n\n# Train posetrack2018-cocokpst\n\n```sh\n# 210226\n\n# first convert from single image to tracking model\npython3 -m openpifpaf_posetrack.imagetotracking --checkpoint shufflenetv2k30\n\n# train\ntime python3 -m torch.distributed.launch --nproc_per_node=4 \\\n  -m openpifpaf.train --ddp \\\n  --lr=0.0003 --momentum=0.95 --b-scale=10.0 \\\n  --epochs=50 --lr-decay 40 45 --lr-decay-epochs=5 \\\n  --batch-size=8 \\\n  --weight-decay=1e-5 \\\n  --dataset=posetrack2018-cocokpst --dataset-weights 1 1 --stride-apply=2 \\\n  --posetrack-upsample=2 \\\n  --cocokp-upsample=2 --cocokp-orientation-invariant=0.1 --cocokp-blur=0.1 \\\n  --checkpoint outputs/tshufflenetv2k30-210217-075056-cocokp-o10s-6f9daa84.pkl\n```\n\n```sh\nCUDA_VISIBLE_DEVICES=3 python -m openpifpaf.eval \\\n  --watch --checkpoint \"outputs/tshufflenetv2k??-20121?-*-posetrack2018-*.pkl.epoch??[0,5]\" \\\n  --dataset=posetrack2018 \\\n  --loader-workers=8 \\\n  --decoder=trackingpose:0 \\\n  --write-predictions\n```\n\nThe training script supports ``--train-annotations`` and ``--val-annotations``\nto restrict the used annotation files. This is useful for local testing.\n\nTo produce submissions to the 2018 test server:\n\n```sh\nCUDA_VISIBLE_DEVICES=0 python -m openpifpaf.eval \\\n  --checkpoint outputs/tshufflenetv2k30-210222-112623-posetrack2018-cocokpst-o10-123ec670.pkl \\\n  --dataset=posetrack2018 --posetrack2018-eval-annotations=\"data-posetrack2018/annotations/test/*.json\" \\\n  --loader-workers=8 \\\n  --decoder=trackingpose:0 \\\n  --write-predictions\n```\n\nFor the 2017 test server:\n\n```sh\nCUDA_VISIBLE_DEVICES=1 python -m openpifpaf.eval \\\n  --checkpoint outputs/tshufflenetv2k30-210222-112623-posetrack2018-cocokpst-o10-123ec670.pkl \\\n  --dataset=posetrack2017 --posetrack2017-eval-annotations=\"data-posetrack2017/annotations/test/*.json\" \\\n  --loader-workers=8 \\\n  --decoder=trackingpose:0 \\\n  --write-predictions\n```\n\n\n## Citation\n\n```\n@article{kreiss2021openpifpaf,\n  title = {{OpenPifPaf: Composite Fields for Semantic Keypoint Detection and Spatio-Temporal Association}},\n  author = {Sven Kreiss and Lorenzo Bertoni and Alexandre Alahi},\n  journal = {arXiv preprint arXiv:2103.02440},\n  month = {March},\n  year = {2021}\n}\n\n@InProceedings{kreiss2019pifpaf,\n  author = {Kreiss, Sven and Bertoni, Lorenzo and Alahi, Alexandre},\n  title = {{PifPaf: Composite Fields for Human Pose Estimation}},\n  booktitle = {The IEEE Conference on Computer Vision and Pattern Recognition (CVPR)},\n  month = {June},\n  year = {2019}\n}\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fopenpifpaf%2Fopenpifpaf_posetrack","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fopenpifpaf%2Fopenpifpaf_posetrack","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fopenpifpaf%2Fopenpifpaf_posetrack/lists"}