{"id":16212986,"url":"https://github.com/chenfengye/sportscap","last_synced_at":"2025-03-16T11:31:00.251Z","repository":{"id":44458297,"uuid":"380897517","full_name":"ChenFengYe/SportsCap","owner":"ChenFengYe","description":"[IJCV 2021] SportsCap: Monocular 3D Human Motion Capture and Fine-grained Understanding in Challenging Sports Videos","archived":false,"fork":false,"pushed_at":"2021-08-17T13:09:39.000Z","size":18157,"stargazers_count":125,"open_issues_count":1,"forks_count":13,"subscribers_count":7,"default_branch":"main","last_synced_at":"2025-03-16T01:53:16.605Z","etag":null,"topics":["action-recognition","dataset","deep-learning","ijcv2021","motion-capture","motion-prior","sport","sports-analytics","sports-data"],"latest_commit_sha":null,"homepage":"https://chenxin.tech/SportsCap.html","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/ChenFengYe.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2021-06-28T03:45:58.000Z","updated_at":"2025-03-14T04:00:33.000Z","dependencies_parsed_at":"2022-08-30T02:20:52.171Z","dependency_job_id":null,"html_url":"https://github.com/ChenFengYe/SportsCap","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ChenFengYe%2FSportsCap","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ChenFengYe%2FSportsCap/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ChenFengYe%2FSportsCap/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ChenFengYe%2FSportsCap/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/ChenFengYe","download_url":"https://codeload.github.com/ChenFengYe/SportsCap/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":243862550,"owners_count":20360164,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["action-recognition","dataset","deep-learning","ijcv2021","motion-capture","motion-prior","sport","sports-analytics","sports-data"],"created_at":"2024-10-10T10:54:17.463Z","updated_at":"2025-03-16T11:30:56.161Z","avatar_url":"https://github.com/ChenFengYe.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# SportsCap: Monocular 3D Human Motion Capture and Fine-grained Understanding in Challenging Sports Videos\n### [ProjectPage](https://chenxin.tech/SportsCap.html) | [Paper](https://arxiv.org/abs/2104.11452) | [Video](https://www.youtube.com/watch?v=TrqAaaX97KY) |  Dataset ([Part01](https://drive.google.com/file/d/1hUlGglrlWdjZNFFQh2ck3UaMDL8sDwQv/view?usp=sharing)|[Part02](https://drive.google.com/file/d/1NswiD-wpuAyHbSgdUCZ2s9QmVIFPcLjA/view?usp=sharing))\n[Xin Chen](https://chenxin.tech/), Anqi Pang, [Wei Yang](https://scholar.google.com/citations?user=fRjxdPgAAAAJ\u0026hl=en), [Yuexin Ma](http://yuexinma.me/aboutme.html), [Lan Xu](http://xu-lan.com/), [Jingyi Yu](http://www.yu-jingyi.com/).\u003c/br\u003e\n\n\nThis repository contains the official implementation for the paper: [SportsCap: Monocular 3D Human Motion Capture and Fine-grained Understanding in Challenging Sports Videos (IJCV 2021)](https://arxiv.org/abs/2104.11452). Our work is capable of simultaneously capturing 3D human motions and understanding fine-grained actions from monocular challenging sports video input.\u003cbr\u003e\n\u003cp float=\"left\"\u003e\n  \u003cimg src=\"./README/teaser.png\" width=\"800\" /\u003e\n\u003c/p\u003e\n\n## Abstract\nMarkerless motion capture and understanding of professional non-daily human movements is an important yet unsolved task, which suffers from complex motion patterns and severe self-occlusion, especially for the monocular setting. In this paper, we propose SportsCap -- the first approach for simultaneously capturing 3D human motions and understanding fine-grained actions from monocular challenging sports video input. Our approach utilizes the semantic and temporally structured sub-motion prior in the embedding space for motion capture and understanding in a data-driven multi-task manner. Comprehensive experiments on both public and our proposed datasets show that with a challenging monocular sports video input, our novel approach not only significantly improves the accuracy of 3D human motion capture, but also recovers accurate fine-grained semantic action attributes.\n\n[comment]: \u003cTo enable robust capture under complex motion patterns, we propose an effective motion embedding module to recover both the implicit motion embedding and explicit 3D motion details via a corresponding mapping function as well as a sub-motion classifier. Based on such hybrid motion information, we introduce a multi-stream spatial-temporal Graph Convolutional Network(ST-GCN) to predict the fine-grained semantic action attributes, and adopt a semantic attribute mapping block to assemble various correlated action attributes into a high-level action label for the overall detailed understanding of the whole sequence, so as to enable various applications like action assessment or motion scoring.\u003e \n\n## Licenses\n\u003ca rel=\"license\" href=\"http://creativecommons.org/licenses/by-nc-sa/4.0/\"\u003e\u003cimg alt=\"Creative Commons License\" style=\"border-width:0\" src=\"https://i.creativecommons.org/l/by-nc-sa/4.0/80x15.png\" /\u003e\u003c/a\u003e\u003cbr /\u003eThis work is licensed under a \u003ca rel=\"license\" href=\"http://creativecommons.org/licenses/by-nc-sa/4.0/\"\u003eCreative Commons Attribution-NonCommercial-ShareAlike 4.0 International License\u003c/a\u003e.\n\nAll material is made available under [Creative Commons BY-NC-SA 4.0](https://creativecommons.org/licenses/by-nc-sa/4.0/legalcode) license. You can **use, redistribute, and adapt** the material for **non-commercial purposes**, as long as you give appropriate credit by **citing our paper** and **indicating any changes** that you've made.\n\n## The SMART Dataset\nSportsCap proposes a challenging sports dataset called Sports Motion and Recognition Tasks (SMART) dataset, which contains per-frame action labels, manually annotated pose, and action assessment of various challenging sports video clips from professional referees.\n\n\u003cp float=\"left\"\u003e\n  \u003cimg src=\"./README/dataset.gif\" width=\"800\" /\u003e\n\u003c/p\u003e\n\n### Download\nYou can download the SMART dataset (17 GB, version 1.0) from the Google Drive [[SMART_part01](https://drive.google.com/file/d/1hUlGglrlWdjZNFFQh2ck3UaMDL8sDwQv/view?usp=sharing) | [SMART_part02](https://drive.google.com/file/d/1NswiD-wpuAyHbSgdUCZ2s9QmVIFPcLjA/view?usp=sharing)]. The SMART dataset includes source images (\u003e60,000), annotations(\u003e45,000, both pose and action), sport motion embedding spaces, videos (coming soon) and tools.\n\n### Annotation\nPlease load these JSON files in python to parse these annotations about 2D key-points of poses and fine-grained action labels.\n```\nTable_VideoInfo_diving.json\nTable_VideoInfo_gym.json\nTable_VideoInfo_polevalut_highjump_badminton.json\n```\n### Tools\nThe tools folder includes several functions to load the annotation and calculate the pose variables. More useful scripts are coming soon.\n```\nutils.py - json_load, crop_img_skes, cal_body_bbox ...\n```\n## Sports Motion Embedding Spaces\nWith the annotated 2D poses and MoCap 3D pose data, we collect the Sports Motion Embedding Spaces (SMES), the 2D/3D pose priors for various sports. SMES provides strong prior and regularization to ensure that the generated pose result lies in the corresponding action space.\n\u003cp float=\"left\"\u003e\n  \u003cimg src=\"./README/MES.png\" width=\"800\" /\u003e\n\u003c/p\u003e\n\n### Download\nYou can download the Motion Embedding Spaces (SMES) (7 MB, version 1.0) separately from [GoogleDrive](https://drive.google.com/file/d/1gWyBxMUrImkWoz8YWIm_XrCsGpxovK0Q/view?usp=sharing). The released SMES-V1.0 includes many sports, like vault, uneven bar, boxing, diving, hurdles, pole vault, high jump, and so on.\n\n### Usage\nComing soon.\n\n## Citation\nIf you find our code or paper useful, please consider citing:\n```\n@article{chen2021sportscap,\n  title={SportsCap: Monocular 3D Human Motion Capture and Fine-Grained Understanding in Challenging Sports Videos},\n  author={Xin Chen and Anqi Pang and Wei Yang and Yuexin Ma and Lan Xu and Jingyi Yu},\n  journal={International Journal of Computer Vision},\n  year={2021},\n  month={Aug},\n  url={https://doi.org/10.1007/s11263-021-01486-4}\n}\n```\n\n## Relevant Works\n[**ChallenCap: Monocular 3D Capture of Challenging Human Performances using Multi-Modal References (CVPR Oral 2021)**](https://arxiv.org/abs/2103.06747)\u003cbr\u003e\nYannan He, Anqi Pang, Xin Chen, Han Liang, Minye Wu, Yuexin Ma, Lan Xu\n\n[**TightCap: 3D Human Shape Capture with Clothing Tightness Field (TOG 2021)**](https://chenxin.tech/TightCap.html)\u003cbr\u003e\nXin Chen, Anqi Pang, Wei Yang, Peihao Wang, Lan Xu, Jingyi Yu\n\n[**AutoSweep: Recovering 3D Editable Objects from a Single Photograph (TVCG 2018)**](https://chenxin.tech/AutoSweep.html)\u003cbr\u003e\nXin Chen, Yuwei Li, Xi Luo, Tianjia Shao, Jingyi Yu, Kun Zhou, Youyi Zheng\n\n[**End-to-end Recovery of Human Shape and Pose (CVPR 2018)**](https://github.com/akanazawa/hmr)\u003cbr\u003e\nAngjoo Kanazawa, Michael J. Black, David W. Jacobs, Jitendra Malik\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fchenfengye%2Fsportscap","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fchenfengye%2Fsportscap","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fchenfengye%2Fsportscap/lists"}