https://github.com/chenfengye/sportscap
[IJCV 2021] SportsCap: Monocular 3D Human Motion Capture and Fine-grained Understanding in Challenging Sports Videos
https://github.com/chenfengye/sportscap
action-recognition dataset deep-learning ijcv2021 motion-capture motion-prior sport sports-analytics sports-data
Last synced: 7 months ago
JSON representation
[IJCV 2021] SportsCap: Monocular 3D Human Motion Capture and Fine-grained Understanding in Challenging Sports Videos
- Host: GitHub
- URL: https://github.com/chenfengye/sportscap
- Owner: ChenFengYe
- Created: 2021-06-28T03:45:58.000Z (over 4 years ago)
- Default Branch: main
- Last Pushed: 2021-08-17T13:09:39.000Z (about 4 years ago)
- Last Synced: 2025-03-16T01:53:16.605Z (7 months ago)
- Topics: action-recognition, dataset, deep-learning, ijcv2021, motion-capture, motion-prior, sport, sports-analytics, sports-data
- Language: Python
- Homepage: https://chenxin.tech/SportsCap.html
- Size: 17.3 MB
- Stars: 125
- Watchers: 7
- Forks: 13
- Open Issues: 1
-
Metadata Files:
- Readme: README.md
Awesome Lists containing this project
README
# SportsCap: Monocular 3D Human Motion Capture and Fine-grained Understanding in Challenging Sports Videos
### [ProjectPage](https://chenxin.tech/SportsCap.html) | [Paper](https://arxiv.org/abs/2104.11452) | [Video](https://www.youtube.com/watch?v=TrqAaaX97KY) | Dataset ([Part01](https://drive.google.com/file/d/1hUlGglrlWdjZNFFQh2ck3UaMDL8sDwQv/view?usp=sharing)|[Part02](https://drive.google.com/file/d/1NswiD-wpuAyHbSgdUCZ2s9QmVIFPcLjA/view?usp=sharing))
[Xin Chen](https://chenxin.tech/), Anqi Pang, [Wei Yang](https://scholar.google.com/citations?user=fRjxdPgAAAAJ&hl=en), [Yuexin Ma](http://yuexinma.me/aboutme.html), [Lan Xu](http://xu-lan.com/), [Jingyi Yu](http://www.yu-jingyi.com/).
This repository contains the official implementation for the paper: [SportsCap: Monocular 3D Human Motion Capture and Fine-grained Understanding in Challenging Sports Videos (IJCV 2021)](https://arxiv.org/abs/2104.11452). Our work is capable of simultaneously capturing 3D human motions and understanding fine-grained actions from monocular challenging sports video input.
## Abstract
Markerless motion capture and understanding of professional non-daily human movements is an important yet unsolved task, which suffers from complex motion patterns and severe self-occlusion, especially for the monocular setting. In this paper, we propose SportsCap -- the first approach for simultaneously capturing 3D human motions and understanding fine-grained actions from monocular challenging sports video input. Our approach utilizes the semantic and temporally structured sub-motion prior in the embedding space for motion capture and understanding in a data-driven multi-task manner. Comprehensive experiments on both public and our proposed datasets show that with a challenging monocular sports video input, our novel approach not only significantly improves the accuracy of 3D human motion capture, but also recovers accurate fine-grained semantic action attributes.
[comment]:
## Licenses
This work is licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License.
All material is made available under [Creative Commons BY-NC-SA 4.0](https://creativecommons.org/licenses/by-nc-sa/4.0/legalcode) license. You can **use, redistribute, and adapt** the material for **non-commercial purposes**, as long as you give appropriate credit by **citing our paper** and **indicating any changes** that you've made.
## The SMART Dataset
SportsCap proposes a challenging sports dataset called Sports Motion and Recognition Tasks (SMART) dataset, which contains per-frame action labels, manually annotated pose, and action assessment of various challenging sports video clips from professional referees.
### Download
You can download the SMART dataset (17 GB, version 1.0) from the Google Drive [[SMART_part01](https://drive.google.com/file/d/1hUlGglrlWdjZNFFQh2ck3UaMDL8sDwQv/view?usp=sharing) | [SMART_part02](https://drive.google.com/file/d/1NswiD-wpuAyHbSgdUCZ2s9QmVIFPcLjA/view?usp=sharing)]. The SMART dataset includes source images (>60,000), annotations(>45,000, both pose and action), sport motion embedding spaces, videos (coming soon) and tools.
### Annotation
Please load these JSON files in python to parse these annotations about 2D key-points of poses and fine-grained action labels.
```
Table_VideoInfo_diving.json
Table_VideoInfo_gym.json
Table_VideoInfo_polevalut_highjump_badminton.json
```
### Tools
The tools folder includes several functions to load the annotation and calculate the pose variables. More useful scripts are coming soon.
```
utils.py - json_load, crop_img_skes, cal_body_bbox ...
```
## Sports Motion Embedding Spaces
With the annotated 2D poses and MoCap 3D pose data, we collect the Sports Motion Embedding Spaces (SMES), the 2D/3D pose priors for various sports. SMES provides strong prior and regularization to ensure that the generated pose result lies in the corresponding action space.
### Download
You can download the Motion Embedding Spaces (SMES) (7 MB, version 1.0) separately from [GoogleDrive](https://drive.google.com/file/d/1gWyBxMUrImkWoz8YWIm_XrCsGpxovK0Q/view?usp=sharing). The released SMES-V1.0 includes many sports, like vault, uneven bar, boxing, diving, hurdles, pole vault, high jump, and so on.
### Usage
Coming soon.
## Citation
If you find our code or paper useful, please consider citing:
```
@article{chen2021sportscap,
title={SportsCap: Monocular 3D Human Motion Capture and Fine-Grained Understanding in Challenging Sports Videos},
author={Xin Chen and Anqi Pang and Wei Yang and Yuexin Ma and Lan Xu and Jingyi Yu},
journal={International Journal of Computer Vision},
year={2021},
month={Aug},
url={https://doi.org/10.1007/s11263-021-01486-4}
}
```
## Relevant Works
[**ChallenCap: Monocular 3D Capture of Challenging Human Performances using Multi-Modal References (CVPR Oral 2021)**](https://arxiv.org/abs/2103.06747)
Yannan He, Anqi Pang, Xin Chen, Han Liang, Minye Wu, Yuexin Ma, Lan Xu
[**TightCap: 3D Human Shape Capture with Clothing Tightness Field (TOG 2021)**](https://chenxin.tech/TightCap.html)
Xin Chen, Anqi Pang, Wei Yang, Peihao Wang, Lan Xu, Jingyi Yu
[**AutoSweep: Recovering 3D Editable Objects from a Single Photograph (TVCG 2018)**](https://chenxin.tech/AutoSweep.html)
Xin Chen, Yuwei Li, Xi Luo, Tianjia Shao, Jingyi Yu, Kun Zhou, Youyi Zheng
[**End-to-end Recovery of Human Shape and Pose (CVPR 2018)**](https://github.com/akanazawa/hmr)
Angjoo Kanazawa, Michael J. Black, David W. Jacobs, Jitendra Malik