{"id":27961725,"url":"https://github.com/cmu-perceptual-computing-lab/panoptic-toolbox","last_synced_at":"2025-05-07T19:11:13.856Z","repository":{"id":41186778,"uuid":"68020994","full_name":"CMU-Perceptual-Computing-Lab/panoptic-toolbox","owner":"CMU-Perceptual-Computing-Lab","description":null,"archived":false,"fork":false,"pushed_at":"2024-05-15T09:26:08.000Z","size":7211,"stargazers_count":392,"open_issues_count":52,"forks_count":76,"subscribers_count":26,"default_branch":"master","last_synced_at":"2024-05-16T00:05:17.653Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"http://domedb.perception.cs.cmu.edu/","language":"Jupyter Notebook","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/CMU-Perceptual-Computing-Lab.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2016-09-12T15:19:02.000Z","updated_at":"2024-05-15T09:26:12.000Z","dependencies_parsed_at":"2024-05-15T22:44:51.684Z","dependency_job_id":null,"html_url":"https://github.com/CMU-Perceptual-Computing-Lab/panoptic-toolbox","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/CMU-Perceptual-Computing-Lab%2Fpanoptic-toolbox","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/CMU-Perceptual-Computing-Lab%2Fpanoptic-toolbox/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/CMU-Perceptual-Computing-Lab%2Fpanoptic-toolbox/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/CMU-Perceptual-Computing-Lab%2Fpanoptic-toolbox/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/CMU-Perceptual-Computing-Lab","download_url":"https://codeload.github.com/CMU-Perceptual-Computing-Lab/panoptic-toolbox/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":252940934,"owners_count":21828769,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2025-05-07T19:11:13.094Z","updated_at":"2025-05-07T19:11:13.829Z","avatar_url":"https://github.com/CMU-Perceptual-Computing-Lab.png","language":"Jupyter Notebook","funding_links":[],"categories":[],"sub_categories":[],"readme":"# PanopticStudio Toolbox\n\nThis repository has a toolbox to download, process, and visualize the [Panoptic Studio](http://domedb.perception.cs.cmu.edu) (Panoptic) data.\n\n# Note:\n\n- May-14-2024: We have built a mirror server at [SNU](http://vcl.snu.ac.kr/panoptic). \n  You may still use the original CMU server, and if the CMU server doesn't respond, You can use the SNU endpoint by simply adding `--snu-endpoint` option in `getData.sh` and `getData_kinoptic` scripts.\n\n# Quick start guide\n\nFollow these steps to set up a simple example:\n\n## 1. Check out the codebase\n\n```\ngit clone https://github.com/CMU-Perceptual-Computing-Lab/panoptic-toolbox\ncd panoptic-toolbox\n```\n\n## 2. Download a sample data and other data\n\nTo download a dataset, named \"171204_pose1_sample\" in this example, run the following script.\n\n```\n./scripts/getData.sh 171204_pose1_sample\n```\n\nThis bash script requires curl or wget.\n\nThis script will create a folder \"./171204_pose1_sample\" and download the following files.\n\n- 171204_pose1_sample/hdVideos/hd_00_XX.mp4 #synchronized HD video files (31 views)\n- 171204_pose1_sample/vgaVideos/KINECTNODE%d/vga_XX_XX.mp4 #synchrponized VGA video files (480 views)\n- 171204_pose1_sample/calibration_171204_pose1_sample.json #calibration files\n- 171204_pose1_sample/hdPose3d_stage1_coco19.tar #3D Body Keypoint Data (coco19 keypoint definition)\n- 171204_pose1_sample/hdFace3d.tar #3D Face Keypoint Data\n- 171204_pose1_sample/hdHand3d.tar #3D Hand Keypoint Data\n\nNote that this sample example currently does not have VGA videos.\n\nYou can also download any other seqeunce through this script. Just use the the name of the target sequence: instead of the \"171204_pose1panopticHD\".\nr example,\n\n```\n./scripts/getData.sh 171204_pose1\n```\n\nfor the full version of [171204_pose1](http://domedb.perception.cs.cmu.edu/171204_pose1.html) sequence:.\nYou can also specify the number of videospanopticHDnt to donwload.\n\n```\n./scripts/getData.sh (sequenceName) (VGA_Video_Number) (HD_Video_Number)\n```\n\nFor example, the following command will download 240 vga videos and 10 videos.\n\n```\n./scripts/getData.sh 171204_pose1_sample 240 10\n```\n\nNote that we have sorted the VGA camera order so that you download uniformly distributed view.\n\n## 3. Downloading All Available Sequences\n\nYou can find the list of currently available sequences in the following link:\n\n[List of released sequences (ver1.2)](https://docs.google.com/spreadsheets/d/1eoe74dHRtoMVVFLKCTJkAtF8zqxAnoo2Nt15CYYvHEE/edit#gid=1333444170)\n\nDownloading all of them (including videos) may take a long time, but downloading 3D keypoint files (body+face+hand upon their availability) should be \"relatively\" quick.\n\nYou can use the following script to download currently available sequences (ver 1.2):\n\n```\n./scripts/getDB_panopticHD_ver1_2.sh\n```\n\nThe default setting is not downloading any videos. Feel free to change the \"vgaVideoNum\" and \"hdVideoNum\" in the script to other numbers if you also want to download videos.\n\nYou can see the example videos and other information of each sequence: in our website:\n[Browsing dataset](http://domedb.perception.cs.cmupanopticHDtaset.html).\n\nCheck the 3D viewer in each sequence: page where you can visualize 3D skeletons in your web browser. For example:\nhttp://domedb.perception.cs.cmu.edu/panopticHDpose1.html\n\n## 4. Extract the images \u0026 3D keypoint data\n\nThis step requires [ffmpeg](https://ffmpeg.org/).\n\n```\n./scripts/extractAll.sh 171204_pose1_sample\n```\n\nThis will extract images, for example `171204_pose1_sample/hdImgs/00_00/00_00_00000000.jpg`, and the corresponding 3D skeleton data, for example `171204_pose1_sample/hdPose3d_stage1_coco19/body3DScene_00000000.json`.\n\n`extractAll.sh` is a simple script that combines the following set of commands (you shouldn't need to run these again):\n\n```bash\ncd 171204_pose1_sample\n../scripts/vgaImgsExtractor.sh # PNG files from VGA video (25 fps)\n../scripts/hdImgsExtractor.sh # PNG files from HD video (29.97 fps)\ntar -xf vgaPose3d_stage1.tar # Extract skeletons at VGA framerate\ntar -xf hdPose3d_stage1.tar # Extract skeletons for HD\ncd ..\n```\n\n## 5. Run demo programs\n\n### Python\n\nThis codes require numpy, matplotlib.\n\nVisualizing 3D keypoints (body, face, hand):\n\n```\ncd python\njupyter notebook demo_3Dkeypoints_3dview.ipynb\n```\n\nThe result should look like [this](https://github.com/CMU-Perceptual-Computing-Lab/panopticapi_d/blob/master/python/demo_3Dkeypoints_3dview.ipynb).\n\nReprojecting 3D keypoints (body, face, hand) on a selected HD view:\n\n```\ncd python\njupyter notebook demo_3Dkeypoints_reprojection_hd.ipynb\n```\n\nThe result should look like [this](https://github.com/CMU-Perceptual-Computing-Lab/panopticapi_d/blob/master/python/demo_3Dkeypoints_reprojection_hd.ipynb).\n\nThis codes require numpy, matplotlib.\n\nVisualizing 3D keypoints (body, face, hand):\n\n```\ncd python\njupyter notebook demo_3Dkeypoints_3dview.ipynb\n```\n\nThe result should look like [this](https://github.com/CMU-Perceptual-Computing-Lab/panopticapi_d/blob/master/python/demo_3Dkeypoints_3dview.ipynb).\n\nReprojecting 3D keypoints (body, face, hand) on a selected HD view:\n\n```\ncd python\njupyter notebook demo_3Dkeypoints_reprojection_hd.ipynb\n```\n\nThe result should look like [this](https://github.com/CMU-Perceptual-Computing-Lab/panopticapi_d/blob/master/python/demo_3Dkeypoints_reprojection_hd.ipynb).\n\n### Python + OpengGL\n\n- This codes require pyopengl.\n\n- Visualizing 3D keypoints (body, face, hand):\n\n```\npython glViewer.py\n```\n\n### Matlab\n\nNote: Matlab code is outdated, and does not handle 3D keypoint outputs (coco19 body, face, hand).\nPlease see this code only for reference. We will update this later.\n\nMatlab example (outdated):\n\n```matlab\n\u003e\u003e\u003e cd matlab\n\u003e\u003e\u003e demo\n```\n\n## Skeleton Output Format\n\nWe reconstruct 3D skeleton of people using the method of [Joo et al. 2018](https://ieeexplore.ieee.org/document/8187699).\n\nThe output of each frame is written in a json file. For example,\n\n```\n{ \"version\": 0.7,\n\"univTime\" :53541.542,\n\"fpsType\" :\"hd_29_97\",\n\"bodies\" :\n[\n{ \"id\": 0,\n\"joints19\": [-19.4528, -146.612, 1.46159, 0.724274, -40.4564, -163.091, -0.521563, 0.575897, -14.9749, -91.0176, 4.24329, 0.361725, -19.2473, -146.679, -16.1136, 0.643555, -14.7958, -118.804, -20.6738, 0.619599, -22.611, -93.8793, -17.7834, 0.557953, -12.3267, -91.5465, -6.55368, 0.353241, -12.6556, -47.0963, -4.83599, 0.455566, -10.8069, -8.31645, -4.20936, 0.501312, -20.2358, -147.348, 19.1843, 0.628022, -13.1145, -120.269, 28.0371, 0.63559, -20.1037, -94.3607, 30.0809, 0.625916, -17.623, -90.4888, 15.0403, 0.327759, -17.3973, -46.9311, 15.9659, 0.419586, -13.1719, -7.60601, 13.4749, 0.519653, -38.7164, -166.851, -3.25917, 0.46228, -28.7043, -167.333, -7.15903, 0.523224, -39.0433, -166.677, 2.55916, 0.395965, -30.0718, -167.264, 8.18371, 0.510041]\n}\n] }\n```\n\nHere, each subject has the following values.\n\n**id**: a unique subject index within a sequence:. Skeletons with the same id across time represent temporally associated moving skeletons (an individual). However, the same person may have multiple ids **joints19**: 19 3D joint locations, formatted as [x1,y1,z1,c1,x2,y2,z2,c2,...] where each c ispanopticHDjoint confidence score.\n\nThe 3D skeletons have the following keypoint order:\n\n```\n0: Neck\n1: Nose\n2: BodyCenter (center of hips)\n3: lShoulder\n4: lElbow\n5: lWrist,\n6: lHip\n7: lKnee\n8: lAnkle\n9: rShoulder\n10: rElbow\n11: rWrist\n12: rHip\n13: rKnee\n14: rAnkle\n15: lEye\n16: lEar\n17: rEye\n18: rEar\n```\n\nNote that this is different from [OpenPose output order](https://github.com/CMU-Perceptual-Computing-Lab/openpose/blob/master/doc/output.md), although our method is based on it.\n\nNote that we used to use an old format (named mpi15 as described in our [outdated document](http://domedb.perception.cs.cmu.edu/tools.html)), but we do not this format anymore.\n\n# KinopticStudio Toolbox\n\nKinoptic Studio is a subsystem of Panoptic Studio, which is composed of 10 Kinect2 sensors.\nPlease see: [README_kinoptic](README_kinoptic.md)\n\n# Panoptic 3D PointCloud DB ver.1\n\nYou can download all sequences included in our [3D PointCloud DB ver.1](https://docs.google.com/spreadsheets/d/1MsD9ioWBToHWz0E33gzFS5nDDjVHRECE2bZ1vM1ff_I/edit?usp=sharing) using the following script:\n\n```\n./scripts/getDB_ptCloud_ver1.sh\n```\n\n# Haggling DB\n\nWe have released the processed data for the haggling sequence. Please see [Social Signal Processing](https://github.com/CMU-Perceptual-Computing-Lab/ssp) repository.\n\n![Teaser Image](https://github.com/jhugestar/jhugestar.github.io/blob/master/img/cvpr19_ssp.gif)\n\n# License\n\nPanoptic Studio Dataset is freely available for non-commercial and research purpose only.\n\n# References\n\nBy using the dataset, you agree to cite at least one of the following papers.\n\n```\n@inproceedings{Joo_2015_ICCV,\nauthor = {Joo, Hanbyul and Liu, Hao and Tan, Lei and Gui, Lin and Nabbe, Bart and Matthews, Iain and Kanade, Takeo and Nobuhara, Shohei and Sheikh, Yaser},\ntitle = {Panoptic Studio: A Massively Multiview System for Social Motion Capture},\nbooktitle = {ICCV},\nyear = {2015} }\n\n@inproceedings{Joo_2017_TPAMI,\ntitle={Panoptic Studio: A Massively Multiview System for Social Interaction Capture},\nauthor={Joo, Hanbyul and Simon, Tomas and Li, Xulong and Liu, Hao and Tan, Lei and Gui, Lin and Banerjee, Sean and Godisart, Timothy Scott and Nabbe, Bart and Matthews, Iain and Kanade, Takeo and Nobuhara, Shohei and Sheikh, Yaser},\njournal={IEEE Transactions on Pattern Analysis and Machine Intelligence},\nyear={2017} }\n\n@inproceedings{Simon_2017_CVPR,\ntitle={Hand Keypoint Detection in Single Images using Multiview Bootstrapping},\nauthor={Simon, Tomas and Joo, Hanbyul and Sheikh, Yaser},\njournal={CVPR},\nyear={2017} }\n\n@inproceedings{joo2019ssp,\n  title={Towards Social Artificial Intelligence: Nonverbal Social Signal Prediction in A Triadic Interaction},\n  author={Joo, Hanbyul and Simon, Tomas and Cikara, Mina and Sheikh, Yaser},\n  booktitle={CVPR},\n  year={2019}\n}\n\n\n\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcmu-perceptual-computing-lab%2Fpanoptic-toolbox","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fcmu-perceptual-computing-lab%2Fpanoptic-toolbox","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcmu-perceptual-computing-lab%2Fpanoptic-toolbox/lists"}