{"id":16320257,"url":"https://github.com/meyerls/pegasus","last_synced_at":"2025-03-16T14:31:01.339Z","repository":{"id":213129398,"uuid":"733106665","full_name":"meyerls/PEGASUS","owner":"meyerls","description":"[IROS24] Offical repository for \"PEGASUS: Physically Enhanced Gaussian Splatting Simulation System for 6DoF Object Pose Dataset Generation\"","archived":false,"fork":false,"pushed_at":"2024-07-18T09:41:02.000Z","size":6635,"stargazers_count":35,"open_issues_count":1,"forks_count":0,"subscribers_count":8,"default_branch":"main","last_synced_at":"2025-02-27T10:36:21.471Z","etag":null,"topics":["6dof-pose","dataset-generation","dataset-generator","gaussian-splatting","novel-view-synthesis","pegasus","pose-estimation"],"latest_commit_sha":null,"homepage":"https://meyerls.github.io/pegasus_web/","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/meyerls.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2023-12-18T15:20:54.000Z","updated_at":"2024-11-13T16:11:22.000Z","dependencies_parsed_at":"2024-07-18T11:51:50.804Z","dependency_job_id":null,"html_url":"https://github.com/meyerls/PEGASUS","commit_stats":null,"previous_names":["meyerls/pegasus"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/meyerls%2FPEGASUS","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/meyerls%2FPEGASUS/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/meyerls%2FPEGASUS/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/meyerls%2FPEGASUS/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/meyerls","download_url":"https://codeload.github.com/meyerls/PEGASUS/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":243818195,"owners_count":20352629,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["6dof-pose","dataset-generation","dataset-generator","gaussian-splatting","novel-view-synthesis","pegasus","pose-estimation"],"created_at":"2024-10-10T22:43:43.601Z","updated_at":"2025-03-16T14:30:58.728Z","avatar_url":"https://github.com/meyerls.png","language":"Python","readme":"# \u003cimg align=\"left\" style=\"padding: 5px\" src=\"assets/pegasus_eye.png\"  height=\"120px\"/\u003e \u003cem\u003ePEGASUS\u003c/em\u003e: Physically Enhanced Gaussian Splatting Simulation System for 6DoF Object Pose Dataset Generation\n\nLukas Meyer*, Floris Erich, Yusuke Yoshiyasu, Marc Stamminger, Noriaki Ando, Yukiyasu Domae\u003cbr\u003e\n\u003cspan style=\"font-size:0.5em;\"\u003e*This work was conducted during an internship at the National Institute\nof Advanced Industrial Science and Technology.\u003c/span\u003e\n\u003cbr\u003e\n\n| [Webpage](https://meyerls.github.io/pegasus_web) | [Full Paper](https://arxiv.org/abs/2401.02281) |  [Ramen Dataset (~50 GB)](https://zenodo.org/records/12624886)  |  [PEGASET (~50 GB)](https://zenodo.org/records/12625040)  |\u003cbr\u003e\n![Teaser image](assets/title.png)\n\n*We introduce Physical Enhanced Gaussian Splatting Simulation System (PEGASUS) for 6DOF object pose dataset\ngeneration, a versatile dataset generator based on 3D Gaussian Splatting. Preparation starts by separate scanning of\nboth environments and objects. PEGASUS allows the composition of new scenes by merging the respective underlying\nGaussian Splatting point cloud of an environment with one or multiple objects. Leveraging a physics engine enables the\nsimulation of natural object placement within a scene by interacting with their\nextracted mesh. Consequently, an extensive amount of new scenes - static or dynamic - can be created by combining\ndifferent environments and objects. By rendering scenes from various perspectives, diverse data points such as RGB\nimages, depth maps, semantic masks, and 6DoF object poses can be extracted. Our study demonstrates that training on data\ngenerated by PEGASUS enables pose estimation networks to successfully transfer from synthetic data to real-world data.\nMoreover, we introduce the CupNoodle dataset, comprising 30 Japanese cup noodle items. This dataset includes spherical\nscans that captures images from both object hemisphere and the Gaussian Splatting reconstruction, making them compatible\nwith PEGASUS.*\n\n\u003cp align=\"center\"\u003e\n\u003ca href=\"https://unit.aist.go.jp/icps/icps-am/en/\"\u003e\u003cimg style=\"padding: 10px\" height=\"150px\" src=\"assets/aist.png\"\u003e \u003c/a\u003e\n\u003ca href=\"https://www.fau.eu/\"\u003e\u003cimg style=\"padding: 10px\" height=\"150px\" src=\"assets/FAU.png\"\u003e \u003c/a\u003e\n\u003ca href=\"https://www.lgdv.tf.fau.de/\"\u003e\u003cimg style=\"padding: 10px\" height=\"150px\" src=\"assets/vce.svg\"\u003e \u003c/a\u003e\n\u003c/p\u003e\n\n\n## Funding and Acknowledgments\n\nThis paper is one of the achievements of joint research with and is owned\ncopyrighted material of ROBOT Industrial Basic Technology Collaborative\nInnovation Partnership. This research has been supported by the New Energy\nand Industrial Technology Development Organization (NEDO), under the\nproject ID JPNP20016.\n\n## Cloning the Repository\n\nThe repository contains submodules, thus please check it out with\n\n```shell\ngit clone https://github.com/meyerls/PEGASUS.git --recursive # HTTPS\ngit submodule update --init --recursive\n```\n\n## Requirements\n\nThe coda has been tested with the following dependencies:\n\n- Python 3.8\n- Cuda 11.6\n- PyTorch 1.12.1\n\n## Setup\n\nOur default, provided install method is based on Conda package and is provided by the following script. This script has\nto be executed in the top layer of the repository. Currently, the setup script has only be tested on Ubuntu 20. An\ninstallation on windows should be possible but will not be provided in this repo.\n\n```shell\n./setup.sh\n```\n\n## Overview\n\n![Teaser image](assets/pegasus.png)\n\nPEGASUS contains of three main components:\n\n- GS Base Environment reconstruction\n- GS Object Reconstruction\n- PEGASUS Dataset Extraction\n\n### GS Base Environment reconstruction\n\n\u003cdetails\u003e\n  \u003csummary\u003eClick me\u003c/summary\u003e\n\nWill be updated soon\n\n\u003c/details\u003e\n\n### GS Object Reconstruction\n\n\u003cdetails\u003e\n  \u003csummary\u003eClick me\u003c/summary\u003e\n\nWill be updated soon! Not yet complete\n\nFor object reconstruction we provide two different processing weights. The first is scanning objects in the wild by\ntaking videos from both sides of the object and the second one is using a camera rig to scan the object on a turntable.\nThe first approach uses [XMEM](https://github.com/hkchengrex/XMem/tree/main) to create a segmentation mask of the\nselected object. For scanning one has to place only an aruco marker into the scene to obtain the correct scale.\nThe turntable approach uses an arbitrary calibration object (I have used a texture rich paper with an aruco marker) to\nreuse its precomputed camera poses. A detailed workflow is provided in the following section.\n\n#### In the Wild scanning\n\nThe workflow for scanning objects in the wild is:\n\n###### 1. Select Object\n\nCurrently it does not work for texture poor objects. Therefore the camera rig is more suitable. The reason is that\ncomputing the poses and also registering images from the bottom view does simply not work with COLMAP. Place the object\nonto a planer scene such as a table and make sure to move all around the object.\n\n###### 2. Aruco Marker\n\nPrint out an aruco marker and place it next to the object. For scaling the object measure and note down the size of the\nsquare aruco marker. A website to create aruco marker can be found [here](https://chev.me/arucogen/).\n\n###### 3. Scanning\n\nRecord two videos with your phone camera or DSLR camera (We have used an iphone 12 in our example). The first video\ncontains a hemispherical scan of the top view of the object. Try to cover a 360 degree view at 2-3 different height\nlevels. For the second video this process must be repeated for the flipped object.\n\n###### 4. Segmentation Mask\n\nFor extracting the semantic masks of the video we used [XMEM](https://github.com/hkchengrex/XMem).\n\nXMEM can be started from the root directory of PEGASUS:\n\n```shell\npython submodules/XMem/interactive_demo.py --video[path to the video] --num_objects 1 --size -1\n```\n\n\u003cimg  align=\"right\" style=\"padding: 10px\" src=\"assets/xmem.png\" alt=\"drawing\" width=\"400px\"/\u003e\n\nIn the XMEM GUI select the object you want to extract (the object should be highlighted in red). Afterward click the\nbutton *Forward Propagate* (\u003cfont color='red'\u003e1\u003c/font\u003e) to extract the masks. Depending on the video length it takes\naround 1-2 min. To save the detected masks click on *Export Overlays as Video* (\u003cfont color='red'\u003e2\u003c/font\u003e) to save the\nbinary masks as images. More info on how to use XMEM can be\nfound [here](https://github.com/hkchengrex/XMem/blob/main/docs/DEMO.md).\n\nNote: please select the image size according to your GPU size or the quality you want to get. -1 uses the original image\nsize. If you set a value it will resize the image according to its shorter side.\n\n###### 6. Dataset Integration\n\nFirst both extracted images and masks have to be put into a common folder. This folder should be placed in a dataset\nfolder where multiple reconstructed objects can be stored.\n\n```shell\n.\n└── bouillon \n    ├── down\n    │   ├── images\n    │   ├── masks\n    └── up\n        ├── images\n        └── masks\n\n```\n\nTo use the scanned object and included it in PEGASUS one has to define the object as a Dataset-Object\nin [in_the_wild_dataset.py](src/dataset/in_the_wild_dataset.py). The class (here *Bouillon*) name takes the name of the\nobject.\n\n```python\nclass Bouillon(InTheWild):\n    OBJECT_NAME = 'bouillon'\n    ID = 201\n    TYPE = 'object'\n    RECORDING_TYPE = 'spherical'  # 'spherical' or 'hemispherical'\n    ALPHA = 0.3\n    DATASET_TYPE = 'wild'\n    ARUCO_SIZE = 0.037  # in meter\n\n    def __init__(self, dataset_path):\n        super().__init__(dataset_path=Path(dataset_path))\n```\n\n\u003cimg  align=\"right\" style=\"padding: 10px\" src=\"assets/reconstruction.png\" alt=\"drawing\" width=\"400px\"/\u003e\n\n- ```OBJECT_NAME```: folder name of the object. By default it is the video name in the ```./workspace``` folder (this\n  folder gets generated by XMEM). Please rename to the object name.\n- ```ID```: Unique object ID\n- ```TYPE```: default type is object. Differs for environment (default: object)\n- ```RECORDING_TYPE```: 'spherical' or 'hemispherical' depending on if you also scanned the bottom. This is recommend if\n  you\n  have texture-less objects. 'spherical': 2 videos (top \u0026 bottom). 'hemispherical': 1 video (top only)\n- ```ALPHA```: Value for alpha shape reconstruction (default: 0.3)\n- ```DATASET_TYPE```: name for your own dataset (default: wild)\n- ```ARUCO_SIZE```: size of the aruco marker in meter(!)\n\n###### 7. GS Reconstruction\n\n```shell\npython src/reconstruction/in_the_wild_object_reconstruction.py\n```\n\n###### 8. Integrate into PEGASUS\n\n- Tbd\n\n\u003c/details\u003e\n\n### Available Objects (Ramen Dataset and PEGASET)\n\nWe provide two different datasets. The IDs for the *Ramen* dataset are between 101 and 130. The YCB-V IDs are identical to the original YCB-V ids.\n\n#### *Ramen* Dataset\n\nThe [*Ramen* Dataset](https://google.de) contains out of 30 cup noodle objects and 9\nenvironments.\n\n\u003cp align=\"center\"\u003e\n  \u003cimg style=\"padding: 5px\" src=\"assets/konbini_dataset.jpg\" alt=\"drawing\" width=\"500px\"/\u003e \n\u003c/p\u003e\n\u003cp align=\"center\"\u003e\n\u003cimg style=\"padding: 5px\" src=\"assets/cup_noodle_04.gif\" alt=\"drawing\" width=\"250px\"/\u003e \u003cimg style=\"padding: 5px\" src=\"assets/cup_noodle_15.gif\" alt=\"drawing\" width=\"250px\"/\u003e \n\u003c/p\u003e\n\n```shell\n.\n└── Dataset \n    ├── calibration\n    │   ├── ...\n    ├── environment\n    │   ├── ...\n    ├── object\n    │   ├── ...\n    └── urdf\n        └── ...\n```\n\n#### PEGASET\n\nThe [PEGASET](https://zenodo.org/records/12625040) contains out of the well known 21 YCB-V and 9 environments. \n\n\u003cp align=\"center\"\u003e\n  \u003cimg style=\"padding: 5px\" src=\"assets/appendix_ycb.png\" alt=\"drawing\" width=\"500px\"/\u003e \n\u003c/p\u003e\n\n\u003cp align=\"center\"\u003e\n\u003cimg style=\"padding: 5px\" src=\"assets/cracker_box.gif\" alt=\"drawing\" width=\"250px\"/\u003e \u003cimg style=\"padding: 5px\" src=\"assets/yellow_mustard.gif\" alt=\"drawing\" width=\"250px\"/\u003e\n\u003c/p\u003e\n\n```shell\n.\n└── Dataset \n    ├── calibration\n    │   ├── ...\n    ├── environment\n    │   ├── ...\n    ├── object\n    │   ├── ...\n    └── urdf\n        └── ...\n```\n\n### PEGASUS Dataset Extraction\n\nBefore rendering a dataset the dataset provided for PEGASUS must have been downloaded\nfrom [*Ramen* Dataset](https://zenodo.org/records/12624886) or [PEGASET](https://zenodo.org/records/12625040). If you use both dataset you should merge both into one folder structure.\n\nAll objects and environments which are relevant for dataset generation should be added to the ```obj_list``` and ```env_list```.\n\nParameters:\n\n- ```mode: str```: Either ```\"dynamic\"``` or ```\"static\"``` rendering of scene\n- ```num_scenes: int```: Number of scenes\n- ```num_objects: int```: Maximum number of objects which are placed in the scene. A random number between 1 and ```num_objects``` is choosen.\n- ```image_height```:    \n- ```image_width```:\n- ```render_data_points: list```: Types of rendering and data points saved to output. e.g. ```['rgb', 'depth', 'seg_vis', 'seg_sil', 'sem_seg']```\n- ```convert_from_scenewise2imagewise: bool```: By default the scene is saved per scene. If you need the data in sceneweise BOP-Format set to ```True```\n\n## BibTex\n\n```\n@Article{PEGASUS2024,\n      author       = {Meyer, Lukas and Erich, Floris and Yoshiyasu, Yusuke and Stamminger, Marc and Ando, Noriaki and Domae, Yukiyasu },\n      title        = {PEGASUS: Physical Enhanced Gaussian Splatting Simulation System for 6DOF Object Pose Dataset Generation},\n      journal      = {IROS},\n      month        = {October},\n      year         = {2024},\n      url          = {https://meyerls.github.io/pegasus_web}\n}\n```\n\nThanks to the authors of [3D Gaussians](https://repo-sam.inria.fr/fungraph/3d-gaussian-splatting/) for their excellent\ncode, please consider to also cite this repository:\n\n```\n@Article{kerbl3Dgaussians,\n      author       = {Kerbl, Bernhard and Kopanas, Georgios and Leimk{\\\"u}hler, Thomas and Drettakis, George},\n      title        = {3D Gaussian Splatting for Real-Time Radiance Field Rendering},\n      journal      = {ACM Transactions on Graphics},\n      number       = {4},\n      volume       = {42},\n      month        = {July},\n      year         = {2023},\n      url          = {https://repo-sam.inria.fr/fungraph/3d-gaussian-splatting/}\n}\n```\n\nAnd thanks to authors of the BOP Toolkit for their benchmark for 6D object pose estimation interface.","funding_links":[],"categories":[],"sub_categories":[],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmeyerls%2Fpegasus","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fmeyerls%2Fpegasus","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmeyerls%2Fpegasus/lists"}