{"id":19855139,"url":"https://github.com/leggedrobotics/rsl_panoptic","last_synced_at":"2025-09-18T14:31:56.719Z","repository":{"id":229289642,"uuid":"765838898","full_name":"leggedrobotics/rsl_panoptic","owner":"leggedrobotics","description":null,"archived":false,"fork":false,"pushed_at":"2024-12-10T08:37:50.000Z","size":8761,"stargazers_count":4,"open_issues_count":0,"forks_count":0,"subscribers_count":14,"default_branch":"main","last_synced_at":"2025-04-01T19:37:29.185Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/leggedrobotics.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-03-01T18:09:34.000Z","updated_at":"2024-12-03T13:39:53.000Z","dependencies_parsed_at":null,"dependency_job_id":"9f85613d-cee6-46ea-adf2-d4337f35b796","html_url":"https://github.com/leggedrobotics/rsl_panoptic","commit_stats":null,"previous_names":["leggedrobotics/rsl_panoptic"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/leggedrobotics/rsl_panoptic","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/leggedrobotics%2Frsl_panoptic","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/leggedrobotics%2Frsl_panoptic/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/leggedrobotics%2Frsl_panoptic/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/leggedrobotics%2Frsl_panoptic/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/leggedrobotics","download_url":"https://codeload.github.com/leggedrobotics/rsl_panoptic/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/leggedrobotics%2Frsl_panoptic/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":275781371,"owners_count":25527359,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-09-18T02:00:09.552Z","response_time":77,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-12T14:11:48.858Z","updated_at":"2025-09-18T14:31:54.928Z","avatar_url":"https://github.com/leggedrobotics.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# M545 Panoptic Segmentation\n\nThis package is for deploying the panoptic segmentation developed by the Robotics Systems Lab.\n\n\u003cimg src=\"https://github.com/leggedrobotics/rsl_panoptic/assets/15898282/a9d77558-02a1-49ba-849e-532573fd5cdc\" alt=\"frame025833_rotated\" style=\"width: 50%;\"/\u003e\u003cimg src=\"https://github.com/leggedrobotics/rsl_panoptic/assets/15898282/1b165512-e324-4993-be34-18c6e660c8fb\" alt=\"image_0002\" style=\"width: 50%;\"/\u003e\n\n\n\n## Panoptic Models \n\nThis section encompasses our efforts in developing state-of-the-art panoptic segmentation models using a supervised learning approache. The models are designed to understand and segment images into meaningful parts, combining both object detection and semantic segmentation tasks to provide a comprehensive view of an image's content.\n\n### Panoptic Segmentation\n\nOur approach utilizes DETR and Mask2Former framework, aiming to generate hierarchical feature embeddings for panoptic segmentation. The models are trained on selected subsets of the COCO Dataset and a specially curated Test Construction Site Dataset, demonstrating their effectiveness in diverse environments. For detailed informations about the training data setup check [dataset README](panoptic_models/data/README.md). For detailed instructions on training and inference, refer to the [Mask2Former README](panoptic_models/panoptic_models/mask2former/README.md) and the [DETR README](panoptic_model/panoptic_models/detr/README.md) \n\nKey Highlights:\n- Utilization of COCO Dataset and Construction site data for robust training.\n- Employment of Mask2Former framework for accurate segmentation results.\n- Sample visualizations of segmentation predictions available for reference.\n\n## Panoptic Ros \n\nIntegrating advanced panoptic segmentation capabilities with ROS (Robot Operating System), the Panoptic Ros project provides real-time image segmentation through a dedicated ROS node. It supports various models like DETR and Mask2Former, allowing seamless integration into robotics applications for enhanced environmental understanding.\n\n### ROS Panoptic Segmentation Node\n\nKey Features:\n- Easy integration with ROS for image topic subscription and segmented image publishing.\n- Support for multiple segmentation models with configurable ROS parameters.\n- Real-time visualization options for segmentation results and labels within the ROS environment.\n\nInstallation and Launch:\n- Docker container recommended for ease of setup; alternatively, dependencies are listed in the official DETR and Mask2Former repos.\n- Launch the node using roslaunch command: `roslaunch panoptic_ros image_segmentation.launch`.\n\nParameters and Customization:\n- Various parameters allow for customization of input topics, model selection, and visualization preferences.\n- Extendability by implementing additional models and adjusting image processing steps to meet specific needs.\n\n## Setup with Docker\n\npip install pandas\npip install tensorflow --upgrade\npip install wandb \npip install timm\npip install scikit-learn\n./rsl_panoptic/panoptic_models/panoptic_models/mask2former/third_party/Mask2Former/mask2former/modeling/pixel_decoder/ops/make.sh\n\n### Docker Image\n\nFor building the docker image execute:\n```bash\ndocker build -t rsl_panoptic_seg -f Dockerfile .\n```\nTo run the container and mount it on your home:\n```bash\ndocker run --gpus all --network=\"host\" -it -e HOST_USERNAME=$(whoami) -e HOST_UID=$(id -u) -e HOST_GID=$(id -g) -v /home/$(whoami):/home/$(whoami) rsl_panoptic_seg\n```\n\n## Deploying\n\nDownload the models available in the [Model Zoo](MODEL_ZOO.md) and follow the detailed instructions in the specific model folder for further fine-tuning and inference.\n\n## Dependencies (if no docker image is used)\n\nThis repository uses python 3.9 and assumes you have installed on your machine CUDA 11.1 drivers. Reccomend using  \nPlase use conda to install the dependencies: \n```bash\nconda env create -f ros/src/conda/environment.yml\n```\nThen install the detectron model: \n```bash \npython -m pip install 'git+https://github.com/facebookresearch/detectron2.git'\n```\nthe panoptic segmentation api:\n```bash\npip install git+https://github.com/cocodataset/panopticapi.git\n```\nthe transformers models from huggingface:\n```bash\npip install -q git+https://github.com/huggingface/transformers.git timm\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fleggedrobotics%2Frsl_panoptic","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fleggedrobotics%2Frsl_panoptic","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fleggedrobotics%2Frsl_panoptic/lists"}