{"id":17938856,"url":"https://github.com/robot-learning-freiburg/ros_sam","last_synced_at":"2025-03-24T10:31:44.335Z","repository":{"id":206495822,"uuid":"692038697","full_name":"robot-learning-freiburg/ros_sam","owner":"robot-learning-freiburg","description":"ROS wrapper for Meta's Segment-Anything model","archived":false,"fork":false,"pushed_at":"2024-04-15T15:50:07.000Z","size":4890,"stargazers_count":148,"open_issues_count":2,"forks_count":7,"subscribers_count":6,"default_branch":"main","last_synced_at":"2024-10-29T00:06:35.273Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"CMake","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"gpl-3.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/robot-learning-freiburg.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null}},"created_at":"2023-09-15T12:32:43.000Z","updated_at":"2024-09-25T08:14:26.000Z","dependencies_parsed_at":"2024-04-15T16:57:32.826Z","dependency_job_id":"9b7bf6cf-1c7e-47f0-a75c-ae579274676e","html_url":"https://github.com/robot-learning-freiburg/ros_sam","commit_stats":null,"previous_names":["robot-learning-freiburg/ros_sam"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/robot-learning-freiburg%2Fros_sam","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/robot-learning-freiburg%2Fros_sam/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/robot-learning-freiburg%2Fros_sam/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/robot-learning-freiburg%2Fros_sam/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/robot-learning-freiburg","download_url":"https://codeload.github.com/robot-learning-freiburg/ros_sam/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":245252302,"owners_count":20585007,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-10-29T00:06:24.023Z","updated_at":"2025-03-24T10:31:43.787Z","avatar_url":"https://github.com/robot-learning-freiburg.png","language":"CMake","funding_links":[],"categories":["Research-Grade Frameworks","Softwares and Libraries"],"sub_categories":[],"readme":"# ROS SAM\n\nThis package is what the name suggests: Meta's `segment-anything` wrapped in a ROS node. In this wrapper we offer...\n\n - ROS services for segmenting images using point and box queries.\n - An RQT interface for specifying point queries interactively.\n - A Python client which handles the serialization of queries.\n\n\n## Installation\n\nInstallation is easy: \n 1. Start by cloning this package into your ROS environment. \n 2. Download the [checkpoints](https://github.com/facebookresearch/segment-anything#model-checkpoints) for the desired SAM models to the `models` directory in this package.\n 3. Install SAM by running `pip install git+https://github.com/facebookresearch/segment-anything.git`.\n\n\n## Using ROS SAM standalone\n\nRun the SAM ROS node using `rosrun`:\n\n```bash\nrosrun ros_sam sam_node.py\n```\n\nThe node has two parameters:\n - `~model` SAM model to use, defaults to `vit_h`. Check SAM documentation for options.\n - `~cuda` whether to use CUDA and which device, defaults to `cuda`. Use `cpu` if you have no CUDA.If you want to use a specific GPU, set someting like `cuda:1`.\n\nThe node currently offers a single service `ros_sam/segment` which can be called to segment an image. Check `rossrv show ros_sam/Segmentation` for request and response specifications.\n\nYou can test SAM by starting the node and then running `rosrun ros_sam sam_test.py`. This should yield the following result:\n\n\u003cimg src=\"figures/segmentation-example.png\" width=50% height=50%\u003e\n\n\n### ROS Services\n\n`ros_sam` offers a single service `segment` of the type `ros_sam/Segmentation.srv`. The service definition is\n```\nsensor_msgs/Image        image            # Image to segment\ngeometry_msgs/Point[]    query_points     # Points to start segmentation from\nint32[]                  query_labels     # Mark points as positive or negative samples\nstd_msgs/Int32MultiArray boxes            # Boxes can only be positive samples\nbool                     multimask        # Generate multiple masks\nbool                     logits           # Send back logits\n\n---\n\nsensor_msgs/Image[]   masks            # Masks generated for the query\nfloat32[]             scores           # Scores for the masks\nsensor_msgs/Image[]   logits           # Logit activations of the masks\n```\n\nThe service request takes input image, input point prompts, corresponding labels and the box prompt. The service response contains the segmentation masks, confidence scores and the logit activations of the masks.\n\nTo learn more about the types and use of different queries, please refer to the [original SAM tutorial](https://github.com/facebookresearch/segment-anything/blob/main/notebooks/predictor_example.ipynb)\n\nThe service calls are wrapped up conveniently in the ROS SAM client.\n\n\n## Using with RQT click interface\n\nTo use the GUI install the following in your ROS workspace:\n\n[rqt_image_view_seg](https://github.com/ipab-slmc/rqt_image_view_seg)\n\nRun the launch file:\n\n`roslaunch ros_sam gui_test.launch`\n\nCheck the terminal and wait until the SAM model has finished loading.\n\nThere will be two windows loaded. One will have the header `rqt_image_view_seg__ImageView` and the other `rqt_image_view__ImageView`, note the lack of `_seg`. The first window is where you should click, so select the topic of the camera you want to view from the drop down. In the second window you should select `/rqt_image_segmentation/masked_image`. This is where the segmented image will be displayed.\n\n\u003cimg src=\"figures/interface-example.png\" width=50% height=50%\u003e\n\n\n## Using ROS SAM client\n\nAlternatively, if you don't feel like assembling the service calls yourself, one can use the ROS SAM client instead of the service calls.\n\nInitialize the client with the service name of the SAM segmentation service\n```python\nfrom ros_sam import SAMClient\nsam_client = SAMClient('ros_sam')\n```\n\nCall the segment method with the input image, input prompt points and corresponding labels. This returns 3 segmentation masks for the object and their corresponding confidence scores\n```python\nimg = cv2.imread('path/to/image.png')\npoints = np.array([[100, 100], [200, 200], [300, 300]])\nlabels = [1, 1, 0]\nmasks, scores = sam_client.segment(img, points, labels)\n```\n\nAdditional utilities for visualizing segmentation masks and input prompts\n```python\nfrom ros_sam import show_mask, show_points\nshow_mask(masks[0], plt.gca())\nshow_points(points, np.asarray(labels), plt.gca())\n```\n\n## Citing ROS SAM\nIf you use ROS SAM in your work, please cite our paper:\n```\n@article{buchanan2023online,\n  title={Online Estimation of Articulated Objects with Factor Graphs using Vision and Proprioceptive Sensing},\n  author={Buchanan, Russell and R{\\\"o}fer, Adrian and Moura, Jo{\\~a}o and Valada, Abhinav and Vijayakumar, Sethu},\n  journal={arXiv preprint arXiv:2309.16343},\n  year={2023}\n}\n```\nAnd please also cite the original Segment Anything paper:\n```\n@article{kirillov2023segany,\n  title={Segment Anything},\n  author={Kirillov, Alexander and Mintun, Eric and Ravi, Nikhila and Mao, Hanzi and Rolland, Chloe and Gustafson, Laura and Xiao, Tete and Whitehead, Spencer and Berg, Alexander C. and Lo, Wan-Yen and Doll{\\'a}r, Piotr and Girshick, Ross},\n  journal={arXiv:2304.02643},\n  year={2023}\n}\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Frobot-learning-freiburg%2Fros_sam","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Frobot-learning-freiburg%2Fros_sam","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Frobot-learning-freiburg%2Fros_sam/lists"}