{"id":18792530,"url":"https://github.com/prbonn/hsmcl","last_synced_at":"2025-07-15T22:09:19.483Z","repository":{"id":61013604,"uuid":"535637612","full_name":"PRBonn/hsmcl","owner":"PRBonn","description":null,"archived":false,"fork":false,"pushed_at":"2023-06-29T20:09:28.000Z","size":64074,"stargazers_count":63,"open_issues_count":1,"forks_count":2,"subscribers_count":2,"default_branch":"master","last_synced_at":"2025-04-13T14:47:28.831Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"C++","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/PRBonn.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2022-09-12T11:35:11.000Z","updated_at":"2024-12-16T08:53:32.000Z","dependencies_parsed_at":"2024-11-07T21:21:19.276Z","dependency_job_id":"0e2b46ee-ba3c-48b2-8331-216ecbcf832f","html_url":"https://github.com/PRBonn/hsmcl","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/PRBonn/hsmcl","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/PRBonn%2Fhsmcl","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/PRBonn%2Fhsmcl/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/PRBonn%2Fhsmcl/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/PRBonn%2Fhsmcl/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/PRBonn","download_url":"https://codeload.github.com/PRBonn/hsmcl/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/PRBonn%2Fhsmcl/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":265463913,"owners_count":23770315,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-07T21:20:22.216Z","updated_at":"2025-07-15T22:09:19.436Z","avatar_url":"https://github.com/PRBonn.png","language":"C++","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Long-Term Localization using Semantic Cues in Floor Plan Maps\nThis repository contains the implementation of the following [publication](https://arxiv.org/abs/2210.01456):\n```bibtex\n@article{zimmerman2023ral,\n  author  = {Zimmerman, Nicky and Guadagnino, Tiziano and Chen, Xieyuanli and Behley, Jens and Stachniss, Cyrill},\n  journal = {IEEE Robotics and Automation Letters (RA-L)},\n  title   = {{Long-Term Localization Using Semantic Cues in Floor Plan Maps}},\n  year    = {2023},\n  volume  = {8},\n  number  = {1},\n  pages   = {176-183}\n}\n```\n\n## Overview\nWe provide a long-term localization solution and a semantic map format, for environments that undergo changes to their interior structure and detailed geometric maps are not available.\n\nA visualization of our semantic-guided localization in a previously unseen environment can seen in the following video:\n[![](http://img.youtube.com/vi/ikmiHHIv70s/0.jpg)](https://www.youtube.com/watch?v=ikmiHHIv70s \"HSMCL\")\n\n### Abstract Semantic Maps\nWe enrich the sparse floor plans with geometric, semantic and textual information. \n\n![semmap](Docs/SemMap.png)\n\n### Object Detection \nWe use [YOLOv5](https://github.com/ultralytics/yolov5) for object detection, but in theory it can be replaced by any other model that provides our algorithm with class prediction, bounding boxes and a confidence score. You can download our trained model [here](https://www.ipb.uni-bonn.de/html/projects/hsmcl/2022_05_09.pt).\n![semmap](Docs/YOLOExample.png)\n\n\n### Room Category Classification\nWe consider heirarchical semantic information, by building a higher level of semantic understanding based on the object detection results. We train a KNN classifier to infer the room category based on the objects detected in the scene. The format for the classification.pickle is a dataframe with 'samples' column which stores a one-hot-encoded vector corresponding to the semantic classes of the detected objects, and a 'predictions' column with an index corresponding to the correct room category.\nThe training data for the classifier can be downloaded [here](https://www.ipb.uni-bonn.de/html/projects/hsmcl/classification.pickle).\n\n\n## Installation\nRequirements:\n* ROS 1 - Noetic\n* OpenCV 4.5 \n* Eigen\n* GTest\n* [JSON parser](https://github.com/nlohmann/json)\n* Packages - map_server, amcl, gmapping\n* [YOLOv5](https://github.com/ultralytics/yolov5) - commit hash 7a2a118\n\nTo save you some pain, please follow to OpenCV installation instructions in this [link](https://docs.opencv.org/4.x/d7/d9f/tutorial_linux_install.html).\n\nRequired installations\n```bash\nsudo apt-get install nlohmann-json3-dev\nsudo apt-get install libgtest-dev\nsudo apt install libeigen3-dev\nsudo apt-get install ros-noetic-gmapping\nsudo apt-get install ros-noetic-amcl\nsudo apt-get install ros-noetic-map-server\n```\n\nFirst you need to build the ncore library since I don't have a global CMake yet. Tough life. \u003cbr/\u003e\n```bash\ngit clone https://github.com/PRBonn/hsmcl.git\ncd hsmcl/ncore\nmkdir build \u0026\u0026 cd build\ncmake .. -DBUILD_TESTING=1 \nmake -j12\n```\nTo see everything is built correctly, run the tests from the build directory\n```bash\n./bin/NMCLUnitTests\n```\nTo integrate OpenCV 4.5 to ROS, you must download [vision_opencv](https://github.com/ros-perception/vision_opencv) to the `ros1_ws/src` folder, and then build the workspace\n```bash\ncd ros1_ws/src\ngit clone https://github.com/ros-perception/vision_opencv.git\ncd vision_opencv\ngit checkout noetic\ncd ../..\ncatkin_make\nsource devel/setup.bash\n```\nIf you encounter errors related to OpenCV versions, you can check [this](https://github.com/PRBonn/hsmcl/blob/master/Docs/vision_opencv%20Issues.md).\n\nYou can download our trained model for [object detection](https://www.ipb.uni-bonn.de/html/projects/hsmcl/2022_05_09.pt) and data for [room classification](https://www.ipb.uni-bonn.de/html/projects/hsmcl/classification.pickle). Please place them in `ros1_ws/src/yolov5_ros/models/`.\n\n\n## Semantic Map Editor\nYou can our GUI app [MAPhisto](https://github.com/FullMetalNicky/Maphisto) for editing semantic maps. \nMAPhisto allows the user to add geometric, semantic and textual information to a floor map. \nMAPhisto requires 3 inputs:\n* occupancy grid map (map_server format)\n* room segmentation \n* floor.config\nFor detailed gmapping-style occupancy grid, the room segmentation must be manually annotated. For axis-aligned floor plans, you can extract it with the CreateMapAssests.py script. Using the script will als create a floor.config file.\n\n## Running the Algo\n\nWe use ROS wrappers to run the NMCL algorithm. \n\n### ConfigNMCLNode\nRequires as input 3 topics, LaserScan from the front lidar, LaserScan from the rear lidar, Odometry from VO or Odometry from the wheel encoders. The topic names are configurable through the launch file. This runs the correct and predict steps asynchronously, so correct is executed whenever we have odometry and predict is executed when an observation arrives. ConfigNMCLNodE publishes PoseWithCovarianceStamped messages with the pose prediction. To see the predictions in RVIZ, run\n```bash\nroslaunch nmcl_ros confignmcl.launch dataFolder:=\"path/to/data/folder/\"\n```\nThe data folder used for the experiments can be found in `ncore/data/floor/JMap`.\nThe data folder should have the following items:\n* nmcl.config - defines algorithm parameters, like sampling stratgy, motion/observation model, particle number, etc.  \n* floor.config - defines where the map resources are and semantic/textual information about the map\n* \u003cyour-room-segmentation\u003e.png/pgm/jpg \n* \u003cyour-occupancy-grid-metadata\u003e.yaml (map_server format)\n* \u003cyour-occupancy-grid\u003e.png/pgm/jpg  (map_server format)\n\n\u003cbr/\u003e\nIf you don't stick to this structure and naming, nothing will work. I didn't feel like putting to much effort on parsing and stuff. For an example of the content of these files, you can check out [this](nmcl/tst/data/floor/JMap). The data folder contains mostly things that are environment specific, like maps and algorithms definitions.\n\u003cbr/\u003e\n\nIn addition, there is a configuration folder for sensors configuration. This includes:\n\n* front_laser.config\n* rear_laser.config\n* cam0.config \n* cam1.config\n* ...\n\nThe folder in which they are stored in also configurable through the launch files, but their names are hardcoded for now. An example of the format of the calibration file can be found in [here](nmcl/tst/data/config/front_laser.config).\n\nConfigNMCLNodE can also use cues from the text spotting and object recognition nodes, which you can also enable through the launch file.\n\n### TextRecoNode\n\nListens to Image topics and publishes to a TextArray topic. \nParameters:\n* textSpottingConfig - the name of the textspotting.config file\n* camIDs - I usually use my side-looking cameras (1, 3)\n* cameraImgTopics - topic names for cameras with corrsponding IDs\nCamera IDs should match the cam*.config file. In the textspotting.config, make sure to give the correct path to where the text spotting models and alphabet files are located.\n\n### YoloV5Node\nListens to Image topics and broadcast to a YoloCombinedArray topic. In my implementation, I sync 4 cameras, get the prediction for all 4 images, and publishes it as a combined message. YoloV5Node.py has a hard-coded path pointing to where ever the YOLOv5 repository has been cloned to. So make sure to adapt it accordingly.\nParameters: \n* camIDs - I usually use all cameras (0,1,2,3) for 360 coverage\n* cameraImgTopics - topic names for cameras with corrsponding IDs\n* yoloTopic - where to publish the YOLO predictions\n* weights - location of the model (.pt)\n* conf_thres\n* imgsize - set as max(image.height, image.width) for the camera streams\n* dataset - the path for the dataset.yaml required for YOLO\nThe YOLOv5 node can be replaced by a user-chosen object detetion/semantic segmentation pipeline, as long as the outputs conform to the format of the YoloCombinedArray message.\n\n### RoomClassifierNode\nListens to YoloCombinedArray topic and publishes to a Float32MultiArray topic. Based on the objects detected by YOLO, the room classifier predict which room category is more likely. \nParameters: \n* yoloTopic - where to listen to the YOLO predictions\n* roomTopic - where to publish room category predictions\n* picklePath - The path to the training data pickle for the KNN classifier \n\n### SemanticMapNode\nA visualizer for our abstract semantic maps. \nParameters: \n* dataset - the path for the dataset.yaml required for YOLO\n* dataFolder - as explained previous\n* mapName \n* markerTopic - where to publish the semantic information\n\n\n\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fprbonn%2Fhsmcl","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fprbonn%2Fhsmcl","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fprbonn%2Fhsmcl/lists"}