{"id":13442283,"url":"https://github.com/PRBonn/OverlapNet","last_synced_at":"2025-03-20T13:33:20.513Z","repository":{"id":39739176,"uuid":"263319282","full_name":"PRBonn/OverlapNet","owner":"PRBonn","description":"OverlapNet - Loop Closing for 3D LiDAR-based SLAM (chen2020rss)","archived":false,"fork":false,"pushed_at":"2023-03-24T22:59:34.000Z","size":30661,"stargazers_count":646,"open_issues_count":8,"forks_count":115,"subscribers_count":26,"default_branch":"master","last_synced_at":"2024-08-01T03:39:26.117Z","etag":null,"topics":["3d-lidar","lidar","loop-closing","loop-closure-detection","slam"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/PRBonn.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE.txt","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null}},"created_at":"2020-05-12T11:37:42.000Z","updated_at":"2024-07-29T06:09:09.000Z","dependencies_parsed_at":"2023-01-31T04:16:41.231Z","dependency_job_id":"4d5d7157-605c-49f5-ae0e-8a069c1c1759","html_url":"https://github.com/PRBonn/OverlapNet","commit_stats":null,"previous_names":[],"tags_count":3,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/PRBonn%2FOverlapNet","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/PRBonn%2FOverlapNet/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/PRBonn%2FOverlapNet/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/PRBonn%2FOverlapNet/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/PRBonn","download_url":"https://codeload.github.com/PRBonn/OverlapNet/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":221768531,"owners_count":16877655,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["3d-lidar","lidar","loop-closing","loop-closure-detection","slam"],"created_at":"2024-07-31T03:01:43.877Z","updated_at":"2025-03-20T13:33:20.507Z","avatar_url":"https://github.com/PRBonn.png","language":"Python","funding_links":[],"categories":["Python","5. Learning based SLAM","Sensor Processing"],"sub_categories":["5.2 Others","Lidar and Point Cloud Processing"],"readme":"# OverlapNet - Loop Closing for 3D LiDAR-based SLAM\n\n### OverlapNet was nominated as the Best System Paper at Robotics: Science and Systems (RSS) 2020 \n\nThis repo contains the code for our RSS2020 paper, OverlapNet.\n \nOverlapNet is modified Siamese Network that predicts the overlap and relative yaw angle of a pair of range images generated by 3D LiDAR scans. \n\nDeveloped by [Xieyuanli Chen](https://www.ipb.uni-bonn.de/people/xieyuanli-chen/) and [Thomas Läbe](https://www.ipb.uni-bonn.de/people/thomas-laebe/).\n\n\n\u003cimg src=\"pics/architecture.png\" width=\"800\"\u003e\n\nPipeline overview of OverlapNet.\n\n\n### Table of Contents\n0. [Introduction](#OverlapNet-was-nominated-as-the-Best-System-Paper-at-Robotics:-Science-and-Systems-(RSS)-2020)\n0. [Publication](#Publication)\n0. [Logs](#Logs)\n0. [Dependencies](#Dependencies)\n0. [How to use](#How-to-use)\n0. [Application](#Application)\n0. [License](#License)\n\n\n## Publication\nIf you use our implementation in your academic work, please cite the corresponding paper ([PDF](https://www.ipb.uni-bonn.de/wp-content/papercite-data/pdf/chen2020rss.pdf)):  \n    \n\t@inproceedings{chen2020rss, \n\t\t\tauthor = {X. Chen and T. L\\\"abe and A. Milioto and T. R\\\"ohling and O. Vysotska and A. Haag and J. Behley and C. Stachniss},\n\t\t\ttitle  = {{OverlapNet: Loop Closing for LiDAR-based SLAM}},\n\t\t\tbooktitle = {Proceedings of Robotics: Science and Systems (RSS)},\n\t\t\tyear = {2020}\n\t}\n\nThe extended journal version of OverlapNet is here ([PDF](http://www.ipb.uni-bonn.de/pdfs/chen2021auro.pdf)):\n\n\t@article{chen2021auro,\n\t\tauthor = {X. Chen and T. L\\\"abe and A. Milioto and T. R\\\"ohling and J. Behley and C. Stachniss},\n\t\ttitle = {{OverlapNet: A Siamese Network for Computing LiDAR Scan Similarity with Applications to Loop Closing and Localization}},\n\t\tjournal = {{Autonomous Robots}},\n\t\tyear = {2021},\n\t\tdoi = {10.1007/s10514-021-09999-0},\n\t\tissn = {1573-7527},\n\t\tvolume={46},\n\t\tpages={61--81},\n\t\turl = {http://www.ipb.uni-bonn.de/pdfs/chen2021auro.pdf}\n\t}\n\n## News and Logs\n\n### New Version\nBased on the OverlapTransformer, we propose a novel transformer exploiting sequential LiDAR data to achieve better LiDAR place recognition ([code](https://github.com/BIT-MJY/SeqOT)). \n\nWe develop a novel lightweight neural network called [**OverlapTransformer**](https://github.com/haomo-ai/OverlapTransformer) exploiting transformer and achieve fast execution with less than 4 ms per frame in estimating LiDAR scans similarity.\n\n### Version 1.2\n- We thank [Junyi Ma](https://github.com/BIT-MJY) to create the PyTorch version of OverlapNet (delta-head-only), which can be found in the [pytorch version branch](https://github.com/PRBonn/OverlapNet/tree/pytorch-version).\n### Version 1.1\n- Added a method to the Infer class for inference with multiple frames versus multiple frames.\n- Updated TensorFlow version in dependencies.\n- Fixed bugs in generating ground truth overlap and yaw.\n- Added an application and a link to our overlap-based MCL implementation. \n### Version 1.0\nOpen source initial submission\n\n## Dependencies\n\nWe are using standalone keras with a tensorflow backend as a library for neural networks.\n\nIn order to do training and testing on a whole dataset, you need a nvidia GPU.\nThe demos still are fast enough when using the neural network on CPU.\n\nTo use a GPU, first you need to install the nvidia driver and CUDA, so have fun!\n\n- CUDA Installation guide: [link](https://docs.nvidia.com/cuda/cuda-installation-guide-linux/index.html)\n\n- System dependencies:\n\n  ```bash\n  sudo apt-get update \n  sudo apt-get install -y python3-pip python3-tk\n  sudo -H pip3 install --upgrade pip\n  ```\n\n- Python dependencies (may also work with different versions than mentioned in the requirements file)\n\n  ```bash\n  sudo -H pip3 install -r requirements.txt\n  ```\n\n\n\n## How to use\n\nThis repository contains the neural network for doing a detection of loop closing candidates.\n\nFor a complete pipline for online LiDAR preprocessing, one could find the fast implementation in our [SuMa++](https://github.com/PRBonn/semantic_suma).\n\nIn this repository, we provide demos to show the functionaly. Additional to that, we explain\nhow to train a model.\n\n### Demos\n\n##### Demo 1: generate different types of data from the LiDAR scan\nTo try demo 1, you could directly run the script with one command line:\n\n```bash\npython3 demo/demo1_gen_data.py\n```\n\nThe generated data are stored in `/data/preprocess_data`, and you will get a visualization like this:\n![](pics/demo1.png)\n\nFor generating range and normal images, we have also a much faster implementation in C (with python interface) available\nin our repo https://github.com/PRBonn/overlap_localization (look at `src/prepare_training`).\n\n##### Demo 2: Inferring overlap and relative yaw angle between two LiDAR scans\nTo run demo 2, you need to first download the pre-trained [model](https://www.ipb.uni-bonn.de/html/projects/overlap_net/model_geo.weight).\n\nThen, you should\n- either copy it into the default location folder `data` or\n- you need to modify the `pretrained_weightsfilename` in the config file `/config/network.yml` accordingly.\n\nRun the second demo script with one command line:\n\n```bash\npython3 demo/demo2_infer.py\n```\nYou will get a visualization like this: \n\n![](pics/demo2.png)\n\n\n##### Demo 3: Loop closure detection\nTo run demo 3, you need to first download several data:\n- pre-trained [model](https://www.ipb.uni-bonn.de/html/projects/overlap_net/model_geo.weight), \n- KITTI odometry [data](https://www.ipb.uni-bonn.de/html/projects/overlap_net/kitti_07.zip), where we also provide the covariance information generated from the SLAM, \n- pre-processed [data](https://www.ipb.uni-bonn.de/html/projects/overlap_net/preprocess_07.zip).\n\nIf you follow the recommended [data structure](#data-structure) below, you extract the downloaded data into the folder `data`.\n\nOtherwise, you need to specify the paths of data in both `config/network.yml` and `config/demo.yml` accordingly,\n\nand then run the third demo script with one command line:\n\n```bash\npython3 demo/demo3_lcd.py\n```\n\nYou will get an animation like this:\n\n\u003cimg src=\"pics/demo3.gif\" width=\"600\"\u003e\n\n\n##### Demo 4: Generate ground truth overlap and yaw for training and testing\nTo run demo 4, you need only the raw KITTI odometry data. We are using the same\nsetup as in demo 3. \n\nRun the fourth demo script with one command line:\n\n```bash\npython3 demo/demo4_gen_gt_files.py\n```\n\nYou will generated the ground truth data in `data/preprocess_data_demo/ground_truth` and get a plot like this:\n\n\u003cimg src=\"pics/demo4.png\" width=\"600\"\u003e\n\nThe colors represent the ground truth overlap value of each frame with respect to the given current frame which is located at (0.0, 0.0).\n\n\n### Train and test a model\n\nFor a quick test of the training and testing procedures, you could use our preprocessed data \nas used in [demo3](#demo-3-loop-closure-detection).\n\nWe only provide the geometric-based preprocessed data. But it will also be possible to generate other inputs\n(semantics, intensity) by yourself. \n\nA simple example to generate different types of data from LiDAR scan is given in [demo1](#demos).\n\nFor 3D LiDAR semantic segmentation, we provide a fast c++ inferring library \n[rangenetlib](https://github.com/PRBonn/rangenet_lib).\n\n#### Data structure\n\nFor training a new model with OverlapNet, you need to first generate preprocessed data and ground truth overlap and yaw angle which you could find examples in [demo1](#demos) and [demo4](#demo-4-generate-ground-truth-overlap-and-yaw-for-training-and-testing).\n\nThe recommended data structure is as follows:\n\n```bash\ndata\n    ├── 07\n    │   ├── calib.txt\n    │   ├── covariance.txt\n    │   ├── poses.txt\n    │   ├── depth\n    │   │   ├── 000000.npy\n    │   │   ├── 000001.npy\n    │   │   └── ...\n    │   ├── normal\n    │   │   ├── 000000.npy\n    │   │   ├── 000001.npy\n    │   │   └── ...\n    │   ├── velodyne\n    │   │   ├── 000000.bin\n    │   │   ├── 000001.bin\n    │   │   └── ...\n    │   └── ground_truth\n    │       ├── ground_truth_overlap_yaw.npz\n    │       ├── test_set.npz\n    │       └── train_set.npz\n    └── model_geo.weight\n```\n\n\n#### Training\n\nThe code for training can be found in `src/two_heads/training.py`. \n\nIf you download our preprocessed data, please put the data into the folder `data`.\nIf you want to use another directory, please change the parameter `data_root_folder` in\nthe configuration file `network.yml`.\n\nNotice that default weight file is set in the configuration file with \nparameter `pretrained_weightsfilename`.\nIf you want to train a completely new model from scratch, leave this parameter empty. \nOtherwise you will fine-tune the provided model.\n\nThen you can start the training with\n\n```\npython3 src/two_heads/training.py config/network.yml\n```\n\nAll configuration data is in the yml file. You will find path definitions and training parameters there. The\nmain path settings are:\n- `experiments_path`: the folder where all the training data and results (log files, tensorboard logs, network weights)\n  will be saved. Default is `/tmp`. Change this according to your needs\n- `data_root_folder`: the dataset folder. Is should contain the sequence folders of the dataset e.g. as `00`, `01`, ..., For the provided preproecessed data, it should be `07`.\n\nWe provide tensorboard logs in `experiment_path/testname/tblog` for visualizing training and validation details.\n\n#### Testing \n\nOnce a model has been trained (thus a file `.weight` with the network weights is available), the performance of the network\ncan be evaluated. Therefore you can start the testing script in the same manner as the training with the testing script:\n\n```\npython3 src/two_heads/testing.py config/network.yml\n```\n\nThe configuration file should have the following additional settings:\n- `pretrained_weightsfilename`: the weight filename mentioned as parameter\n- `testing_seqs`: sequences to test on, e.g. `00 01`. (Please comment out `training_seqs`.) The pairs where the tests are computed are in \n   the file `ground_truth/ground_truth_overlap_yaw.npz`. If one still uses the parameter `training_seqs`, the validation\n   is done on the test sets of the sequences (`ground_truth/validation_set.npz`) which contain only a small amount of data used \n   for validation during training.\n\nNote that: the provided pre-trained model and preprocessed ground truth are with the constraint that the current frame only finds loop closures in the previous frames.\n\n\n## Application\n### [Overlap-based Monte Carlo Localization](https://github.com/PRBonn/overlap_localization)\nThis repo contains the code for our IROS2020 paper: Learning an Overlap-based Observation Model for 3D LiDAR Localization.\n\nIt uses the OverlapNet to train an observation model for Monte Carlo Localization and achieves global localization with 3D LiDAR scans.\n\n\n\n## License\n\nCopyright 2020, Xieyuanli Chen, Thomas Läbe, Cyrill Stachniss, Photogrammetry and Robotics Lab, University of Bonn.\n\nThis project is free software made available under the MIT License. For details see the LICENSE file.\n\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FPRBonn%2FOverlapNet","html_url":"https://awesome.ecosyste.ms/projects/github.com%2FPRBonn%2FOverlapNet","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FPRBonn%2FOverlapNet/lists"}