{"id":18792493,"url":"https://github.com/prbonn/lidar-mos","last_synced_at":"2025-04-05T00:10:12.675Z","repository":{"id":38444918,"uuid":"378589251","full_name":"PRBonn/LiDAR-MOS","owner":"PRBonn","description":"(LMNet) Moving Object Segmentation in 3D LiDAR Data: A Learning-based Approach Exploiting Sequential Data (RAL/IROS 2021)","archived":false,"fork":false,"pushed_at":"2022-12-21T12:52:58.000Z","size":42709,"stargazers_count":525,"open_issues_count":16,"forks_count":99,"subscribers_count":15,"default_branch":"main","last_synced_at":"2024-02-06T16:03:09.634Z","etag":null,"topics":["deep-learning","deep-neural-networks","dynamic-slam","lidar","lidar-slam","mapping","motion-detection","moving-object-segmentation","point-cloud","segmentation","slam"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/PRBonn.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2021-06-20T07:53:20.000Z","updated_at":"2024-02-05T02:34:15.000Z","dependencies_parsed_at":"2022-07-08T02:20:35.094Z","dependency_job_id":null,"html_url":"https://github.com/PRBonn/LiDAR-MOS","commit_stats":null,"previous_names":[],"tags_count":2,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/PRBonn%2FLiDAR-MOS","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/PRBonn%2FLiDAR-MOS/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/PRBonn%2FLiDAR-MOS/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/PRBonn%2FLiDAR-MOS/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/PRBonn","download_url":"https://codeload.github.com/PRBonn/LiDAR-MOS/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":247266565,"owners_count":20910836,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["deep-learning","deep-neural-networks","dynamic-slam","lidar","lidar-slam","mapping","motion-detection","moving-object-segmentation","point-cloud","segmentation","slam"],"created_at":"2024-11-07T21:20:07.166Z","updated_at":"2025-04-05T00:10:12.652Z","avatar_url":"https://github.com/PRBonn.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# LMNet: Moving Object Segmentation in 3D LiDAR Data\nThis repo contains the code for our paper: Moving Object Segmentation in 3D LiDAR Data:\nA Learning-based Approach Exploiting Sequential Data [PDF](https://www.ipb.uni-bonn.de/pdfs/chen2021ral-iros.pdf).\n\nOur approach accurately segments the scene into moving and static objects, i.e., distinguishing between moving cars vs. parked cars. This task is also called 3D motion detection or segmentation.\nOur method runs faster than the frame rate of the sensor and can be used to improve 3D LiDAR-based odometry/SLAM and mapping results as shown [below](#Applications).\n\nAdditionally, we created a new benchmark for LiDAR-based moving object segmentation based on SemanticKITTI [here](https://codalab.lisn.upsaclay.fr/competitions/7088).\n\n\u003cimg src=\"data/pics/mos_benchmark.png\" width=\"800\"\u003e\n\nComplete demo video can be found in YouTube [here](https://youtu.be/NHvsYhk4dhw). LiDAR-MOS in action:\n\n\u003cimg src=\"data/pics/urban.gif\" width=\"400\"\u003e \u003cimg src=\"data/pics/highway.gif\" width=\"400\"\u003e\n\n### Table of Contents\n1. [Introduction of the repo and benchmark](#LiDAR-MOS:-Moving-Object-Segmentation-in-3D-LiDAR-Data)\n2. [Publication](#Publication)\n3. [Log](#Log)\n4. [Dependencies](#Dependencies)\n5. [How to use](#How-to-use)\n6. [Applications](#Applications)\n7. [Collection of downloads](#Collection-of-downloads)\n8. [License](#License)\n\n\n## Publication\nIf you use our code and benchmark in your academic work, please cite the corresponding [paper](https://www.ipb.uni-bonn.de/pdfs/chen2021ral-iros.pdf):\n    \n    @article{chen2021ral,\n\t\ttitle={{Moving Object Segmentation in 3D LiDAR Data: A Learning-based Approach Exploiting Sequential Data}},\n\t\tauthor={X. Chen and S. Li and B. Mersch and L. Wiesmann and J. Gall and J. Behley and C. Stachniss},\n\t\tyear={2021},\n\t\tvolume=6,\n\t\tissue=4,\n\t\tpages={6529-6536},\n\t\tjournal={IEEE Robotics and Automation Letters (RA-L)},\n\t\turl = {http://www.ipb.uni-bonn.de/pdfs/chen2021ral-iros.pdf},\n\t\tdoi = {10.1109/LRA.2021.3093567},\n\t\tissn = {2377-3766},\n\t}\n\n\n## Log\n### News 20220907\nThe old codalab server stopped the service.\n\nPlease use the new link [here](https://codalab.lisn.upsaclay.fr/competitions/7088) to submit your results to the benchmark.\nYou could still find the old results [here](https://competitions.codalab.org/competitions/28894).\n\n### News 20220706\nOur MotionSeg3D is open-source [here](https://github.com/haomo-ai/MotionSeg3D).\n\nIt uses a dual-branch and dual-head structure to fuse Spatial-Temporal information for LiDAR moving object segmentation.\n\n\n### News 20220615\nOur 4DMOS is open-source [here](https://github.com/PRBonn/4DMOS).\n\nIt uses sparse CNN on 4D point clouds for LiDAR moving object segmentation.\n\n\n### v1.1\nThanks **Jiadai Sun** for testing and correcting some bugs of SalsaNext-MOS.\n\nMore setups can also be found here: [#47](https://github.com/PRBonn/LiDAR-MOS/issues/47)\n\n### v1.0\nOpen-source version\n\n\n## Dependencies\nWe built and tested our work based on [SalsaNext](https://github.com/Halmstad-University/SalsaNext), [RangeNet++](https://github.com/PRBonn/lidar-bonnetal) and [MINet](https://arxiv.org/pdf/2008.09162v1.pdf).\nWe thank the original authors for their nice work and implementation. If you are interested in fast LiDAR-based semantic segmentation, we strongly recommend having a look at the original repositories.\n\n**Note that**, in this repo, we show that how easily we could achieve LiDAR-based moving object segmentation exploiting sequential information with existing segmentation networks.\nWe didn't change the original pipeline of the segmentation networks, but only changed the data loader and input of the network as shown in the figure below.\nTherefore, our method can be used with any range-image-based LiDAR segmentation networks.\n\nOur method is based on range images. To use range projection with fast c++ library, please find the usage doc [here](https://github.com/PRBonn/overlap_localization/tree/master/src/prepare_training).\n\n\u003cimg src=\"data/pics/pipeline.png\" width=\"600\"\u003e\n\n\n## How to use\nFor a quick test of all the steps below, one could download a toy dataset [here](https://www.ipb.uni-bonn.de/html/projects/LiDAR-MOS/LiDAR_MOS_toy_dataset.zip) and decompress it in the `data\\` folder following the data structure [data/README.md](data/README.md).\n\n### Prepare training data\nTo use our method, one needs to generate the residual images. Here is a quick demo:\n```sh\n  $ python3 utils/gen_residual_images.py\n```\nMore setup about the data preparation can be found in the yaml file [config/data_preparing.yaml](config/data_preparing.yaml).\nTo prepare the training data for the whole KITTI-Odometry dataset, please download the [original website](https://www.cvlibs.net/datasets/kitti/eval_odometry.php). \n\n### Using SalsaNext as the baseline\nTo use SalsaNext as the baseline segmentation network for LiDAR-MOS, one should follow the [mos_SalsaNext/README.md](mos_SalsaNext/README.md) to set it up.\n\nNote that, we use pytorch v1.5.1+cu101 which is different from the original one. More information about the related issue is [here](https://github.com/PRBonn/LiDAR-MOS/issues/24).\n\n#### Inferring\nTo generate the LiDAR-MOS predictions with pretrained model with one residual image ([download](https://www.ipb.uni-bonn.de/html/projects/LiDAR-MOS/model_salsanext_residual_1.zip), please `unzip` before using). Quick test on toy dataset, directly run\n```sh\n  $ cd mos_SalsaNext/train/tasks/semantic\n  $ python3 infer.py -d ../../../../data -m ../../../../data/model_salsanext_residual_1 -l ../../../../data/predictions_salsanext_residual_1_new -s valid\n```\n\nInferring the whole dataset, please download the KITTI-Odometry dataset from the [original website](https://www.cvlibs.net/datasets/kitti/eval_odometry.php), and change the corresponding paths.\n```sh\n  $ cd mos_SalsaNext/train/tasks/semantic\n  $ python3 infer.py -d path/to/kitti/dataset -m path/to/pretrained_model -l path/to/log -s train/valid/test # depending of desired split to evaluate\n```\n\n#### Training\nTo train a LiDAR-MOS network with SalsaNext from scratch, one has to download the [KITTI-Odometry dataset](https://www.cvlibs.net/datasets/kitti/eval_odometry.php) and [Semantic-Kitti dataset](http://semantic-kitti.org/):\nChange the corresponding paths and run:\n```sh\n  $ cd mos_SalsaNext/train/tasks/semantic\n  $ ./train.sh -d path/to/kitti/dataset -a salsanext_mos.yml -l path/to/log -c 0  # the number of used gpu cores\n```\n\n### Using RangeNet++ as the baseline \nTo use RangeNet++ as the baseline segmentation network for LiDAR-MOS, one should follow the [mos_RangeNet/README.md](mos_RangeNet/README.md) to set it up.\n\n#### Inferring\nInferring the whole dataset, please download the KITTI-Odometry dataset from the [original website](https://www.cvlibs.net/datasets/kitti/eval_odometry.php), the [pretrained model](https://www.ipb.uni-bonn.de/html/projects/LiDAR-MOS/model_rangenet_residual_1.zip) and change the corresponding paths.\n```sh\n  $ cd mos_RangeNet/tasks/semantic\n  $ python3 infer.py -d path/to/kitti/dataset -m path/to/pretrained_model -l path/to/log -s train/valid/test # depending of desired split to evaluate\n```\n\n#### Training\nTo train a LiDAR-MOS network with RangeNet++ from scratch, one has to download the [KITTI-Odometry dataset](https://www.cvlibs.net/datasets/kitti/eval_odometry.php) and [Semantic-Kitti dataset](http://semantic-kitti.org/) and\nchange the corresponding paths and run:\n```sh\n  $ cd mos_RangeNet/tasks/semantic\n  $ python3 train.py -d path/to/kitti/dataset -ac rangenet_mos.yaml -l path/to/log\n```\n\nMore pretrained model and LiDAR-MOS predictions can be found in [collection of downloads](#Collection-of-downloads).\n\n### Evaluation and visualization\n#### How to evaluate\n**Evaluation metrics**. Let's call the **moving (dynamic)** status as **D** and the **static** status as **S**.\n\nSince we ignore the unlabelled and invalid status, therefore in MOD there are only two classes.\n\n| GT\\Prediction | dynamic | static |\n| --------      | ------- | ------ |\n| dynamic       | TD      |   FS   |\n| static        | FD      |   TS   |\n\n- $$ IoU_{MOS} = \\frac{TD}{TD+FD+FS} $$\n\nTo evaluate the MOS results on the toy dataset just run:\n```sh\n  $ python3 utils/evaluate_mos.py -d data -p data/predictions_salsanext_residual_1_valid -s valid\n```\n\nTo evaluate the MOS results on our **LiDAR-MOS benchmark** please have a look at our [semantic-kitti-api](https://github.com/PRBonn/semantic-kitti-api) and benchmark [website](https://bit.ly/mos-benchmark).\n\n#### How to visualize the predictions\nTo visualize the MOS results on the toy dataset just run:\n```sh\n  $ python3 utils/visualize_mos.py -d data -p data/predictions_salsanext_residual_1_valid -s 8  # here we use a specific sequence number\n```\nwhere:\n - `sequence` is the sequence to be accessed.\n - `dataset` is the path to the kitti dataset where the sequences directory is.\n\nNavigation:\n - `n` is next scan,\n - `b` is previous scan,\n - `esc` or `q` exits.\n\n\n## Applications\nLiDAR-MOS is very important for building consistent maps, making future state predictions, avoiding collisions, and planning.\nIt can also improve and robustify pose estimation, sensor data registration, and SLAM.\nHere we show two obvious applications of our LiDAR-MOS which are LiDAR-based odometry/SLAM as well as 3D mapping.\nBefore that, we show two simple examples of how to combine our method with semantics and clean the scans.\nAfter cleaning scans we can get better odometry/SLAM and 3D mapping results.\n\n**Note that**, here we show two direct use cases of our MOS approach without any further optimizations employed.\n\n\n### Enhanced with semantics\nTo show a simple way of combining our LiDAR-MOS with semantics, we provide a quick demo with the toy dataset:\n```sh\n  $ python3 utils/combine_semantics.py\n```\nIt just simply checks whether the moving objects are movable classes or not. If not, re-assigned as static.\n\n\n### Clean the scans\nTo clean the LiDAR scans with our LiDAR-MOS as masks, we also provide a quick demo on the toy dataset:\n```sh\n  $ python3 utils/scan_cleaner.py\n```\n\n### Odometry/SLAM\nUsing the cleaned LiDAR scans, we see that by simply applying our MOS predictions as a preprocessing mask, the odometry results are improved in both\nthe KITTI training and test data and even slightly better than the carefully-designed full classes semantic-enhanced SuMa++.\n\n\u003cimg src=\"data/pics/odometry_results.png\" width=\"400\"\u003e\n\nThe testing results of our methods can also be found in [KITTI-Odometry benchmark](https://www.cvlibs.net/datasets/kitti/eval_odometry.php).\n\n### Mapping\nwe compare the aggregated point cloud maps (left) directly with the raw LiDAR\nscans, (right) with the cleaned LiDAR scans by applying our MOS predictions as masks.\nAs can be seen, there are moving objects present that pollute the map, which might have adversarial effects, when used for localization or path\nplanning. By using our MOS predictions as masks, we can effectively remove these artifacts and get a clean map. \n\n\u003cimg src=\"data/pics/map_raw.png\" width=\"400\"\u003e \u003cimg src=\"data/pics/map_clean.png\" width=\"400\"\u003e\n\n#### Map cleaning\nFor offline map cleaning, **Giseop Kim** combined his [Removert](https://github.com/irapkaist/removert) and LiDAR-MOS, and got very good results.\nMore information can be found in [#28](https://github.com/PRBonn/LiDAR-MOS/issues/28).\n\n\n## Collection of downloads\n - [LiDAR_MOS_toy_dataset](https://www.ipb.uni-bonn.de/html/projects/LiDAR-MOS/LiDAR_MOS_toy_dataset.zip)                           (toy dataset used for the quick demos)\n - [predictions_salsanext_semantic](https://www.ipb.uni-bonn.de/html/projects/LiDAR-MOS/predictions_salsanext_semantic.zip)        (semantic segmentation results from SalsaNext for all sequences 00 - 21) \n - [predictions_salsanext_residual_8_sem](https://www.ipb.uni-bonn.de/html/projects/LiDAR-MOS/predictions_salsanext_residual_8_sem.zip)  (**Our best!** LiDAR-MOS results using SalsaNext with 8 residual images + semantics) \n - [model_rangenet_residual_1](https://www.ipb.uni-bonn.de/html/projects/LiDAR-MOS/model_rangenet_residual_1.zip)             (pretrained model using RangeNet++ with 1 residual image) \n - [model_salsanext_residual_1](https://www.ipb.uni-bonn.de/html/projects/LiDAR-MOS/model_salsanext_residual_1.zip)            (pretrained model using SalsaNext with 1 residual image)\n - [model_salsanext_residual_8](https://www.ipb.uni-bonn.de/html/projects/LiDAR-MOS/model_salsanext_residual_8.zip)            (pretrained model using SalsaNext with 8 residual image)\n \n\n## License\nThis project is free software made available under the MIT License. For details see the LICENSE file.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fprbonn%2Flidar-mos","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fprbonn%2Flidar-mos","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fprbonn%2Flidar-mos/lists"}