{"id":13442854,"url":"https://github.com/PRBonn/4DMOS","last_synced_at":"2025-03-20T15:31:07.529Z","repository":{"id":37483942,"uuid":"503735734","full_name":"PRBonn/4DMOS","owner":"PRBonn","description":"Receding Moving Object Segmentation in 3D LiDAR Data Using Sparse 4D Convolutions (RAL 2022)","archived":false,"fork":false,"pushed_at":"2025-02-25T12:57:24.000Z","size":32288,"stargazers_count":295,"open_issues_count":0,"forks_count":28,"subscribers_count":8,"default_branch":"main","last_synced_at":"2025-03-20T14:14:22.386Z","etag":null,"topics":["4d","deep-learning","lidar","minkowski","minkowskiengine","mos","moving","object","point-cloud","point-cloud-processing","segmentation"],"latest_commit_sha":null,"homepage":"https://www.ipb.uni-bonn.de/wp-content/papercite-data/pdf/mersch2022ral.pdf","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/PRBonn.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":"CITATION.cff","codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2022-06-15T11:26:08.000Z","updated_at":"2025-03-01T17:43:48.000Z","dependencies_parsed_at":"2024-10-28T04:01:18.386Z","dependency_job_id":"511479c6-9dd2-444a-b274-0b05bf2dc1b6","html_url":"https://github.com/PRBonn/4DMOS","commit_stats":null,"previous_names":[],"tags_count":2,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/PRBonn%2F4DMOS","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/PRBonn%2F4DMOS/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/PRBonn%2F4DMOS/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/PRBonn%2F4DMOS/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/PRBonn","download_url":"https://codeload.github.com/PRBonn/4DMOS/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":244639986,"owners_count":20485962,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["4d","deep-learning","lidar","minkowski","minkowskiengine","mos","moving","object","point-cloud","point-cloud-processing","segmentation"],"created_at":"2024-07-31T03:01:52.181Z","updated_at":"2025-03-20T15:31:07.524Z","avatar_url":"https://github.com/PRBonn.png","language":"Python","readme":"\u003cdiv align=\"center\"\u003e\n  \u003ch1\u003eReceding Moving Object Segmentation in 3D LiDAR Data Using Sparse 4D Convolutions\u003c/h1\u003e\n  \u003ca href=\"https://github.com/PRBonn/4DMOS#how-to-use-it\"\u003e\u003cimg src=\"https://img.shields.io/badge/python-3670A0?style=flat-square\u0026logo=python\u0026logoColor=ffdd54\" /\u003e\u003c/a\u003e\n    \u003ca href=\"https://github.com/PRBonn/4DMOS#installation\"\u003e\u003cimg src=\"https://img.shields.io/badge/Linux-FCC624?logo=linux\u0026logoColor=black\" /\u003e\u003c/a\u003e\n    \u003ca href=\"https://www.ipb.uni-bonn.de/pdfs/mersch2022ral.pdf\"\u003e\u003cimg src=\"https://img.shields.io/badge/Paper-pdf-\u003cCOLOR\u003e.svg?style=flat-square\" /\u003e\u003c/a\u003e\n    \u003ca href=\"LICENSE\"\u003e\u003cimg src=\"https://img.shields.io/badge/License-MIT-blue.svg?style=flat-square\" /\u003e\u003c/a\u003e\n\u003c/div\u003e\n\n![example](docs/4dmos.gif)\n*Our moving object segmentation on the unseen SemanticKITTI test sequences 18 and 21. Red points are predicted as moving.*\n\nPlease find the corresponding video [here](https://youtu.be/5aWew6caPNQ).\n\n\u003cp align=\"center\"\u003e\n    \u003cimg src=\"docs/introduction.png\" width=\"600\"\u003e\n\u003c/p\u003e\n\n*Given a sequence of point clouds, our method segments moving (red) from non-moving (black) points.*\n\n\u003cp align=\"center\"\u003e\n    \u003cimg src=\"docs/architecture.png\"\u003e\n\u003c/p\u003e\n\n*We first create a sparse 4D point cloud of all points in a given receding window. We use sparse 4D convolutions from the [MinkowskiEngine](https://github.com/NVIDIA/MinkowskiEngine) to extract spatio-temporal features and predict per-points moving object scores.*\n\n## Important Update\nPlease also check out our latest map-based moving object segmentation pipeline [MapMOS](https://github.com/PRBonn/MapMOS).\n\nThe current state of the repository is improved by internally aligning the scans using [KISS-ICP](https://github.com/PRBonn/kiss-icp). Also, the build system and pipeline have been updated after the release of our newer method [MapMOS](https://github.com/PRBonn/MapMOS), so you can run it on most point cloud data formats. If you want to reproduce the original results from the paper, this version is tagged under `0.1`. You can checkout by\n\n```bash\ngit checkout v0.1\n```\n\n## Installation\nFirst, make sure the [MinkowskiEngine](https://github.com/NVIDIA/MinkowskiEngine) is installed on your system, see [here](https://github.com/NVIDIA/MinkowskiEngine#installation) for more details.\n\nNext, clone our repository\n```bash\ngit clone git@github.com:PRBonn/4DMOS \u0026\u0026 cd 4DMOS\n```\n\nand install with\n```bash\nmake install\n```\n\n**or**\n```bash\nmake install-all\n```\nif you want to install the project with all optional dependencies (needed for the visualizer). In case you want to edit the Python code, install in editable mode:\n```bash\nmake editable\n```\n\n## How to Use It\nJust type\n\n```bash\nmos4d_pipeline --help\n```\nto see how to run 4DMOS.\n\nCheck the [Download](#downloads) section for a pre-trained model. Like [KISS-ICP](https://github.com/PRBonn/kiss-icp), our pipeline runs on a variety of point cloud data formats like `bin`, `pcd`, `ply`, `xyz`, `rosbags`, and more. To visualize these, just type\n\n```bash\nmos4d_pipeline --visualize /path/to/weights.ckpt /path/to/data\n```\n\n\u003cdetails\u003e\n\u003csummary\u003eWant to evaluate with ground truth labels?\u003c/summary\u003e\n\nBecause these labels come in all shapes, you need to specify a dataloader. This is currently available for SemanticKITTI, NuScenes, HeLiMOS, and our labeled KITTI Tracking sequence 19 and Apollo sequences (see [Downloads](#downloads)).\n\n\u003c/details\u003e\n\n## Training\nTo train our approach, you need to first cache your data. To see how to do that, just `cd` into the `4DMOS` repository and type\n\n```bash\npython3 scripts/precache.py --help\n```\n\nAfter this, you can run the training script. Again, `--help` shows you how:\n```bash\npython3 scripts/train.py --help\n```\n\n\u003cdetails\u003e\n\u003csummary\u003eWant to verify the cached data?\u003c/summary\u003e\n\nYou can inspect the cached training samples by using the script `python3 scripts/cache_to_ply.py --help`.\n\n\u003c/details\u003e\n\n\u003cdetails\u003e\n\u003csummary\u003eWant to change the logging directory?\u003c/summary\u003e\n\nThe training log and checkpoints will be saved by default to the current working directory. To change that, export the `export LOGS=/your/path/to/logs` environment variable before running the training script.\n\n\u003c/details\u003e\n\n## HeLiMOS\nTo train on the HeLiMOS data with different sensor configurations, use the following commands:\n\n```shell\npython3 scripts/precache.py /path/to/HeLiMOS helimos /path/to/cache --config config/helimos/*_training.yaml\npython3 scripts/train.py /path/to/HeLiMOS helimos /path/to/cache --config config/helimos/*_training.yaml\n```\n\nby replacing the paths and the config file names. To evaluate for example on the Velodyne test data, run\n\n```shell\nmos4d_pipeline /path/to/weights.ckpt /path/to/HeLiMOS --dataloader helimos -s Velodyne/test.txt\n```\n\n## Evaluation and Visualization\nWe use the [SemanticKITTI API](https://github.com/PRBonn/semantic-kitti-api) to evaluate the intersection-over-union (IOU) of the moving class as well as to visualize the predictions. Clone the repository in your workspace, install the dependencies and then run the following command to visualize your predictions for e.g. sequence 8:\n\n```\ncd semantic-kitti-api\n./visualize_mos.py --sequence 8 --dataset /path/to/dataset --predictions /path/to/4DMOS/predictions/ID/POSES/labels/STRATEGY/\n```\n\n## Benchmark\nTo submit the results to the LiDAR-MOS benchmark, please follow the instructions [here](https://competitions.codalab.org/competitions/28894).\n\n## Downloads\n\u003cp align=\"center\"\u003e\n    \u003cimg src=\"docs/table.png\" width=\"600\"\u003e\n\u003c/p\u003e\n\n* [Model [A]: 5 scans @ 0.1s](https://www.ipb.uni-bonn.de/html/projects/4DMOS/5_scans.zip)\n* [Model [B]: 5 scans @ 0.2s](https://www.ipb.uni-bonn.de/html/projects/4DMOS/5_scans_dt_0p2.zip)\n* [Model [C]: 5 scans @ 0.3s](https://www.ipb.uni-bonn.de/html/projects/4DMOS/5_scans_dt_0p3.zip)\n* [Model [D]: 5 scans, no poses](https://www.ipb.uni-bonn.de/html/projects/4DMOS/5_scans_no_poses.zip)\n* [Model [E]: 5 scans input, 1 scan output](https://www.ipb.uni-bonn.de/html/projects/4DMOS/5_scans_single_output.zip)\n* [Model [F]: 2 scans](https://www.ipb.uni-bonn.de/html/projects/4DMOS/2_scans.zip)\n* [Model [G]: 10 scans](https://www.ipb.uni-bonn.de/html/projects/4DMOS/10_scans.zip)\n\n## Publication\nIf you use our code in your academic work, please cite the corresponding [paper](https://www.ipb.uni-bonn.de/pdfs/mersch2022ral.pdf):\n\n```bibtex\n@article{mersch2022ral,\nauthor = {B. Mersch and X. Chen and I. Vizzo and L. Nunes and J. Behley and C. Stachniss},\ntitle = {{Receding Moving Object Segmentation in 3D LiDAR Data Using Sparse 4D Convolutions}},\njournal={IEEE Robotics and Automation Letters (RA-L)},\nyear = 2022,\nvolume = {7},\nnumber = {3},\npages = {7503--7510},\ncodeurl = {https://github.com/PRBonn/4DMOS},\n}\n```\n\n## Acknowledgments\nThis implementation is heavily inspired by [KISS-ICP](https://github.com/PRBonn/kiss-icp).\n\n## License\nThis project is free software made available under the MIT License. For details see the LICENSE file.\n","funding_links":[],"categories":["Python"],"sub_categories":[],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FPRBonn%2F4DMOS","html_url":"https://awesome.ecosyste.ms/projects/github.com%2FPRBonn%2F4DMOS","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FPRBonn%2F4DMOS/lists"}