{"id":18792451,"url":"https://github.com/prbonn/4dndf","last_synced_at":"2025-04-11T21:22:50.298Z","repository":{"id":238370687,"uuid":"774848548","full_name":"PRBonn/4dNDF","owner":"PRBonn","description":"3D LiDAR Mapping in Dynamic Environments using a 4D Implicit Neural Representation (CVPR 2024)","archived":false,"fork":false,"pushed_at":"2024-07-04T16:33:49.000Z","size":33883,"stargazers_count":155,"open_issues_count":0,"forks_count":8,"subscribers_count":9,"default_branch":"main","last_synced_at":"2025-03-25T17:21:56.362Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/PRBonn.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-03-20T09:57:29.000Z","updated_at":"2025-02-26T22:25:01.000Z","dependencies_parsed_at":"2024-05-05T22:37:58.243Z","dependency_job_id":null,"html_url":"https://github.com/PRBonn/4dNDF","commit_stats":null,"previous_names":["prbonn/4dndf"],"tags_count":1,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/PRBonn%2F4dNDF","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/PRBonn%2F4dNDF/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/PRBonn%2F4dNDF/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/PRBonn%2F4dNDF/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/PRBonn","download_url":"https://codeload.github.com/PRBonn/4dNDF/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248480906,"owners_count":21111039,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-07T21:19:55.217Z","updated_at":"2025-04-11T21:22:50.266Z","avatar_url":"https://github.com/PRBonn.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"\u003cp align=\"center\"\u003e\n  \u003ch1 align=\"center\"\u003e3D LiDAR Mapping in Dynamic Environments using a 4D Implicit Neural Representation\u003c/h1\u003e\n\n  \u003cp align=\"center\"\u003e\n    \u003ca href=\"https://github.com/PRBonn/4dNDF\"\u003e\u003cimg src=\"https://img.shields.io/badge/python-3670A0?style=flat-square\u0026logo=python\u0026logoColor=ffdd54\" /\u003e\u003c/a\u003e\n    \u003ca href=\"https://github.com/PRBonn/4dNDF\"\u003e\u003cimg src=\"https://img.shields.io/badge/Linux-FCC624?logo=linux\u0026logoColor=black\" /\u003e\u003c/a\u003e\n    \u003ca href=\"https://www.ipb.uni-bonn.de/wp-content/papercite-data/pdf/zhong2024cvpr.pdf\"\u003e\u003cimg src=\"https://img.shields.io/badge/Paper-pdf-\u003cCOLOR\u003e.svg?style=flat-square\" /\u003e\u003c/a\u003e\n    \u003ca href=\"https://github.com/PRBonn/4dNDF/blob/main/LICENSE\"\u003e\u003cimg src=\"https://img.shields.io/badge/License-MIT-blue.svg?style=flat-square\" /\u003e\u003c/a\u003e\n  \u003c/p\u003e\n\n\n  \u003cp align=\"center\"\u003e\n    \u003ca href=\"https://www.ipb.uni-bonn.de/people/xingguang-zhong/index.html\"\u003e\u003cstrong\u003eXingguang Zhong\u003c/strong\u003e\u003c/a\u003e\n    ·\n    \u003ca href=\"https://www.ipb.uni-bonn.de/people/yue-pan/index.html\"\u003e\u003cstrong\u003eYue Pan\u003c/strong\u003e\u003c/a\u003e\n    ·\n    \u003ca href=\"https://www.ipb.uni-bonn.de/people/cyrill-stachniss/\"\u003e\u003cstrong\u003eCyrill Stachniss\u003c/strong\u003e\u003c/a\u003e\n    .\n    \u003ca href=\"https://www.ipb.uni-bonn.de/people/jens-behley/\"\u003e\u003cstrong\u003eJens Behley\u003c/strong\u003e\u003c/a\u003e\n  \u003c/p\u003e\n  \u003cp align=\"center\"\u003e\u003ca href=\"https://www.ipb.uni-bonn.de\"\u003e\u003cstrong\u003eUniversity of Bonn\u003c/strong\u003e\u003c/a\u003e\n  \u003ch3 align=\"center\"\u003e\u003ca href=\"https://www.ipb.uni-bonn.de/pdfs/zhong2024cvpr.pdf\"\u003ePaper \u003c/a\u003e|\u003c/a\u003e\u003ca  href=\"https://youtu.be/pRNKRcTkxjs\"\u003e Video\u003c/a\u003e\u003c/h3\u003e\n  \u003cdiv align=\"center\"\u003e\u003c/div\u003e\n\u003c/p\u003e\n\n![teaser](media/overview.jpg)\n| Demo |\n| :-: |\n![teaser](media/4dndf.gif) |\n\n\u003c!-- TABLE OF CONTENTS --\u003e\n\u003cdetails open=\"open\" style='padding: 10px; border-radius:5px 30px 30px 5px; border-style: solid; border-width: 1px;'\u003e\n  \u003csummary\u003eTable of Contents\u003c/summary\u003e\n  \u003col\u003e\n    \u003cli\u003e\n      \u003ca href=\"#abstract\"\u003eAbstract\u003c/a\u003e\n    \u003c/li\u003e\n    \u003cli\u003e\n      \u003ca href=\"#installation\"\u003eInstallation\u003c/a\u003e\n    \u003c/li\u003e\n    \u003cli\u003e\n      \u003ca href=\"#run\"\u003eHow to run it\u003c/a\u003e\n    \u003c/li\u003e\n    \u003cli\u003e\n      \u003ca href=\"#contact\"\u003eContact\u003c/a\u003e\n    \u003c/li\u003e\n    \u003cli\u003e\n      \u003ca href=\"#citation\"\u003eCitation\u003c/a\u003e\n    \u003c/li\u003e\n  \u003c/ol\u003e\n\u003c/details\u003e\n\n\n## Abstract\n\n\u003cdetails\u003e\n  \u003csummary\u003e[Details (click to expand)]\u003c/summary\u003e\nBuilding accurate maps is a key building block to enable reliable localization, planning, and navigation of autonomous vehicles. We propose a novel approach for building accurate maps of dynamic environments utilizing a sequence of LiDAR scans. To this end, we propose encoding the 4D scene into a novel spatio-temporal implicit neural map representation by fitting a time-dependent truncated signed distance function to each point. Using our representation, we extract the static map by filtering the dynamic parts. Our neural representation is based on sparse feature grids, a globally shared decoder, and time-dependent basis functions, which we jointly optimize in an unsupervised fashion. To learn this representation from a sequence of LiDAR scans, we design a simple yet efficient loss function to supervise the map optimization in a piecewise way. We evaluate our approach on various scenes containing moving objects in terms of the reconstruction quality of static maps and the segmentation of dynamic point clouds. The experimental results demonstrate that our method is capable of removing the dynamic part of the input point clouds while reconstructing accurate and complete 3D maps, outperforming several state-of-the-art methods.\n\u003c/details\u003e\n\n## Installation\n\nWe tested our code on Ubuntu 22.04 with an NVIDIA RTX 5000.\n\n### 1. Set up conda environment\n\n```\nconda create --name 4dndf python=3.8\nconda activate 4dndf\n```\n\n### 2. Install PyTorch\n\n```\nconda install pytorch==1.13.0 torchvision==0.14.0 torchaudio==0.13.0 pytorch-cuda=11.6 -c pytorch -c nvidia\n```\n\nThe commands depend on your CUDA version. You may check the instructions [here](https://pytorch.org/get-started/previous-versions/).\n\n### 3. Install PyTorch3D\n\nFollow the official instructions [here](https://github.com/facebookresearch/pytorch3d/blob/main/INSTALL.md) to install PyTorch3D with conda.\n\n### 4. Install other dependencies\n\n```\npip install open3d==0.17 scikit-image tqdm pykdtree plyfile\nconda install -c conda-forge quaternion\n```\n\n## How to run it\n\n### Clone the repository\n```\ngit clone https://github.com/PRBonn/4dNDF\ncd 4dNDF\n```\n\n### Sanity test and Demo\n\nFor a sanity test, run the following script to download the test data  (20 frames from KITTI seq 00) :\n```\nsh ./script/download_test_data.bash\n```\n\nThen run:\n```\npython static_mapping.py config/test/test.yaml\n```\nAfter training, It will generate a static mesh in `output/test` and visualize the dynamics segmentation result. For the visualizer, press 'space' to start the playback of the sequence (yellow points correspond to the static parts, and red points correspond to the identified dynamics).\n\n## Evaluation\nAs reported in the paper, we evaluate our method in surface reconstruction and dynamic object segmentation.\n\n### Surface reconstruction\n\nWe use [Co-Fusion](https://github.com/martinruenz/co-fusion)'s car4 dataset and the Newer College dataset to evaluate the quality of our reconstruction. \nThe original Co-Fusion dataset provides depth images rendered in Blender. We convert these depth images into point clouds and use 150 frames for our experiments. (We also provide the script to convert the Co-Fusion's data into our format in `script/cofusion_data_converter.py`.) Run the following script to download the already converted data: \n\n```\nsh ./script/download_cofusion.bash\n```\n\nThen run the following script, which runs our pipeline (i.e., `static_mapping.py`) and computes the metrics:\n```\nsh ./script/run_cofusion.bash\n```\n\nThe reconstructed mesh will be stored in the `output/cofusion` folder.\n\nFor the [newer college dataset](https://ori-drs.github.io/newer-college-dataset/),  we selected part of the data (1300 frames) in the yard for our experiment.\nRun the following script to download the pre-processed data:\n\n```\nsh ./script/download_newer_college.bash\n```\n\nAnd then \n```\nsh ./script/run_newer_college.bash\n```\nto run the training code and evaluation script.\n\nThe reconstructed mesh will be stored in the `output/newer_college` folder. Note that the ground truth point has a different cover area with input scans, so we provide the reference mesh to crop the reconstruction result automatically.\n\nWe also provide the meshes reconstructed by the baseline methods. You can download them by running:\n\n```\nsh ./script/download_baseline.bash\n```\n\nThen change the path of `est_ply ` in `eval/eval_cofusion.py` and `eval/eval_newercollege.py` and run them to check the numbers.\n\n### Dynamic objects segmentation\n\nWe use [KTH_DynamicMap_Benchmark](https://github.com/KTH-RPL/DynamicMap_Benchmark) to evaluate the result of our Dynamic objects segmentation.\n\nDownload the data from the official link [here](https://zenodo.org/records/8160051) and unzip it to our data folder as:\n\n```\n./4dNDF/\n└── data/\n    └── KTH_dynamic/\n            ├── 00/\n            │   ├── gt_cloud.pcd\n            │   ├── pcd/\n            |   |    ├── 004390.pcd\n            |   |    ├── 004391.pcd\n            |   |    └── ...\n            ├── 05/ \n            ├── av2/\n            ├── semindoor/\n            └── translations/\n```\n\nThe benchmark doesn't explicitly provide the pose files, so we extract poses from the data and store them in the `data/translations/` folder.\n\nFor evaluating, you need to clone (to somewhere you like) and compile the [KTH_DynamicMap_Benchmark](https://github.com/KTH-RPL/DynamicMap_Benchmark) 's repo. The following commands should work if you have ROS-full installed on your machine.\n\n```\ngit clone --recurse-submodules https://github.com/KTH-RPL/DynamicMap_Benchmark.git\ncd DynamicMap_Benchmark/script\nmkdir build \u0026\u0026 cd build\ncamke ..\nmake\n```\n\nOr check the guidance from the benchmark [here](https://github.com/KTH-RPL/DynamicMap_Benchmark/tree/master/methods).\n\nThen, copy the Python file from `4dNDF/eval/evaluate_single_kth.py` to `/path/to/DynamicMap_Benchmark/scripts/py/eval/`\n\nTake sequence 00 as an example. Run:\n\n```\npython static_mapping.py config/kth/00.yaml\n```\n\nAfter training, the static point cloud can be found here: `data/kth/00/static_points.pcd`\n\nTo evaluate it, run ( change the `/your/path/to` and `/path/to` to the correct path):\n\n```\ncd /your/path/to/DynamicMap_Benchmark/scripts/build/\n./export_eval_pcd  /path/to/4dNDF/data/KTH_dynamic/00 static_points.pcd 0.05\n```\nIt will generate the eval point cloud. Finally, Run:\n\n```\npython /your/path/to/DynamicMap_Benchmark/scripts/py/eval/evaluate_single_kth.py /path/to/4dNDF/data/KTH_dynamic/00\n```\nto check the number. For other sequences, we need to change all the `00` to `05`,` av2`, or `semindoor`. You can organize the commands as a bash script to make it more convenient.\n\n\n\n## Citation\nIf you use 4dNDF for your academic work, please cite:\n```\n@inproceedings{zhong2024cvpr,\n  author = {Xingguang Zhong and Yue Pan and Cyrill Stachniss and Jens Behley},\n  title = {{3D LiDAR Mapping in Dynamic Environments using a 4D Implicit Neural Representation}},\n  booktitle = {{Proc. of the IEEE/CVF Conf. on Computer Vision and Pattern Recognition (CVPR)}},\n  year = 2024,\n}\n```\n\n## Contact\nIf you have any questions, Feel free to contact:\n\n- Xingguang Zhong {[zhong@igg.uni-bonn.de]()}\n- Yue Pan {[yue.pan@igg.uni-bonn.de]()}\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fprbonn%2F4dndf","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fprbonn%2F4dndf","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fprbonn%2F4dndf/lists"}