{"id":13441121,"url":"https://github.com/ispc-lab/LiDAR4D","last_synced_at":"2025-03-20T11:35:18.589Z","repository":{"id":231464972,"uuid":"767950402","full_name":"ispc-lab/LiDAR4D","owner":"ispc-lab","description":"💫 [CVPR 2024] LiDAR4D: Dynamic Neural Fields for Novel Space-time View LiDAR Synthesis","archived":false,"fork":false,"pushed_at":"2024-06-18T14:59:10.000Z","size":89,"stargazers_count":125,"open_issues_count":1,"forks_count":10,"subscribers_count":4,"default_branch":"main","last_synced_at":"2024-08-01T03:33:15.826Z","etag":null,"topics":["autonomous-driving","computer-vision","cvpr2024","dynamic-scene","lidar","neural-rendering","novel-view-synthesis","point-cloud","reconstruction"],"latest_commit_sha":null,"homepage":"https://dyfcalid.github.io/LiDAR4D","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/ispc-lab.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-03-06T07:39:59.000Z","updated_at":"2024-08-01T03:01:28.000Z","dependencies_parsed_at":"2024-06-18T18:10:13.919Z","dependency_job_id":"82159c9a-e794-4d0f-a5c1-7664935f1d2e","html_url":"https://github.com/ispc-lab/LiDAR4D","commit_stats":null,"previous_names":["ispc-lab/lidar4d"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ispc-lab%2FLiDAR4D","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ispc-lab%2FLiDAR4D/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ispc-lab%2FLiDAR4D/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ispc-lab%2FLiDAR4D/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/ispc-lab","download_url":"https://codeload.github.com/ispc-lab/LiDAR4D/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":221752294,"owners_count":16874953,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["autonomous-driving","computer-vision","cvpr2024","dynamic-scene","lidar","neural-rendering","novel-view-synthesis","point-cloud","reconstruction"],"created_at":"2024-07-31T03:01:30.169Z","updated_at":"2024-10-28T00:31:52.463Z","avatar_url":"https://github.com/ispc-lab.png","language":"Python","funding_links":[],"categories":["Python"],"sub_categories":[],"readme":"\u003cdiv align=\"center\"\u003e\n\u003ch1\u003e\u003cimg src=\"https://github.com/ispc-lab/LiDAR4D/assets/51731102/7f3dd959-9b97-481e-8c13-45abbc2b712d\" width=25%\u003e\u003c/h1\u003e\n\n\u003ch3\u003eLiDAR4D: Dynamic Neural Fields for Novel Space-time View LiDAR Synthesis\u003c/h3\u003e  \n\n[Zehan Zheng](https://dyfcalid.github.io/), [Fan Lu](https://fanlu97.github.io/), Weiyi Xue, [Guang Chen](https://ispc-group.github.io/)†, Changjun Jiang  († Corresponding author)  \n**CVPR 2024**\n\n\n**[Paper (arXiv)](https://arxiv.org/abs/2404.02742) | [Paper (CVPR)](https://openaccess.thecvf.com/content/CVPR2024/html/Zheng_LiDAR4D_Dynamic_Neural_Fields_for_Novel_Space-time_View_LiDAR_Synthesis_CVPR_2024_paper.html) | [Project Page](https://dyfcalid.github.io/LiDAR4D) | [Video](https://www.youtube.com/watch?v=E6XyG3A3EZ8) | [Poster](https://drive.google.com/file/d/13cf0rSjCjGRyBsYOcQSa6Qf1Oe1a5QCy/view?usp=sharing) | [Slides](https://drive.google.com/file/d/1Q6yTVGoBf_nfWR4rW9RcSGlxRMufmSXc/view?usp=sharing)**  \n\nThis repository is the official PyTorch implementation for LiDAR4D.\n\n\u003cimg src=\"https://github.com/ispc-lab/LiDAR4D/assets/51731102/e23640bf-bd92-4ee0-88b4-375faf8c9b4d\" width=50%\u003e\n\u003c/div\u003e\n\n\u003c!-- TABLE OF CONTENTS --\u003e\n\u003cdetails open=\"open\" style='padding: 10px; border-radius:5px 30px 30px 5px; border-style: solid; border-width: 1px;'\u003e\n  \u003csummary\u003eTable of Contents\u003c/summary\u003e\n  \u003col\u003e\n    \u003cli\u003e\n      \u003ca href=\"#changelog\"\u003eChangelog\u003c/a\u003e\n    \u003c/li\u003e\n    \u003cli\u003e\n      \u003ca href=\"#demo\"\u003eDemo\u003c/a\u003e\n    \u003c/li\u003e\n    \u003cli\u003e\n      \u003ca href=\"#introduction\"\u003eIntroduction\u003c/a\u003e\n    \u003c/li\u003e\n    \u003cli\u003e\n      \u003ca href=\"#getting-started\"\u003eGetting started\u003c/a\u003e\n    \u003c/li\u003e\n    \u003cli\u003e\n      \u003ca href=\"#results\"\u003eResults\u003c/a\u003e\n    \u003c/li\u003e\n    \u003cli\u003e\n      \u003ca href=\"#simulation\"\u003eSimulation\u003c/a\u003e\n    \u003c/li\u003e\n    \u003cli\u003e\n      \u003ca href=\"#citation\"\u003eCitation\u003c/a\u003e\n    \u003c/li\u003e\n  \u003c/ol\u003e\n\u003c/details\u003e\n\n\n## Changelog\n2024-6-1:🕹️ We release the simulator for easier rendering and manipulation. *Happy Children's Day and Have Fun!*   \n2024-5-4:📈 We update flow fields and improve temporal interpolation.   \n2024-4-13:📈 We update U-Net of LiDAR4D for better ray-drop refinement.   \n2024-4-5:🚀 Code of LiDAR4D is released.  \n2024-4-4:🔥 You can reach the preprint paper on arXiv as well as the project page.  \n2024-2-27:🎉 Our paper is accepted by CVPR 2024.  \n\n\n## Demo\n\u003cvideo src=\"https://github.com/ispc-lab/LiDAR4D/assets/51731102/34f898ec-404d-4f10-afe5-1e471df2cfe2\"\u003e\u003c/video\u003e\n\n\n## Introduction\n\u003cimg src=\"https://github.com/ispc-lab/LiDAR4D/assets/51731102/42083b63-2459-4eb9-bb8f-651eca0a1148\" width=90%\u003e  \n\nLiDAR4D is a differentiable LiDAR-only framework for novel space-time LiDAR view synthesis, which reconstructs dynamic driving scenarios and generates realistic LiDAR point clouds end-to-end. It adopts 4D hybrid neural representations and motion priors derived from point clouds for geometry-aware and time-consistent large-scale scene reconstruction.\n\n\n## Getting started\n\n\n### 🛠️ Installation\n\n```bash\ngit clone https://github.com/ispc-lab/LiDAR4D.git\ncd LiDAR4D\n\nconda create -n lidar4d python=3.9\nconda activate lidar4d\n\n# PyTorch\n# CUDA 12.1\npip install torch==2.1.0 torchvision==0.16.0 torchaudio==2.1.0 --index-url https://download.pytorch.org/whl/cu121\n# CUDA 11.8\n# pip install torch==2.1.0 torchvision==0.16.0 torchaudio==2.1.0 --index-url https://download.pytorch.org/whl/cu118\n# CUDA \u003c= 11.7\n# pip install torch==2.0.0 torchvision torchaudio\n\n# Dependencies\npip install -r requirements.txt\n\n# Local compile for tiny-cuda-nn\ngit clone --recursive https://github.com/nvlabs/tiny-cuda-nn\ncd tiny-cuda-nn/bindings/torch\npython setup.py install\n\n# compile packages in utils\ncd utils/chamfer3D\npython setup.py install\n```\n\n\n### 📁 Dataset\n\n#### KITTI-360 dataset ([Download](https://www.cvlibs.net/datasets/kitti-360/download.php))\nWe use sequence00 (`2013_05_28_drive_0000_sync`) for experiments in our paper.  \n\n\u003cimg src=\"https://github.com/ispc-lab/LiDAR4D/assets/51731102/c9f5d5c5-ac48-4d54-8109-9a8b745bbca0\" width=65%\u003e  \n\nDownload KITTI-360 dataset (2D images are not needed) and put them into `data/kitti360`.  \n(or use symlinks: `ln -s DATA_ROOT/KITTI-360 ./data/kitti360/`).  \nThe folder tree is as follows:  \n\n```bash\ndata\n└── kitti360\n    └── KITTI-360\n        ├── calibration\n        ├── data_3d_raw\n        └── data_poses\n```\n\nNext, run KITTI-360 dataset preprocessing: (set `DATASET` and `SEQ_ID`)  \n\n```bash\nbash preprocess_data.sh\n```\n\nAfter preprocessing, your folder structure should look like this:  \n\n```bash\nconfigs\n├── kitti360_{sequence_id}.txt\ndata\n└── kitti360\n    ├── KITTI-360\n    │   ├── calibration\n    │   ├── data_3d_raw\n    │   └── data_poses\n    ├── train\n    ├── transforms_{sequence_id}test.json\n    ├── transforms_{sequence_id}train.json\n    └── transforms_{sequence_id}val.json\n```\n\n### 🚀 Run LiDAR4D\n\nSet corresponding sequence config path in `--config` and you can modify logging file path in `--workspace`. Remember to set available GPU ID in `CUDA_VISIBLE_DEVICES`.   \nRun the following command:\n```bash\n# KITTI-360\nbash run_kitti_lidar4d.sh\n```\n\n\n\u003ca id=\"results\"\u003e\u003c/a\u003e\n\n## 📊 Results \n\n**KITTI-360 *Dynamic* Dataset** (Sequences: `2350` `4950` `8120` `10200` `10750` `11400`)\n\n\u003ctable\u003e\n\u003ctbody align=\"center\" valign=\"center\"\u003e\n  \u003ctr\u003e\n    \u003cth rowspan=\"2\"\u003eMethod\u003c/th\u003e\n    \u003cth colspan=\"2\"\u003ePoint Cloud\u003c/th\u003e\n    \u003cth colspan=\"5\"\u003eDepth\u003c/th\u003e\n    \u003cth colspan=\"5\"\u003eIntensity\u003c/th\u003e\n  \u003c/tr\u003e\n  \u003ctr\u003e\n    \u003cth\u003eCD↓\u003c/th\u003e\n    \u003cth nowrap=\"true\"\u003eF-Score↑\u003c/th\u003e\n    \u003cth\u003eRMSE↓\u003c/th\u003e\n    \u003cth\u003eMedAE↓\u003c/th\u003e\n    \u003cth\u003eLPIPS↓\u003c/th\u003e\n    \u003cth\u003eSSIM↑\u003c/th\u003e\n    \u003cth\u003ePSNR↑\u003c/th\u003e\n    \u003cth\u003eRMSE↓\u003c/th\u003e\n    \u003cth\u003eMedAE↓\u003c/th\u003e\n    \u003cth\u003eLPIPS↓\u003c/th\u003e\n    \u003cth\u003eSSIM↑\u003c/th\u003e\n    \u003cth\u003ePSNR↑\u003c/th\u003e\n  \u003c/tr\u003e\n  \u003ctr\u003e\n    \u003ctd\u003eLiDAR-NeRF\u003c/td\u003e\n    \u003ctd\u003e0.1438\u003c/td\u003e\n    \u003ctd\u003e0.9091\u003c/td\u003e\n    \u003ctd\u003e4.1753\u003c/td\u003e\n    \u003ctd\u003e0.0566\u003c/td\u003e\n    \u003ctd\u003e0.2797\u003c/td\u003e\n    \u003ctd\u003e0.6568\u003c/td\u003e\n    \u003ctd\u003e25.9878\u003c/td\u003e\n    \u003ctd\u003e0.1404\u003c/td\u003e\n    \u003ctd\u003e0.0443\u003c/td\u003e\n    \u003ctd\u003e0.3135\u003c/td\u003e\n    \u003ctd\u003e0.3831\u003c/td\u003e\n    \u003ctd\u003e17.1549\u003c/td\u003e\n  \u003c/tr\u003e\n  \u003ctr\u003e\n    \u003ctd\u003eLiDAR4D (Ours) †\u003c/td\u003e\n    \u003ctd\u003e\u003cb\u003e0.1002\u003c/b\u003e\u003c/td\u003e\n    \u003ctd\u003e\u003cb\u003e0.9320\u003c/b\u003e\u003c/td\u003e\n    \u003ctd\u003e\u003cb\u003e3.0589\u003c/b\u003e\u003c/td\u003e\n    \u003ctd\u003e\u003cb\u003e0.0280\u003c/b\u003e\u003c/td\u003e\n    \u003ctd\u003e\u003cb\u003e0.0689\u003c/b\u003e\u003c/td\u003e\n    \u003ctd\u003e\u003cb\u003e0.8770\u003c/b\u003e\u003c/td\u003e\n    \u003ctd\u003e\u003cb\u003e28.7477\u003c/b\u003e\u003c/td\u003e\n    \u003ctd\u003e\u003cb\u003e0.0995\u003c/b\u003e\u003c/td\u003e\n    \u003ctd\u003e\u003cb\u003e0.0262\u003c/b\u003e\u003c/td\u003e\n    \u003ctd\u003e\u003cb\u003e0.1498\u003c/b\u003e\u003c/td\u003e\n    \u003ctd\u003e\u003cb\u003e0.6561\u003c/b\u003e\u003c/td\u003e\n    \u003ctd\u003e\u003cb\u003e20.0884\u003c/b\u003e\u003c/td\u003e\n  \u003c/tr\u003e\n\u003c/tbody\u003e\n\u003c/table\u003e\n\n\u003cbr\u003e\n\n**KITTI-360 *Static* Dataset** (Sequences: `1538` `1728` `1908` `3353`)\n\n\u003ctable\u003e\n\u003ctbody align=\"center\" valign=\"center\"\u003e\n  \u003ctr\u003e\n    \u003cth rowspan=\"2\"\u003eMethod\u003c/th\u003e\n    \u003cth colspan=\"2\"\u003ePoint Cloud\u003c/th\u003e\n    \u003cth colspan=\"5\"\u003eDepth\u003c/th\u003e\n    \u003cth colspan=\"5\"\u003eIntensity\u003c/th\u003e\n  \u003c/tr\u003e\n  \u003ctr\u003e\n    \u003cth\u003eCD↓\u003c/th\u003e\n    \u003cth nowrap=\"true\"\u003eF-Score↑\u003c/th\u003e\n    \u003cth\u003eRMSE↓\u003c/th\u003e\n    \u003cth\u003eMedAE↓\u003c/th\u003e\n    \u003cth\u003eLPIPS↓\u003c/th\u003e\n    \u003cth\u003eSSIM↑\u003c/th\u003e\n    \u003cth\u003ePSNR↑\u003c/th\u003e\n    \u003cth\u003eRMSE↓\u003c/th\u003e\n    \u003cth\u003eMedAE↓\u003c/th\u003e\n    \u003cth\u003eLPIPS↓\u003c/th\u003e\n    \u003cth\u003eSSIM↑\u003c/th\u003e\n    \u003cth\u003ePSNR↑\u003c/th\u003e\n  \u003c/tr\u003e\n  \u003ctr\u003e\n    \u003ctd\u003eLiDAR-NeRF\u003c/td\u003e\n    \u003ctd\u003e0.0923\u003c/td\u003e\n    \u003ctd\u003e0.9226\u003c/td\u003e\n    \u003ctd\u003e3.6801\u003c/td\u003e\n    \u003ctd\u003e0.0667\u003c/td\u003e\n    \u003ctd\u003e0.3523\u003c/td\u003e\n    \u003ctd\u003e0.6043\u003c/td\u003e\n    \u003ctd\u003e26.7663\u003c/td\u003e\n    \u003ctd\u003e0.1557\u003c/td\u003e\n    \u003ctd\u003e0.0549\u003c/td\u003e\n    \u003ctd\u003e0.4212\u003c/td\u003e\n    \u003ctd\u003e0.2768\u003c/td\u003e\n    \u003ctd\u003e16.1683\u003c/td\u003e\n  \u003c/tr\u003e\n  \u003ctr\u003e\n    \u003ctd\u003eLiDAR4D (Ours) †\u003c/td\u003e\n    \u003ctd\u003e\u003cb\u003e0.0834\u003c/b\u003e\u003c/td\u003e\n    \u003ctd\u003e\u003cb\u003e0.9312\u003c/b\u003e\u003c/td\u003e\n    \u003ctd\u003e\u003cb\u003e2.7413\u003c/b\u003e\u003c/td\u003e\n    \u003ctd\u003e\u003cb\u003e0.0367\u003c/b\u003e\u003c/td\u003e\n    \u003ctd\u003e\u003cb\u003e0.0995\u003c/b\u003e\u003c/td\u003e\n    \u003ctd\u003e\u003cb\u003e0.8484\u003c/b\u003e\u003c/td\u003e\n    \u003ctd\u003e\u003cb\u003e29.3359\u003c/b\u003e\u003c/td\u003e\n    \u003ctd\u003e\u003cb\u003e0.1116\u003c/b\u003e\u003c/td\u003e\n    \u003ctd\u003e\u003cb\u003e0.0335\u003c/b\u003e\u003c/td\u003e\n    \u003ctd\u003e\u003cb\u003e0.1799\u003c/b\u003e\u003c/td\u003e\n    \u003ctd\u003e\u003cb\u003e0.6120\u003c/b\u003e\u003c/td\u003e\n    \u003ctd\u003e\u003cb\u003e19.0619\u003c/b\u003e\u003c/td\u003e\n  \u003c/tr\u003e\n\u003c/tbody\u003e\n\u003c/table\u003e\n\n†: The latest results better than the paper.  \n*Experiments are conducted on the NVIDIA 4090 GPU. Results may be subject to some variation and randomness.*\n\n\n\u003ca id=\"simulation\"\u003e\u003c/a\u003e\n\n## 🕹️ Simulation\n\u003cimg src=\"https://github.com/ispc-lab/LiDAR4D/assets/51731102/ada49a62-8b53-47fe-8cc0-4d99af1ebad8\" width=75%\u003e  \n\u003c!-- \u003cimg src=\"https://github.com/ispc-lab/LiDAR4D/assets/51731102/1b34a7b4-4238-470a-acfd-499fe697e3d1\" width=75%\u003e   --\u003e\n\nAfter reconstruction, you can use the simulator to render and manipulate LiDAR point clouds in the whole scenario. It supports dynamic scene re-play, novel LiDAR configurations (`--fov_lidar`, `--H_lidar`, `--W_lidar`) and novel trajectory (`--shift_x`, `--shift_y`, `--shift_z`).  \nWe also provide a simple demo setting to transform LiDAR configurations from KITTI-360 to NuScenes, using `--kitti2nus` in the bash script.    \nCheck the sequence config and corresponding workspace and model path (`--ckpt`).  \nRun the following command:\n```bash\nbash run_kitti_lidar4d_sim.sh\n```\nThe results will be saved in the workspace folder.\n\n\n## Acknowledgement\nWe sincerely appreciate the great contribution of the following works:\n- [tiny-cuda-nn](https://github.com/NVlabs/tiny-cuda-nn/tree/master)\n- [LiDAR-NeRF](https://github.com/tangtaogo/lidar-nerf)\n- [NFL](https://research.nvidia.com/labs/toronto-ai/nfl/)\n- [K-Planes](https://github.com/sarafridov/K-Planes)\n\n\n## Citation\nIf you find our repo or paper helpful, feel free to support us with a star 🌟 or use the following citation:  \n```bibtex\n@inproceedings{zheng2024lidar4d,\n  title     = {LiDAR4D: Dynamic Neural Fields for Novel Space-time View LiDAR Synthesis},\n  author    = {Zheng, Zehan and Lu, Fan and Xue, Weiyi and Chen, Guang and Jiang, Changjun},\n  booktitle = {Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)},\n  year      = {2024}\n  }\n```\n\n\n## License\nAll code within this repository is under [Apache License 2.0](https://www.apache.org/licenses/LICENSE-2.0).\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fispc-lab%2FLiDAR4D","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fispc-lab%2FLiDAR4D","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fispc-lab%2FLiDAR4D/lists"}