{"id":18678235,"url":"https://github.com/wvangansbeke/sparse-depth-completion","last_synced_at":"2025-04-05T19:14:49.258Z","repository":{"id":37405512,"uuid":"167838974","full_name":"wvangansbeke/Sparse-Depth-Completion","owner":"wvangansbeke","description":"Predict dense depth maps from sparse and noisy LiDAR frames guided by RGB images.  (Ranked 1st place on KITTI) [MVA 2019]","archived":false,"fork":false,"pushed_at":"2022-05-01T07:07:07.000Z","size":150,"stargazers_count":492,"open_issues_count":7,"forks_count":78,"subscribers_count":16,"default_branch":"master","last_synced_at":"2025-03-29T18:09:03.584Z","etag":null,"topics":["computer-vision","deep-learning","depth-completion","depth-prediction","kitti","lidar","noisy-data","pytorch","sensor-fusion"],"latest_commit_sha":null,"homepage":"https://arxiv.org/pdf/1902.05356.pdf","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"other","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/wvangansbeke.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE.txt","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2019-01-27T17:47:45.000Z","updated_at":"2025-03-12T10:04:39.000Z","dependencies_parsed_at":"2022-07-20T12:17:15.852Z","dependency_job_id":null,"html_url":"https://github.com/wvangansbeke/Sparse-Depth-Completion","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/wvangansbeke%2FSparse-Depth-Completion","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/wvangansbeke%2FSparse-Depth-Completion/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/wvangansbeke%2FSparse-Depth-Completion/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/wvangansbeke%2FSparse-Depth-Completion/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/wvangansbeke","download_url":"https://codeload.github.com/wvangansbeke/Sparse-Depth-Completion/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":247386266,"owners_count":20930619,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["computer-vision","deep-learning","depth-completion","depth-prediction","kitti","lidar","noisy-data","pytorch","sensor-fusion"],"created_at":"2024-11-07T09:36:32.780Z","updated_at":"2025-04-05T19:14:49.216Z","avatar_url":"https://github.com/wvangansbeke.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Sparse-Depth-Completion\n\nThis repo contains the implementation of our paper [Sparse and Noisy LiDAR Completion with RGB Guidance and Uncertainty](https://arxiv.org/abs/1902.05356) by [Wouter Van Gansbeke](https://github.com/wvangansbeke), Davy Neven, Bert De Brabandere and Luc Van Gool.\n\nIf you find this interesting or relevant to your work, consider citing:\n\n```\n@inproceedings{wvangansbeke_depth_2019,\n    author={Van Gansbeke, Wouter and Neven, Davy and De Brabandere, Bert and Van Gool, Luc},\n    booktitle={2019 16th International Conference on Machine Vision Applications (MVA)},\n    title={Sparse and Noisy LiDAR Completion with RGB Guidance and Uncertainty},\n    year={2019},\n    pages={1-6},\n    organization={IEEE}\n}\n```\n\n## License\n\nThis software is released under a creative commons license which allows for personal and research use only. For a commercial license please contact the authors. You can view a license summary [here](http://creativecommons.org/licenses/by-nc/4.0/)\n\n## Introduction\nMonocular depth prediction methods fail to generate absolute and precise depth maps and stereoscopic approaches are still significantly outperformed by LiDAR based approaches. The goal of the depth completion task is to generate dense depth predictions from sparse and irregular point clouds. This project makes use of uncertainty to combine multiple sensor data in order to generate accurate depth predictions. Mapped lidar points together with RGB images (monocular) are used in this framework. This method holds the **1st place** entry on the [KITTI depth completion benchmark](http://www.cvlibs.net/datasets/kitti/eval_depth.php?benchmark=depth_completion) at the time of submission of the paper.\n\nThe contribution of this paper is threefold:\n* Global and local information are combined in order to accurately complete and correct the sparse and noisy LiDAR input. Monocular RGB images are used for the guidance of this depth completion task.\n* Confidence maps are learned for the global branch and the local branch in an unsupervised manner. The predicted depth maps are weighted by their respective confidence map. This is the late fusion technique used in our framework.\n* This method ranks first on the KITTI depth completion benchmark without using additional data or postprocessing.\n\nSee full demo on [YouTube](https://www.youtube.com/watch?v=Kr0W7io5rHw\u0026feature=youtu.be). The predictions of our model for the KITTI test set can be downloaded [here](https://drive.google.com/drive/folders/1U7dvH4sC85KRVuV19fRpaMzJjE-m3D9x).\n\n![demo](https://user-images.githubusercontent.com/9694230/51806092-db766c00-2275-11e9-8de0-888bed0fc9e8.gif)\n\n\n## Requirements\nPython 3.7\nThe most important packages are pytorch, torchvision, numpy, pillow and matplotlib.\n(Works with Pytorch 1.1)\n\n\n## Dataset\nThe [Kitti dataset](www.cvlibs.net/datasets/kitti/) has been used. First download the dataset of the depth completion. Secondly, you'll need to unzip and download the camera images from kitti. \nI used the file `download_raw_files.sh`, but this is at your own risk. Make sure you understand it, otherwise don't use it. If you want to keep it safe, go to kitti's website. \n\nThe complete dataset consists of 85898 training samples, 6852 validation samples, 1000 selected validation samples and 1000 test samples.\n\n## Preprocessing\nThis step is optional, but allows you to transform the images to jpgs and to downsample the original lidar frames. This will create a new dataset in $dest.\nYou can find the required preprocessing in:\n`Datasets/Kitti_loader.py`\n\nRun:\n\n`source Shell/preprocess $datapath $dest $num_samples`\n\n(Firstly, I transformed the png's to jpg - images to save place. Secondly, two directories are built i.e. one for training and one for validation. See `Datasets/Kitti_loader.py`)\n\nDataset structure should look like this:\n```\n|--depth selection\n|-- Depth\n     |-- train\n           |--date\n               |--sequence1\n               | ...\n     |--validation\n|--RGB\n    |--train\n         |--date\n             |--sequence1\n             | ...\n    |--validation\n```\n\n\n## Run Code\nTo run the code:\n\n`python main.py --data_path /path/to/data/ --lr_policy plateau`\n\nFlags:\n- Set flag \"input_type\" to rgb or depth.\n- Set flag \"pretrained\" to true or false to use a model pretrained on Cityscapes for the global branch.\n- See `python main.py --help` for more information.\n\nor \n\n`source Shell/train.sh $datapath`\n\ncheckout more details in the bash file.\n\n## Trained models\nOur network architecture is based on [ERFNet](https://github.com/Eromera/erfnet_pytorch).\n\nYou can find the model pretrained on Cityscapes [here](https://drive.google.com/drive/folders/1U7dvH4sC85KRVuV19fRpaMzJjE-m3D9x?usp=sharing). This model is used for the global network.\n\nYou can find a fully trained model and its corresponding predictions for the KITTI test set [here](https://drive.google.com/drive/folders/1U7dvH4sC85KRVuV19fRpaMzJjE-m3D9x?usp=sharing). \nThe RMSE is around 802 mm on the selected validation set for this model as reported in the paper. \n\nTo test it: \nSave the model in a folder in the `Saved` directory.\n\nand execute the following command:\n\n`source Test/test.sh /path/to/directory_with_saved_model/ $num_samples /path/to/dataset/ /path/to/directory_with_ground_truth_for_selected_validation_files/`\n\n(You might have to recompile the C files for testing, provided by KITTI, if your architecture is different from mine)\n\n## Results\n\nComparision with state-of-the-art:\n\n![results](https://user-images.githubusercontent.com/9694230/59205060-49c32780-8ba2-11e9-8a87-34d8c3f99756.PNG)\n\n\n## Discussion\n\nPractical discussion:\n\n- I recently increased the stability of the training process and I also made the convergence faster by adding some skip connections between the global and local networks.\nInitially I only used guidance by multiplication with an attention map (=probability), but found out that it is less robust and that differences between a focal MSE and vanilla MSE loss function were now negligible.\nBe aware that this change will alter the appearance of the confidence maps since fusion happens at mutliple stages now.\n\n- Feel free to experiment with different architectures for the global or local network. It is easy to add new architectures to `Models/__init__.py`\n\n- I used a Tesla V100 GPU for evaluation.\n\n## Acknowledgement\nThis work was supported by Toyota, and was carried out at the TRACE Lab at KU Leuven (Toyota Research on Automated Cars in Europe - Leuven)\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fwvangansbeke%2Fsparse-depth-completion","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fwvangansbeke%2Fsparse-depth-completion","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fwvangansbeke%2Fsparse-depth-completion/lists"}