{"id":13442493,"url":"https://github.com/JUGGHM/PENet_ICRA2021","last_synced_at":"2025-03-20T14:31:11.448Z","repository":{"id":37374284,"uuid":"342535906","full_name":"JUGGHM/PENet_ICRA2021","owner":"JUGGHM","description":"ICRA 2021 \"Towards Precise and Efficient Image Guided Depth Completion\"","archived":false,"fork":false,"pushed_at":"2021-05-16T15:50:31.000Z","size":2420,"stargazers_count":324,"open_issues_count":38,"forks_count":46,"subscribers_count":6,"default_branch":"main","last_synced_at":"2024-10-28T05:59:16.477Z","etag":null,"topics":["depth-completion","pytorch"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/JUGGHM.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2021-02-26T10:07:18.000Z","updated_at":"2024-10-16T04:53:33.000Z","dependencies_parsed_at":"2022-07-09T07:00:49.176Z","dependency_job_id":null,"html_url":"https://github.com/JUGGHM/PENet_ICRA2021","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/JUGGHM%2FPENet_ICRA2021","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/JUGGHM%2FPENet_ICRA2021/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/JUGGHM%2FPENet_ICRA2021/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/JUGGHM%2FPENet_ICRA2021/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/JUGGHM","download_url":"https://codeload.github.com/JUGGHM/PENet_ICRA2021/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":244630106,"owners_count":20484314,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["depth-completion","pytorch"],"created_at":"2024-07-31T03:01:46.392Z","updated_at":"2025-03-20T14:31:11.070Z","avatar_url":"https://github.com/JUGGHM.png","language":"Python","funding_links":[],"categories":["Python"],"sub_categories":[],"readme":"## Important: About Improper Inference Time Statistics\nThe model's inference time is not properly reported in the original paper. This is because the original code ignores CUDA's asynchronous execution on CPU and GPU. To measure the inference time more precisely, the processes should be synchronized before recording current time:\n```\ntorch.cuda.synchronize()\n```\nWe restimated the inference time of following open-source models:\n\n|methods|runtime_not_synchronized|runtime_synchronized|\n|:----:|:----:|:----:|\n|PENet|0.032s|0.161s|\n|ENet|0.019s|0.064s|\n|[NLSPN](https://github.com/zzangjinsun/NLSPN_ECCV20)|0.127s|0.130s|\n|[ACMNet](https://github.com/sshan-zhao/ACMNet)|0.330s|0.350s|\n|[DeepLiDAR](https://github.com/JiaxiongQ/DeepLiDAR)|0.051s|0.351s|\n|[MSG-CHN](https://github.com/anglixjtu/msg_chn_wacv20)|0.011s|0.035s|\n|[FusionNet](https://github.com/wvangansbeke/Sparse-Depth-Completion)|0.022s|0.029s|\n\nWe thank [wdjose](https://github.com/JUGGHM/PENet_ICRA2021/issues/4) for pointing out this problem. In addition,  ENet is more recommanded for real-time applications.\n\n# PENet: Precise and Efficient Depth Completion\nThis repo is the PyTorch implementation of our paper to appear in ICRA2021 on [\"Towards Precise and Efficient Image Guided Depth Completion\"](https://arxiv.org/abs/2103.00783), developed by\nMu Hu, Shuling Wang, Bin Li, Shiyu Ning, Li Fan, and [Xiaojin Gong](https://person.zju.edu.cn/en/gongxj) at Zhejiang University and Huawei Shanghai.\n\nCreate a new issue for any code-related questions. Feel free to direct me as well at muhu@zju.edu.cn for any paper-related questions.\n\n## Results\n+ The proposed full model ranks 1st in the [KITTI depth completion](http://www.cvlibs.net/datasets/kitti/eval_depth.php?benchmark=depth_completion) online leaderboard at the time of submission.\n+ It infers much faster than most of the top ranked methods.\n\u003cdiv align=center\u003e\u003cimg src=\"https://github.com/JUGGHM/PENet_ICRA2021/blob/main/images/Comparison.png\" width = \"100%\" height = \"100%\" /\u003e\u003c/div\u003e\n\n+ Both ENet and PENet can be trained thoroughly on 2x11G GPU.\n+ Our network is trained with the KITTI dataset alone, not pretrained on Cityscapes or other similar driving dataset (either synthetic or real).\n\n## Method\n### A Strong Two-branch Backbone\n#### Revisiting the popular two-branch architecture\n\u003cdiv align=center\u003e\u003cimg src=\"https://github.com/JUGGHM/PENet_ICRA2021/blob/main/images/Backbone.png\" width = \"100%\" height = \"100%\" /\u003e\u003c/div\u003e\n\nThe two-branch backbone is designed to thoroughly exploit color-dominant and depth-dominant information from\ntheir respective branches and make the fusion of two modalities effective. Note that it is the depth prediction result\nobtained from the color-dominant branch that is input to the depth-dominant branch, not a guidance map like those in [DeepLiDAR](https://github.com/JiaxiongQ/DeepLiDAR) and [FusionNet](https://github.com/wvangansbeke/Sparse-Depth-Completion).\n\n#### Geometric convolutional Layer\n\u003cdiv align=center\u003e\u003cimg src=\"https://github.com/JUGGHM/PENet_ICRA2021/blob/main/images/Geometric_Encoding.png\" width = \"60%\" height = \"60%\" /\u003e\u003c/div\u003e\n\nTo encode 3D geometric information, it simply augments a conventional convolutional layer via concatenating a 3D position map to the layer’s input.\n\n### Dilated and Accelerated CSPN++\n#### Dilated CSPN\n\u003cdiv align=center\u003e\u003cimg src=\"https://github.com/JUGGHM/PENet_ICRA2021/blob/main/images/Dilated_CSPN.png\" width = \"60%\" height = \"60%\" /\u003e\u003c/div\u003e\n\nWe introduce a dilation strategy similar to the well known dilated convolutions to enlarge the propagation neighborhoods.\n\n#### Accelerated CSPN\n\u003cdiv align=center\u003e\u003cimg src=\"https://github.com/JUGGHM/PENet_ICRA2021/blob/main/images/Accelerated_CSPN.png\" width = \"100%\" height = \"100%\" /\u003e\u003c/div\u003e\n\nWe design an implementation that makes the propagation from each neighbor truly parallel, which greatly accelerates the propagation procedure.\n\n## Contents\n1. [Dependency](#dependency)\n0. [Data](#data)\n0. [Trained Models](#trained-models)\n0. [Commands](#commands)\n0. [Citation](#citation)\n\n\n## Dependency\nOur released implementation is tested on.\n+ Ubuntu 16.04\n+ Python 3.7.4 (Anaconda 2019.10)\n+ PyTorch 1.3.1 / torchvision 0.4.2\n+ NVIDIA CUDA 10.0.130\n+ 4x NVIDIA GTX 2080 Ti GPUs\n\n```bash\npip install numpy matplotlib Pillow\npip install scikit-image\npip install opencv-contrib-python==3.4.2.17\n```\n\n## Data\n- Download the [KITTI Depth](http://www.cvlibs.net/datasets/kitti/eval_depth.php?benchmark=depth_completion) Dataset and [KITTI Raw](http://www.cvlibs.net/datasets/kitti/raw_data.php) Dataset from their websites.\nThe overall data directory is structured as follows:\n```\n├── kitti_depth\n|   ├── depth\n|   |   ├──data_depth_annotated\n|   |   |  ├── train\n|   |   |  ├── val\n|   |   ├── data_depth_velodyne\n|   |   |  ├── train\n|   |   |  ├── val\n|   |   ├── data_depth_selection\n|   |   |  ├── test_depth_completion_anonymous\n|   |   |  |── test_depth_prediction_anonymous\n|   |   |  ├── val_selection_cropped\n```\n\n```\n├── kitti_raw\n|   ├── 2011_09_26\n|   ├── 2011_09_28\n|   ├── 2011_09_29\n|   ├── 2011_09_30\n|   ├── 2011_10_03\n```\n\n## Trained Models\nDownload our pre-trained models:\n- PENet (*i.e.*, the proposed full model with dilation_rate=2): [Download Here](https://drive.google.com/file/d/1RDdKlKJcas-G5OA49x8OoqcUDiYYZgeM/view?usp=sharing)\n- ENet (*i.e.*, the backbone): [Download Here](https://drive.google.com/file/d/1TRVmduAnrqDagEGKqbpYcKCT307HVQp1/view?usp=sharing)\n\n Note that we don't need to decompress the pre-trained models. Just load the files of .pth.tar format directly.\n\n## Commands\nA complete list of training options is available with\n```bash\npython main.py -h\n```\n### Training\n![Training Pipeline](https://github.com/JUGGHM/PENet_ICRA2021/blob/main/images/Training.png \"Training\")\n\nHere we adopt a multi-stage training strategy to train the backbone, DA-CSPN++, and the full model progressively. However, end-to-end training is feasible as well.\n\n1. Train ENet (Part Ⅰ)\n```bash\nCUDA_VISIBLE_DEVICES=\"0,1\" python main.py -b 6 -n e\n# -b for batch size\n# -n for network model\n```\n\n2. Train DA-CSPN++ (Part Ⅱ)\n```bash\n\nCUDA_VISIBLE_DEVICES=\"0,1\" python main.py -b 6 -f -n pe --resume [enet-checkpoint-path]\n# -f for freezing the parameters in the backbone\n# --resume for initializing the parameters from the checkpoint\n```\n\n3. Train PENet (Part Ⅲ)\n```bash\nCUDA_VISIBLE_DEVICES=\"0,1\" python main.py -b 10 -n pe -he 160 -w 576 --resume [penet-checkpoint-path]\n# -he, -w for the image size after random cropping\n```\n\n### Evalution\n```bash\nCUDA_VISIBLE_DEVICES=\"0\" python main.py -b 1 -n e --evaluate [enet-checkpoint-path]\nCUDA_VISIBLE_DEVICES=\"0\" python main.py -b 1 -n pe --evaluate [penet-checkpoint-path]\n# test the trained model on the val_selection_cropped data\n```\n\n### Test\n```bash\nCUDA_VISIBLE_DEVICES=\"0\" python main.py -b 1 -n pe --evaluate [penet-checkpoint-path] --test\n# generate and save results of the trained model on the test_depth_completion_anonymous data\n```\n\n## Citation\nIf you use our code or method in your work, please cite the following:\n\n\t@article{hu2020PENet,\n\t\ttitle={Towards Precise and Efficient Image Guided Depth Completion},\n\t\tauthor={Hu, Mu and Wang, Shuling and Li, Bin and Ning, Shiyu and Fan, Li and Gong, Xiaojin},\n\t\tbooktitle={ICRA},\n\t\tyear={2021}\n\t}\n\n## Related Repositories\nThe original code framework is rendered from [\"Self-supervised Sparse-to-Dense:  Self-supervised Depth Completion from LiDAR and Monocular Camera\"](https://github.com/fangchangma/self-supervised-depth-completion). It is developed by [Fangchang Ma](http://www.mit.edu/~fcma/), Guilherme Venturelli Cavalheiro, and [Sertac Karaman](http://karaman.mit.edu/) at MIT.\n\nThe part of CoordConv is rendered from [\"An intriguing failing of convolutional neural networks and the CoordConv\"](https://github.com/mkocabas/CoordConv-pytorch).\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FJUGGHM%2FPENet_ICRA2021","html_url":"https://awesome.ecosyste.ms/projects/github.com%2FJUGGHM%2FPENet_ICRA2021","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FJUGGHM%2FPENet_ICRA2021/lists"}