{"id":13563302,"url":"https://github.com/baegwangbin/MaGNet","last_synced_at":"2025-04-03T19:32:50.488Z","repository":{"id":37709591,"uuid":"473633935","full_name":"baegwangbin/MaGNet","owner":"baegwangbin","description":"[CVPR 2022 Oral] Multi-View Depth Estimation by Fusing Single-View Depth Probability with Multi-View Geometry","archived":false,"fork":false,"pushed_at":"2022-06-20T23:31:44.000Z","size":2821,"stargazers_count":217,"open_issues_count":11,"forks_count":15,"subscribers_count":8,"default_branch":"master","last_synced_at":"2025-03-31T23:26:58.243Z","etag":null,"topics":["3d-reconstruction","computer-vision","cvpr2022","deep-learning","depth-estimation","multi-view-stereo","multiview-geometry","multiview-stereo","uncertainty","uncertainty-estimation"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/baegwangbin.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2022-03-24T14:10:01.000Z","updated_at":"2025-02-25T02:18:16.000Z","dependencies_parsed_at":"2022-07-16T18:16:58.027Z","dependency_job_id":null,"html_url":"https://github.com/baegwangbin/MaGNet","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/baegwangbin%2FMaGNet","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/baegwangbin%2FMaGNet/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/baegwangbin%2FMaGNet/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/baegwangbin%2FMaGNet/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/baegwangbin","download_url":"https://codeload.github.com/baegwangbin/MaGNet/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":247065470,"owners_count":20877785,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["3d-reconstruction","computer-vision","cvpr2022","deep-learning","depth-estimation","multi-view-stereo","multiview-geometry","multiview-stereo","uncertainty","uncertainty-estimation"],"created_at":"2024-08-01T13:01:17.559Z","updated_at":"2025-04-03T19:32:47.714Z","avatar_url":"https://github.com/baegwangbin.png","language":"Python","readme":"\n# Multi-View Depth Estimation by Fusing Single-View Depth Probability with Multi-View Geometry\n\nOfficial implementation of the paper\n\n\u003e **Multi-View Depth Estimation by Fusing Single-View Depth Probability with Multi-View Geometry**\n\u003e\n\u003e CVPR 2022 [oral] \n\u003e\n\u003e [Gwangbin Bae](https://baegwangbin.com), [Ignas Budvytis](https://mi.eng.cam.ac.uk/~ib255/), and [Roberto Cipolla](https://mi.eng.cam.ac.uk/~cipolla/)\n\u003e\n\u003e [[arXiv]](https://arxiv.org/abs/2112.08177) [[openaccess]](https://openaccess.thecvf.com/content/CVPR2022/html/Bae_Multi-View_Depth_Estimation_by_Fusing_Single-View_Depth_Probability_With_Multi-View_CVPR_2022_paper.html) [[oral presentation]](https://www.youtube.com/watch?v=LM113ibJVmQ)\n\n\u003cp align=\"center\"\u003e\n  \u003cimg width=100% src=\"https://github.com/baegwangbin/MaGNet/blob/master/figs/method.png?raw=true?raw=true\"\u003e\n\u003c/p\u003e\n\n*We present **MaGNet** (**M**onocular **a**nd **G**eometric **Net**work), a novel framework for fusing single-view depth probability with multi-view geometry, to improve the accuracy, robustness and efficiency of multi-view depth estimation. For each frame, MaGNet estimates a single-view depth probability distribution, parameterized as a pixel-wise Gaussian. The distribution estimated for the reference frame is then used to sample per-pixel depth candidates. Such probabilistic sampling enables the network to achieve higher accuracy while evaluating fewer depth candidates. We also propose depth consistency weighting for the multi-view matching score, to ensure that the multi-view depth is consistent with the single-view predictions. The proposed method achieves state-of-the-art performance on ScanNet, 7-Scenes and KITTI. Qualitative evaluation demonstrates that our method is more robust against challenging artifacts such as texture-less/reflective surfaces and moving objects.*\n\n\n## Datasets\n\nWe evaluated MaGNet on ScanNet, 7-Scenes and KITTI\n\n### ScanNet\n\n* In order to download ScanNet, you should submit an agreement to the Terms of Use. Please follow the instructions in [this link](http://kaldir.vc.in.tum.de/scannet_benchmark/documentation).\n* The folder should be organized as\n\n\u003e`/path/to/ScanNet` \\\n\u003e`/path/to/ScanNet/scans` \\\n\u003e`/path/to/ScanNet/scans/scene0000_00 ...` \\\n\u003e`/path/to/ScanNet/scans_test` \\\n\u003e`/path/to/ScanNet/scans_test/scene0707_00 ...` \n\n### 7-Scenes\n\n* Download all seven scenes (Chess, Fire, Heads, Office, Pumpkin, RedKitchen, Stairs) from [this link](https://www.microsoft.com/en-us/research/project/rgb-d-dataset-7-scenes/).\n* The folder should be organized as:\n\n\u003e`/path/to/SevenScenes` \\\n\u003e`/path/to/SevenScenes/chess ...` \n\n### KITTI \n\n* Download raw data from [this link](https://www.cvlibs.net/datasets/kitti/raw_data.php).\n* Download depth maps from [this link](https://www.cvlibs.net/datasets/kitti/eval_depth_all.php)\n* The folder should be organized as:\n\n\u003e`/path/to/KITTI` \\\n\u003e`/path/to/KITTI/rawdata` \\\n\u003e`/path/to/KITTI/rawdata/2011_09_26 ...` \\\n\u003e`/path/to/KITTI/train` \\\n\u003e`/path/to/KITTI/train/2011_09_26_drive_0001_sync ...` \\\n\u003e`/path/to/KITTI/val` \\\n\u003e`/path/to/KITTI/val/2011_09_26_drive_0002_sync ...`\n\n## Download model weights\n\nDownload model weights by\n\n```python\npython ckpts/download.py\n```\n\nIf some files are not downloaded properly, download them manually from [this link](https://drive.google.com/drive/u/0/folders/1O4yng6XFe8Wy2fHZp_xjO9KDTfYE7yDB) and place the files under `./ckpts`.\n\n## Install dependencies\n\nWe recommend using a virtual environment.\n```\npython3.6 -m venv --system-site-packages ./venv\nsource ./venv/bin/activate\n```\n\nInstall the necessary dependencies by\n```\npython3.6 -m pip install -r requirements.txt\n```\n\n## Test scripts\n\nIf you wish to evaluate the accuracy of our D-Net (single-view), run\n\n```python\npython test_DNet.py ./test_scripts/dnet/scannet.txt\npython test_DNet.py ./test_scripts/dnet/7scenes.txt\npython test_DNet.py ./test_scripts/dnet/kitti_eigen.txt\npython test_DNet.py ./test_scripts/dnet/kitti_official.txt\n```\n\nYou should get the following results:\n\n|Dataset|abs_rel|abs_diff|sq_rel|rmse|rmse_log|irmse|log_10|silog|a1|a2|a3|NLL|\n|-|-|-|-|-|-|-|-|-|-|-|-|-|\n|ScanNet|0.1186|0.2070|0.0493|0.2708|0.1461|0.1086|0.0515|10.0098|0.8546|0.9703|0.9928|2.2352|\n|7-Scenes|0.1339|0.2209|0.0549|0.2932|0.1677|0.1165|0.0566|12.8807|0.8308|0.9716|0.9948|2.7941|\n|KITTI (eigen)|0.0605|1.1331|0.2086|2.4215|0.0921|0.0075|0.0261|8.4312|0.9602|0.9946|0.9989|2.6443|\n|KITTI (official)|0.0629|1.1682|0.2541|2.4708|0.1021|0.0080|0.0270|9.5752|0.9581|0.9905|0.9971|1.7810|\n\nIn order to evaluate the accuracy of the full pipeline (multi-view), run\n\n```python\npython test_MaGNet.py ./test_scripts/magnet/scannet.txt\npython test_MaGNet.py ./test_scripts/magnet/7scenes.txt\npython test_MaGNet.py ./test_scripts/magnet/kitti_eigen.txt\npython test_MaGNet.py ./test_scripts/magnet/kitti_official.txt\n```\n\nYou should get the following results:\n\n|Dataset|abs_rel|abs_diff|sq_rel|rmse|rmse_log|irmse|log_10|silog|a1|a2|a3|NLL|\n|-|-|-|-|-|-|-|-|-|-|-|-|-|\n|ScanNet|0.0810|0.1466|0.0302|0.2098|0.1101|0.1055|0.0351|8.7686|0.9298|0.9835|0.9946|0.1454|\n|7-Scenes|0.1257|0.2133|0.0552|0.2957|0.1639|0.1782|0.0527|13.6210|0.8552|0.9715|0.9935|1.5605|\n|KITTI (eigen)|0.0535|0.9995|0.1623|2.1584|0.0826|0.0566|0.0235|7.4645|0.9714|0.9958|0.9990|1.8053|\n|KITTI (official)|0.0503|0.9135|0.1667|1.9707|0.0848|0.2423|0.0219|7.9451|0.9769|0.9941|0.9979|1.4750|\n\n## Training scripts\n\nIf you wish to train the models, run\n\n```python\npython train_DNet.py ./test_scripts/dnet/{scannet, kitti_eigen, kitti_official}.txt\npython train_FNet.py ./test_scripts/dnet/{scannet, kitti_eigen, kitti_official}.txt\npython train_MaGNet.py ./test_scripts/dnet/{scannet, kitti_eigen, kitti_official}.txt\n```\n\nNote that the `dataset_path` argument in the script `.txt` files should be modified\n\n## Citation\n\nIf you find our work useful in your research please consider citing our paper:\n\n```\n@InProceedings{Bae2022,\n  title = {Multi-View Depth Estimation by Fusing Single-View Depth Probability with Multi-View Geometry}\n  author = {Gwangbin Bae and Ignas Budvytis and Roberto Cipolla},\n  booktitle = {Proc. IEEE Conference on Computer Vision and Pattern Recognition (CVPR)},\n  year = {2022}                         \n}\n```\n","funding_links":[],"categories":["Python"],"sub_categories":[],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fbaegwangbin%2FMaGNet","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fbaegwangbin%2FMaGNet","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fbaegwangbin%2FMaGNet/lists"}