{"id":17595008,"url":"https://github.com/andreaconti/multi-view-guided-multi-view-stereo","last_synced_at":"2025-05-12T14:21:09.796Z","repository":{"id":73009876,"uuid":"551161595","full_name":"andreaconti/multi-view-guided-multi-view-stereo","owner":"andreaconti","description":"Code to reproduce results for Multi-View Guided Multi-View Stereo Paper","archived":false,"fork":false,"pushed_at":"2023-07-29T11:55:33.000Z","size":150,"stargazers_count":25,"open_issues_count":2,"forks_count":2,"subscribers_count":5,"default_branch":"main","last_synced_at":"2025-04-20T13:36:47.427Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/andreaconti.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2022-10-13T23:53:51.000Z","updated_at":"2025-03-03T02:13:57.000Z","dependencies_parsed_at":"2024-05-11T13:31:05.293Z","dependency_job_id":null,"html_url":"https://github.com/andreaconti/multi-view-guided-multi-view-stereo","commit_stats":null,"previous_names":[],"tags_count":1,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/andreaconti%2Fmulti-view-guided-multi-view-stereo","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/andreaconti%2Fmulti-view-guided-multi-view-stereo/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/andreaconti%2Fmulti-view-guided-multi-view-stereo/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/andreaconti%2Fmulti-view-guided-multi-view-stereo/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/andreaconti","download_url":"https://codeload.github.com/andreaconti/multi-view-guided-multi-view-stereo/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":253754346,"owners_count":21958843,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-10-22T07:25:54.163Z","updated_at":"2025-05-12T14:21:09.740Z","avatar_url":"https://github.com/andreaconti.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Multi-View Guided Multi-View Stereo\n\n\u003cp\u003e\n\u003cdiv align=\"center\"\u003e\n    \u003ca href=\"https://mattpoggi.github.io\"\u003eMatteo Poggi\u003c/a\u003e*\n    \u0026middot;\n    \u003ca href=\"https://andreaconti.github.io\"\u003eAndrea Conti\u003c/a\u003e*\n    \u0026middot;\n    \u003ca href=\"http://vision.deis.unibo.it/~smatt/Site/Home.html\"\u003eStefano Mattoccia\u003c/a\u003e\n    \u003ci\u003e*joint first authorship\u003c/i\u003e\n\u003c/div\u003e\n\u003cdiv align=\"center\"\u003e\n    \u003ca href=\"https://arxiv.org/pdf/2210.11467v1.pdf\"\u003e[Arxiv]\u003c/a\u003e\n    \u003ca href=\"https://andreaconti.github.io/projects/multiview_guided_multiview_stereo/\"\u003e[Project Page]\u003c/a\u003e\n    \u003ca href=\"https://github.com/andreaconti/multi-view-guided-multi-view-stereo/blob/main/results.ipynb\"\u003e[Demo]\u003c/a\u003e\n\u003c/div\u003e\n\u003c/p\u003e\n\nThis is the official source code of Multi-View Guided Multi-View Stereo presented at [IEEE/RSJ International Conference on Intelligent Robots and Systems](https://iros2022.org/)\n\n## Citation\n\n```\n@inproceedings{poggi2022guided,\n  title={Multi-View Guided Multi-View Stereo},\n  author={Poggi, Matteo and Conti, Andrea and Mattoccia, Stefano},\n  booktitle={IEEE/RSJ International Conference on Intelligent Robots and Systems},\n  note={IROS},\n  year={2022}\n}\n```\n\n## Load pretrained models and evaluate\n\nWe release many of the mvs networks tested in the paper trained on Blended-MVG or on Blended-MVG and fine-tuned on DTU, with and without sparse depth points. To load these models can be simply used the `torch.hub` API.\n\n```python\nmodel = torch.hub.load(\n    \"andreaconti/multi-view-guided-multi-view-stereo\",\n    \"mvsnet\",               # mvsnet | ucsnet | d2hc_rmvsnet | patchmatchnet | cas_mvsnet\n    pretrained=True,\n    dataset=\"blended_mvg\",  # blended_mvg | dtu_yao_blended_mvg\n    hints=\"not_guided\",     # mvguided_filtered | not_guided | guided | mvguided\n)\n```\n\nOnce loaded each model have the same following interface, moreover each pretrained models provides its training parameters under the attribute `train_params`.\n\n```python\ndepth = model(\n    images,        # B x N x 3 x H x W\n    intrinsics,    # B x N x 3 x 3\n    extrinsics,    # B x N x 4 x 4\n    depth_values,  # B x D (128 usually)\n    hints,         # B x 1 x H x W (optional)\n)\n```\n\nFinally, we provide also an interface over the datasets used as follows. In this case is required Pytorch Lightning as dependency and the dataset must be stored locally.\n\n```python\ndm = torch.hub.load(\n    \"andreaconti/multi-view-guided-multi-view-stereo\",\n    \"blended-mvg\",  # blended_mvg | blended_mvs | dtu\n    root=\"data/blended-mvg\",\n    hints=\"not_guided\",     # mvguided_filtered | not_guided | guided | mvguided\n    hints_density=0.03,\n)\ndm.prepare_data()\ndm.setup()\ndl = dm.train_dataloader()\n```\n\nIn [results.ipynb](https://github.com/andreaconti/multi-view-guided-multi-view-stereo/blob/main/results.ipynb) there is an example of how to reproduce some of the results showed in the paper through the `torch.hub` API.\n\n## Installation\n\nInstall the dependencies using Conda or [Mamba](https://github.com/mamba-org/mamba):\n\n```bash\n$ conda env create -f environment.yml\n$ conda activate guided-mvs\n```\n\n## Download the dataset(s)\n\nDownload the used datasets:\n\n* [DTU](http://roboimagedata.compute.dtu.dk/?page_id=36) preprocessed by [patchmatchnet](https://github.com/FangjinhuaWang/PatchmatchNet), [train](https://polybox.ethz.ch/index.php/s/ugDdJQIuZTk4S35) and [val](https://drive.google.com/file/d/1jN8yEQX0a-S22XwUjISM8xSJD39pFLL_/view?usp=sharing)\n* [BlendedMVG](https://github.com/YoYo000/BlendedMVS), download the lowres version from [BlendedMVS](https://1drv.ms/u/s!Ag8Dbz2Aqc81gVDu7FHfbPZwqhIy?e=BHY07t), [BlendedMVS+](https://1drv.ms/u/s!Ag8Dbz2Aqc81gVLILxpohZLEYiIa?e=MhwYSR), [BlendedMVS++](https://1drv.ms/u/s!Ag8Dbz2Aqc81gVHCxmURGz0UBGns?e=Tnw2KY)\n\nAnd organize them as follows under the data folder (sym-links works fine):\n \n```\ndata/dtu\n|-- train_data\n    |-- Cameras_1\n    |-- Depths_raw\n    |-- Rectified\n|-- test_data\n    |-- scan1 \n    |-- scan2\n    |-- ..\n\ndata/blended-mvs\n|-- \u003cscan hash name\u003e\n    |-- blended_images\n    |-- cams \n    |-- rendered_depth_maps\n|-- ..\n```\n\n## [Optional] Test everything is fine\n\nThis project implements some tests to preliminarily check everything is fine. Tests are grouped by different tags.\n\n``` \n# tages:\n# - data: tests related to the datasets\n# - dtu: tests related to the DTU dataset\n# - blended_mvs: tests related to blended MVS\n# - blended_mvg: tests related to blended MVG\n# - train: tests to launch all networks in fast dev run mode (1 batch of train val for each network for each dataset)\n# - slow: tests slow to be executed\n\n# EXAMPLES\n\n# runs all tests\n$ pytest\n\n# runs tests excluding slow ones\n$ pytest -m \"not slow\"\n\n# runs tests only on dtu\n$ pytest -m dtu\n\n# runs tests on data except for dtu ones\n$ pytest -m \"data and not dtu\"\n```\n\n## Training\n\nTo train a model, edit ``params.yaml`` specifying the model to be trained among the following:\n\n* [cas_mvsnet](https://arxiv.org/pdf/1912.06378.pdf)\n* [d2hc_rmvsnet](https://www.ecva.net/papers/eccv_2020/papers_ECCV/papers/123490647.pdf)\n* [mvsnet](https://arxiv.org/pdf/1804.02505.pdf)\n* [patchmatchnet](https://arxiv.org/pdf/2012.01411.pdf)\n* [ucsnet](https://arxiv.org/abs/1911.12012)\n\nThe dataset between ``dtu_yao``, `blended_mvs`, ``blended_mvg`` and the other training parameters, then hit:\n\n```\n# python3 train.py --help to see the options\n$ python3 train.py\n```\n\nThe best model is stored in ``output`` folder as ``model.ckpt`` along with a ``meta`` folder containing useful informations about the training executed.\n\n### Resume a training\n\nIf something bad happens or if you stop the training process using a keyboard interrupt (``Ctrl-C``) the checkpoints will not be deleted and you can resume\nthe training with the following option:\n\n```\n# resume the last checkpoint saved in output/ckpts (last epoch)\n$ python3 train.py --resume-from-checkpoint\n\n# resume a choosen checkpoint elsewere\n$ python3 train.py --resume-from-checkpoint my_checkpoint.ckpt\n```\n\nIt takes care to properly update the correct training logs on tensorboard.\n\n## Evaluation\n\nOnce you have trained the model you can evaluate it using ``eval.py``, here you have few options, specifically:\n\n* the **dataset** on which test\n* if evaluate using **guided** hints, **guided_integral** hints or none\n* the **hints density** to be used\n* the number of **views**\n\n```\n# see the options\n$ python3 eval.py --help\nusage: eval.py [-h] [--dataset {dtu_yao,blended_mvs,blended_mvg}]\n               [--hints {not_guided,guided,guided_integral}]\n               [--hints-density HINTS_DENSITY] [--views VIEWS]\n               [--loadckpt LOADCKPT] [--limit-scans LIMIT_SCANS]\n               [--skip-steps {1,2,3} [{1,2,3} ...]] [--save-scans-output]\n\n# EXAMPLES\n# without guided hints on dtu_yao, 3 views\n$ python3 eval.py\n\n# with guided hints and 5 views and density of 0.01\n$ python3 eval.py --hints guided --views 5\n\n# with integral guided hints 3 views and 0.03 density\n$ python3 eval.py --hints guided_integral --hints-density 0.03\n```\n\nResults will be stored under ``output/eval_\u003cdataset\u003e/[guided|not_guided|guided_integral]-[density=\u003chints-density\u003e]-views=\u003cviews\u003e``, for instance guiding on DTU with a 0.01 density and using 3 views the results will be in:\n\n* ``output/eval_dtu_yao/guided-density=0.01-views=3/``\n\nEach of these folders will contain the point cloud for each testing scene and a ``metrics.json`` file containing the final metrics, they will differ depending on the dataset used for evaluation.\n\n## Development\n\n### Environment\n\nTo develop you have to create a conda virtual environment and **also** install git hooks:\n\n```bash\n$ conda env create -f environment.yml\n$ conda activate guided-MVS\n$ pre-commit install\n```\n\nWhen you will commit [Black](https://github.com/psf/black) and [Isort](https://pypi.org/project/isort/) will be executed on the modified\nfiles.\n\n### VSCode specific settings\n\nIf you use Visual Studio Code its configuration and needed extensions are stored in ``.vscode``. Create a file in the root folder called ``.guided-mvs.code-workspace`` containing the following to load the conda environment properly:\n\n```json\n{\n    \"folders\": [\n        {\n            \"path\": \".\"\n        }\n    ],\n    \"settings\": {\n        \"python.pythonPath\": \"\u003cpath to your python executable (see in conda)\u003e\"\n    }\n}\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fandreaconti%2Fmulti-view-guided-multi-view-stereo","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fandreaconti%2Fmulti-view-guided-multi-view-stereo","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fandreaconti%2Fmulti-view-guided-multi-view-stereo/lists"}