{"id":13497351,"url":"https://github.com/YoYo000/MVSNet","last_synced_at":"2025-03-28T21:32:25.189Z","repository":{"id":40988662,"uuid":"139677225","full_name":"YoYo000/MVSNet","owner":"YoYo000","description":"MVSNet (ECCV2018) \u0026 R-MVSNet (CVPR2019)","archived":false,"fork":false,"pushed_at":"2021-06-02T05:35:09.000Z","size":6741,"stargazers_count":1398,"open_issues_count":75,"forks_count":304,"subscribers_count":44,"default_branch":"master","last_synced_at":"2024-10-31T13:34:22.949Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/YoYo000.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2018-07-04T06:20:01.000Z","updated_at":"2024-10-31T00:48:08.000Z","dependencies_parsed_at":"2022-08-03T02:30:31.999Z","dependency_job_id":null,"html_url":"https://github.com/YoYo000/MVSNet","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/YoYo000%2FMVSNet","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/YoYo000%2FMVSNet/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/YoYo000%2FMVSNet/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/YoYo000%2FMVSNet/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/YoYo000","download_url":"https://codeload.github.com/YoYo000/MVSNet/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":246105564,"owners_count":20724331,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-07-31T20:00:29.408Z","updated_at":"2025-03-28T21:32:23.522Z","avatar_url":"https://github.com/YoYo000.png","language":"Python","readme":"# MVSNet \u0026 R-MVSNet\n\n\u003cfont color=\"red\"\u003e [News] BlendedMVS dataset is released!!!\u003c/font\u003e ([project link](https://github.com/YoYo000/BlendedMVS)).\n\n## About\n[MVSNet](https://arxiv.org/abs/1804.02505) is a deep learning architecture for depth map inference from unstructured multi-view images, and [R-MVSNet](https://arxiv.org/abs/1902.10556) is its extension for scalable learning-based MVS reconstruction. If you find this project useful for your research, please cite:\n```\n@article{yao2018mvsnet,\n  title={MVSNet: Depth Inference for Unstructured Multi-view Stereo},\n  author={Yao, Yao and Luo, Zixin and Li, Shiwei and Fang, Tian and Quan, Long},\n  journal={European Conference on Computer Vision (ECCV)},\n  year={2018}\n}\n```\n```\n@article{yao2019recurrent,\n  title={Recurrent MVSNet for High-resolution Multi-view Stereo Depth Inference},\n  author={Yao, Yao and Luo, Zixin and Li, Shiwei and Shen, Tianwei and Fang, Tian and Quan, Long},\n  journal={Computer Vision and Pattern Recognition (CVPR)},\n  year={2019}\n}\n```\nIf [BlendedMVS dataset](https://github.com/YoYo000/BlendedMVS) is used in your research, please also cite:\n```\n@article{yao2020blendedmvs,\n  title={BlendedMVS: A Large-scale Dataset for Generalized Multi-view Stereo Networks},\n  author={Yao, Yao and Luo, Zixin and Li, Shiwei and Zhang, Jingyang and Ren, Yufan and Zhou, Lei and Fang, Tian and Quan, Long},\n  journal={Computer Vision and Pattern Recognition (CVPR)},\n  year={2020}\n}\n```\n\n## How to Use\n\n### Installation\n\n* Check out the source code ```git clone https://github.com/YoYo000/MVSNet```\n* Install cuda 9.0, cudnn 7.0 and python 2.7\n* Install Tensorflow and other dependencies by ```sudo pip install -r requirements.txt```\n\n### Download\n\n* Preprocessed training/validation data: [BlendedMVS](https://drive.google.com/open?id=1ilxls-VJNvJnB7IaFj7P0ehMPr7ikRCb), [DTU](https://drive.google.com/file/d/1eDjh-_bxKKnEuz5h-HXS7EDJn59clx6V/view) and [ETH3D](https://drive.google.com/open?id=1eqcv0Urr-c3Of8RKmTLhXrY_5cZCUvu5). More training resources could be found in [BlendedMVS github page](https://github.com/YoYo000/BlendedMVS) \n* Preprocessed testing data: [DTU testing set](https://drive.google.com/open?id=135oKPefcPTsdtLRzoDAQtPpHuoIrpRI_), [ETH3D testing set](https://drive.google.com/open?id=1hGft7rEFnoSrnTjY_N6vp5j1QBsGcnBB), [Tanks and Temples testing set](https://drive.google.com/open?id=1YArOJaX9WVLJh4757uE8AEREYkgszrCo) and [training set](https://drive.google.com/open?id=1vOfxAMFJUalhZzydzJa1AluRhzG7ZxHS)\n* Pretrained models: pretrained on [BlendedMVS](https://drive.google.com/open?id=1HacSpLl49xB77uBuI67ceZV9Rx_8wjPV), on [DTU](https://drive.google.com/open?id=1-1JyFT9ClqPO0kz0d_5I1_IHX05paS4h) and on [ETH3D](https://drive.google.com/open?id=1A3eZch06gkmvj-R0fasX6M_KNWGIOfYG)\n\n### Training\n\n* Enter mvsnet script folder: ``cd MVSNet/mvsnet``  \n* Train MVSNet on BlendedMVS, DTU and ETH3D: \u003cbr\u003e\n``python train.py --regularization '3DCNNs' --train_blendedmvs --max_w 768 --max_h 576 --max_d 128 --online_augmentation``  \u003cbr\u003e\n``python train.py --regularization '3DCNNs' --train_dtu --max_w 640 --max_h 512 --max_d 128``  \u003cbr\u003e\n``python train.py --regularization '3DCNNs' --train_eth3d --max_w 896 --max_h 480 --max_d 128``  \u003cbr\u003e\n* Train R-MVSNet: \u003cbr\u003e\n``python train.py --regularization 'GRU' --train_blendedmvs --max_w 768 --max_h 576 --max_d 128 --online_augmentation``  \u003cbr\u003e\n``python train.py --regularization 'GRU' --train_dtu --max_w 640 --max_h 512 --max_d 128``  \u003cbr\u003e\n``python train.py --regularization 'GRU' --train_eth3d --max_w 896 --max_h 480 --max_d 128``  \u003cbr\u003e\n* Specify your input training data folders using  ``--blendedmvs_data_root``, ``--dtu_data_root`` and ``--eth3d_data_root``\n* Specify your output log and model folders using ``--log_folder`` and  ``--model_folder``\n* Switch from BlendeMVS to BlendedMVG by replacing  using  ``--train_blendedmvs`` with ``--train_blendedmvg``\n\n### Validation\n\n* Validate MVSNet on BlendedMVS, DTU and ETH3D: \u003cbr\u003e\n``python validate.py --regularization '3DCNNs' --validate_set blendedmvs --max_w 768 --max_h 576 --max_d 128``\u003cbr\u003e\n``python validate.py --regularization '3DCNNs' --validate_set dtu --max_w 640 --max_h 512 --max_d 128``\u003cbr\u003e\n``python validate.py --regularization '3DCNNs' --validate_set eth3d --max_w 896 --max_h 480 --max_d 128``\u003cbr\u003e\n* Validate R-MVSNet: \u003cbr\u003e\n``python validate.py --regularization 'GRU' --validate_set blendedmvs --max_w 768 --max_h 576 --max_d 128``\u003cbr\u003e\n``python validate.py --regularization 'GRU' --validate_set dtu --max_w 640 --max_h 512 --max_d 128``\u003cbr\u003e\n``python validate.py --regularization 'GRU' --validate_set eth3d --max_w 896 --max_h 480 --max_d 128``\u003cbr\u003e\n* Specify your input model check point using  ``--pretrained_model_ckpt_path`` and ``--ckpt_step``\n* Specify your input training data folders using  ``--blendedmvs_data_root``, ``--dtu_data_root`` and ``--eth3d_data_root``\n* Specify your output result file using ``--validation_result_path``\n\n### Testing\n* Download test data [scan9](https://drive.google.com/file/d/17ZoojQSubtzQhLCWXjxDLznF2vbKz81E/view?usp=sharing) and unzip it to ``TEST_DATA_FOLDER`` folder\n* Run MVSNet (GTX1080Ti):  \u003cbr\u003e\n``python test.py --dense_folder TEST_DATA_FOLDER  --regularization '3DCNNs' --max_w 1152 --max_h 864 --max_d 192 --interval_scale 1.06``\n* Run R-MVSNet (GTX1080Ti):  \u003cbr\u003e\n``python test.py --dense_folder TEST_DATA_FOLDER  --regularization 'GRU' --max_w 1600 --max_h 1200 --max_d 256 --interval_scale 0.8``\n* Specify your input model check point using  ``--pretrained_model_ckpt_path`` and ``--ckpt_step``\n* Specify your input dense folder using ``--dense_folder``\n* Inspect the .pfm format outputs in ``TEST_DATA_FOLDER/depths_mvsnet`` using ``python visualize.py .pfm``. For example, the depth map and probability map for image `00000012` should look like:\n\n\u003cimg src=\"doc/image.png\" width=\"250\"\u003e   | \u003cimg src=\"doc/depth_example.png\" width=\"250\"\u003e |  \u003cimg src=\"doc/probability_example.png\" width=\"250\"\u003e\n:---------------------------------------:|:---------------------------------------:|:---------------------------------------:\nreference image                          |depth map                                |  probability map \n\n\n### Post-Processing\n\nR/MVSNet itself only produces per-view depth maps. To generate the 3D point cloud, we need to apply depth map filter/fusion for post-processing. As our implementation of this part is depended on the [Altizure](https://www.altizure.com/) internal library, currently we could not provide the corresponding code. Fortunately, depth map filter/fusion is a general step in MVS reconstruction, and there are similar implementations in other open-source MVS algorithms. We provide the script ``depthfusion.py`` to utilize [fusibile](https://github.com/kysucix/fusibile) for post-processing (thank Silvano Galliani for the excellent code!). \n\nTo run the post-processing: \n* Check out the modified version fusibile ```git clone https://github.com/YoYo000/fusibile```\n* Install fusibile by ```cmake .``` and ```make```, which will generate the executable at ``FUSIBILE_EXE_PATH``\n* Run post-processing (--prob_threshold 0.8 if using 3DCNNs):\n``python depthfusion.py --dense_folder TEST_DATA_FOLDER --fusibile_exe_path FUSIBILE_EXE_PATH --prob_threshold 0.3``\n* The final point cloud is stored in `TEST_DATA_FOLDER/points_mvsnet/consistencyCheck-TIME/final3d_model.ply`.\n\nWe observe that ``depthfusion.py`` produce similar but quantitatively worse result to our own implementation. For detailed differences, please refer to [MVSNet paper](https://arxiv.org/abs/1804.02505) and [Galliani's paper](https://www.cv-foundation.org/openaccess/content_iccv_2015/papers/Galliani_Massively_Parallel_Multiview_ICCV_2015_paper.pdf). The point cloud for `scan9` should look like:\n\n\n\u003cimg src=\"doc/fused_point_cloud.png\" width=\"375\"\u003e   | \u003cimg src=\"doc/gt_point_cloud.png\" width=\"375\"\u003e \n:--------------------------------------------------:|:----------------------------------------------:\npoint cloud result                          |ground truth point cloud\n\n\n### Reproduce Paper Results\n\nThe following steps are required to reproduce depth map/point cloud results:\n\n* Generate R/MVSNet inputs from SfM outputs.You can use our preprocessed testing data in the download section. (provided)\n* Run R/MVSNet testing script to generate depth maps for all views (provided)\n* Run R/MVSNet validation script to generate depth map validation results. (provided)\n* Apply variational depth map refinement for all views (optional, not provided)\n* Apply depth map filter and fusion to generate point cloud results (partially provided via fusibile)\n\nR-MVSNet point cloud results with full post-processing are also provided: [DTU evaluation point clouds](https://drive.google.com/open?id=1L0sQjIVYu2hYjwpwbWSN8k42QhkQDjbQ) \n\n\n## File Formats\n\nEach project folder should contain the following\n```\n.                          \n├── images                 \n│   ├── 00000000.jpg       \n│   ├── 00000001.jpg       \n│   └── ...                \n├── cams                   \n│   ├── 00000000_cam.txt   \n│   ├── 00000001_cam.txt   \n│   └── ...                \n└── pair.txt               \n```\nIf you want to apply R/MVSNet to your own data, please structure your data into such a folder. We also provide a simple script `colmap2mvsnet.py` to convert COLMAP SfM result to R/MVSNet input.\n\n### Image Files\nAll image files are stored in the `images` folder. We index each image using an 8 digit number starting from `00000000`. The following camera and output files use the same indexes as well. \n\n### Camera Files\nThe camera parameter of one image is stored in a ``cam.txt`` file. The text file contains the camera extrinsic `E = [R|t]`, intrinsic `K` and the depth range:\n```\nextrinsic\nE00 E01 E02 E03\nE10 E11 E12 E13\nE20 E21 E22 E23\nE30 E31 E32 E33\n\nintrinsic\nK00 K01 K02\nK10 K11 K12\nK20 K21 K22\n\nDEPTH_MIN DEPTH_INTERVAL (DEPTH_NUM DEPTH_MAX) \n```\nNote that the depth range and depth resolution are determined by the minimum depth `DEPTH_MIN`, the interval between two depth samples `DEPTH_INTERVAL`, and also the depth sample number `DEPTH_NUM` (or `max_d` in the training/testing scripts if `DEPTH_NUM` is not provided). We also left the `interval_scale` for controlling the depth resolution. The maximum depth is then computed as:\n```\nDEPTH_MAX = DEPTH_MIN + (interval_scale * DEPTH_INTERVAL) * (max_d - 1)\n``` \n\n### View Selection File\nWe store the view selection result in the `pair.txt`. For each reference image, we calculate its view selection scores with each of the other views, and store the 10 best views in the pair.txt file:\n```\nTOTAL_IMAGE_NUM\nIMAGE_ID0                       # index of reference image 0 \n10 ID0 SCORE0 ID1 SCORE1 ...    # 10 best source images for reference image 0 \nIMAGE_ID1                       # index of reference image 1\n10 ID0 SCORE0 ID1 SCORE1 ...    # 10 best source images for reference image 1 \n...\n``` \n\n### MVSNet input from SfM output\nWe provide a script to convert COLMAP SfM result to R/MVSNet input. After recovering SfM result and undistorting all images, COLMAP should generate a dense folder `COLMAP/dense/` containing an undistorted image folder `COLMAP/dense/images/` and an undistorted camera folder `COLMAP/dense/sparse/`. Then, you can apply the following script to generate the R/MVSNet input:\n\n``python colmap2mvsnet.py --dense_folder COLMAP/dense``\n\nThe depth sample number will be automatically computed using the inverse depth setting. If you want to generate the MVSNet input with a fixed depth sample number (e.g., 256), you could specified the depth number via ``--max_d 256``.\n\n### Output Format\nThe ``test.py`` script will create a `depths_mvsnet` folder to store the running results, including the depth maps, probability maps, scaled/cropped images and the corresponding cameras. The depth and probability maps are stored in `.pfm` format. We provide the python IO for pfm files in the `preprocess.py` script, and for the c++ IO, we refer users to the [Cimg](http://cimg.eu/) library. To inspect the pfm format results, you can simply type `python visualize.py .pfm`. \n\n## Changelog\n\n### 2020 April 13\n* Update BlendedMVG interface\n\n### 2020 March 2\n* Pretrained models on BlendedMVS, DTU and ETH3D (trained for 150000 iterations)\n* Update instructions in ``README.md``\n\n### 2020 Feb 29\n* Training with BlendedMVS dataset\n* Add `validate.py` script for depth map validation\n* Add `photometric_augmentation.py` script for online augmentation durig training\n\n### 2019 April 29\n* Add `colmap2mvsnet.py` script to convert COLMAP SfM result to MVSNet input, including depth range estimation and view selection\n\n### 2019 April 10\n* Add [Baiduyun](https://pan.baidu.com/s/1Wb9E6BWCJu4wZfwxm_t4TQ#list/path=%2F) (code: s2v2) link for mainland China users\n\n### 2019 March 14\n* Add R-MVSNet point clouds of DTU evaluation set\n\n### 2019 March 11\n* Add \"Reproduce Benchmarking Results\" section\n\n### 2019 March 7\n* MVSNet/R-MVSNet training/testing scripts\n* MVSNet and R-MVSNet models (trained for 100000 iterations)\n\n### 2019 March 1 \n* Implement R-MVSNet and GRU regularization\n* Network change: enable scale and center in batch normalization\n* Network change: replace UniNet with 2D UNet \n* Network change: use group normalization in R-MVSNet\n\n### 2019 Feb 28 \n* Use `tf.contrib.image.transform` for differentiable homography warping. Reconstruction is now x2 faster!\n\n","funding_links":[],"categories":["3DVision"],"sub_categories":["Depth/StereoMatching"],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FYoYo000%2FMVSNet","html_url":"https://awesome.ecosyste.ms/projects/github.com%2FYoYo000%2FMVSNet","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FYoYo000%2FMVSNet/lists"}