{"id":13699629,"url":"https://github.com/NVlabs/intrinsic3d","last_synced_at":"2025-05-04T16:35:25.750Z","repository":{"id":44546184,"uuid":"164033138","full_name":"NVlabs/intrinsic3d","owner":"NVlabs","description":"Intrinsic3D - High-Quality 3D Reconstruction by Joint Appearance and Geometry Optimization with Spatially-Varying Lighting (ICCV 2017)","archived":false,"fork":false,"pushed_at":"2022-07-04T09:37:08.000Z","size":132,"stargazers_count":453,"open_issues_count":1,"forks_count":80,"subscribers_count":34,"default_branch":"master","last_synced_at":"2024-11-13T05:36:01.529Z","etag":null,"topics":["3d","3d-reconstruction","computer-vision","geometry","iccv","iccv-2017","iccv2017","intrinsic3d","keyframes","lighting","nvidia","rgbd","sdf","shape-from-shading","spherical-harmonics","surface-reconstruction","tum"],"latest_commit_sha":null,"homepage":"https://vision.in.tum.de/_media/spezial/bib/maier2017intrinsic3d.pdf","language":"C++","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"bsd-3-clause","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/NVlabs.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2019-01-03T23:22:14.000Z","updated_at":"2024-11-10T03:58:00.000Z","dependencies_parsed_at":"2022-08-12T11:20:22.346Z","dependency_job_id":null,"html_url":"https://github.com/NVlabs/intrinsic3d","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/NVlabs%2Fintrinsic3d","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/NVlabs%2Fintrinsic3d/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/NVlabs%2Fintrinsic3d/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/NVlabs%2Fintrinsic3d/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/NVlabs","download_url":"https://codeload.github.com/NVlabs/intrinsic3d/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":252366656,"owners_count":21736612,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["3d","3d-reconstruction","computer-vision","geometry","iccv","iccv-2017","iccv2017","intrinsic3d","keyframes","lighting","nvidia","rgbd","sdf","shape-from-shading","spherical-harmonics","surface-reconstruction","tum"],"created_at":"2024-08-02T20:00:38.599Z","updated_at":"2025-05-04T16:35:25.320Z","avatar_url":"https://github.com/NVlabs.png","language":"C++","funding_links":[],"categories":["3D reconstruction","C++"],"sub_categories":["Intrinsic3D"],"readme":"# Intrinsic3D\n\n## High-Quality 3D Reconstruction by Joint Appearance and Geometry Optimization with Spatially-Varying Lighting (ICCV 2017)\n\n[![Intrinsic3D](https://vision.in.tum.de/_media/data/datasets/intrinsic3d/maier2017intrinsic3d_teaser.jpg?w=700\u0026tok=b8e6f7)](https://vision.in.tum.de/_media/spezial/bib/maier2017intrinsic3d.pdf)\n\n### License ###\nCopyright (c) 2019, NVIDIA Corporation and Technical University of Munich. All Rights Reserved.\nThe Intrinsic3D source code is available under the [BSD license](http://opensource.org/licenses/BSD-3-Clause), please see the [LICENSE](LICENSE) file for details.\nAll data in the [Intrinsic3D Dataset](https://vision.in.tum.de/data/datasets/intrinsic3d) is licensed under a [Creative Commons 4.0 Attribution License (CC BY 4.0)](https://creativecommons.org/licenses/by/4.0/), unless stated otherwise.\n\n### Resources ###\n* [Project page @ NVIDIA Research](https://research.nvidia.com/publication/2017-10_Intrinsic3D%3A-High-Quality-3D)\n* [Intrinsic3D Dataset @ TUM](https://vision.in.tum.de/data/datasets/intrinsic3d)\n* [ICCV 2017 Paper (PDF)](https://vision.in.tum.de/_media/spezial/bib/maier2017intrinsic3d.pdf)\n* [ArXiv (PDF)](https://arxiv.org/pdf/1708.01670.pdf)\n \n### Project members ###\n* [Robert Maier](https://vision.in.tum.de/members/maierr), Technische Universität München\n* [Kihwan Kim](https://research.nvidia.com/person/kihwan-kim), NVIDIA\n* [Daniel Cremers](https://vision.in.tum.de/members/cremers), Technische Universität München\n* [Jan Kautz](https://research.nvidia.com/person/jan-kautz), NVIDIA\n* [Matthias Nießner](http://www.niessnerlab.org/members/matthias_niessner/profile.html), Technische Universität München\n\n## Summary\n**Intrinsic3D** is a method to obtain high-quality 3D reconstructions from low-cost RGB-D sensors.\nThe algorithm recovers fine-scale geometric details and sharp surface textures by simultaneously optimizing for reconstructed geometry, surface albedos, camera poses and scene lighting.\n\nThis work is based on our publication  \n* [Intrinsic3D: High-Quality 3D Reconstruction by Joint Appearance and Geometry Optimization with Spatially-Varying Lighting](https://vision.in.tum.de/_media/spezial/bib/maier2017intrinsic3d.pdf)  \nInternational Conference on Computer Vision (ICCV) 2017.\n\nIf you find our source code or [dataset](https://vision.in.tum.de/data/datasets/intrinsic3d) useful in your research, please cite our work as follows:\n```\n@inproceedings{maier2017intrinsic3d,\n   title = {{Intrinsic3D}: High-Quality {3D} Reconstruction by Joint Appearance and Geometry Optimization with Spatially-Varying Lighting},\n   author = {Maier, Robert and Kim, Kihwan and Cremers, Daniel and Kautz, Jan and Nie{\\ss}ner, Matthias},\n   booktitle = {International Conference on Computer Vision (ICCV)},\n   year = {2017}\n}\n```\n\n\n## Installation\nAs the code was mostly developed and tested on Ubuntu Linux (16.10 and 18.10), we only provide the build instructions for Ubuntu in the following.\nHowever, the code should also work on Windows with Visual Studio 2013.\n\nPlease first clone the source code:\n```\ngit clone https://github.com/NVlabs/intrinsic3d.git\n```\n\n### Dependencies\nBuilding Intrinsic3D requires\n[CMake](https://cmake.org/download/),\n[Eigen](http://eigen.tuxfamily.org/),\n[OpenCV 4](https://opencv.org/releases.html),\n[Boost](http://www.boost.org/users/download/) and\n[Ceres Solver](http://ceres-solver.org/) (with [CXSparse](https://github.com/TheFrenchLeaf/CXSparse) on Windows)\nas third-party libraries.\nThe following command installs the dependencies from the default Ubuntu repositories:\n```\nsudo apt install cmake libeigen3-dev libboost-dev libboost-filesystem-dev libboost-graph-dev libboost-system-dev libopencv-dev\n```\n\nPlease install [Ceres Solver](http://ceres-solver.org/installation.html) using the following commands (if not installed already):\n```\n# create third_party subdirectory (in Intrinsic3D root folder)\nmkdir third_party \u0026\u0026 cd third_party\n\n# install Ceres Solver dependencies\nsudo apt install libgoogle-glog-dev libatlas-base-dev libsuitesparse-dev\n\n# download and extract Ceres Solver source code package\nwget http://ceres-solver.org/ceres-solver-2.1.0.tar.gz\ntar xvzf ceres-solver-2.1.0.tar.gz\ncd ceres-solver-2.1.0\n\n# compile and install Ceres Solver\nmkdir build-ceres \u0026\u0026 cd build-ceres\ncmake .. -DCMAKE_BUILD_TYPE=Release -DCMAKE_INSTALL_PREFIX=$PWD/../../ -DCXX11=ON -DSUITESPARSE=ON -DCXSPARSE=ON -DEIGENSPARSE=ON -DBUILD_EXAMPLES=OFF -DBUILD_TESTING=OFF\nmake -j6\nmake install\n\n# go back into source code root\ncd ../../../\n```\n\n### Build\nTo compile Intrinsic3D, use the standard CMake approach:\n```\nmkdir build\ncd build\ncmake .. -DCeres_DIR=$PWD/../third_party/lib/cmake/Ceres\nmake -j6\n```\n\n\n## Data\nDownload one of the RGB-D sequences from the [Intrinsic3D Dataset](https://vision.in.tum.de/data/datasets/intrinsic3d). If you want to use one of your own datasets, you can convert it into the dataset format specified [here](https://vision.in.tum.de/data/datasets/intrinsic3d#format).\nIn the following, we will reconstruct and refine the [Lion sequence](https://vision.in.tum.de/_media/data/datasets/intrinsic3d/lion-rgbd.zip) using our approach.\n```\n# creating working folder in data/\ncd ../data\nmkdir lion \u0026\u0026 cd lion\n\n# download, unzip and rename lion dataset\nwget https://vision.in.tum.de/_media/data/datasets/intrinsic3d/lion-rgbd.zip\nunzip lion-rgbd.zip\nmv lion-rgbd rgbd\n```\n\n\n## Intrinsic3D Usage\nTo run the Intrinsic3D applications, we continue in the current data folder ```data/lion/``` and copy the default YAML configuration files into it:\n```\n# copy default configuration files into current data folder\ncp ../*.yml .\n```\nThe dataset configuration file ```sensor.yml``` is loaded in all applications and specifies the input RGB-D dataset.\nPlease note that the working folder of each application will be set to the folder containing the ```sensor.yml``` passed as argument.\nThe outputs will be generated in the subfolders ```fusion/``` and  ```intrinsic3d/``` respectively.\n\n### Keyframe Selection\nWe first run the keyframe selection to discard blurry frames, based on a single-frame blurriness measure:\n```\n../../build/bin/AppKeyframes -s=sensor.yml -k=keyframes.yml\n```\nThis will generate the keyframes file ```fusion/keyframes.txt```.\nThe window size for keyframe selection can be adjusted through the parameter ```window_size``` in the ```keyframes.yml``` configuration file.\n\n### SDF Fusion\nNext, the RGB-D input frames are fused in a voxel-hashed signed distance field (SDF), which produces an output 3D triangle mesh ```fusion/mesh_0.004.ply``` and an initial SDF volume\n```fusion/volume_0.004.tsdf```:\n\n```\n../../build/bin/AppFusion -s=sensor.yml -f=fusion.yml\n```\nSince the framework is very computationally demanding, it is recommended to crop the 3D reconstruction already during the SDF fusion process.\nYou can therefore specify the 3D clip coordinates (in absolute coordinates of the first camera frame) by setting the ```crop_*``` parameters in the ```fusion.yml``` configuration file. \nTo disable clipping, you can set all ```crop_*``` parameters to 0.0.\nTo accelerate finding a suitable clip volume, increase the ```voxel_size``` value and integrate keyframes only (by setting the parameter ```keyframes```).\nFor reference, here are suitable crop bounds for the [Intrinsic3D Dataset](https://vision.in.tum.de/data/datasets/intrinsic3d) sequences: \n```\nLion: left -0.09, right 1.55; top -0.58, bottom 0.26; front 0.0, back 2.0.\nGate: left -0.52, right 0.55; top -1.1, bottom 0.35; front 0.0, back 1.0.\nHieroglyphics: left -0.5, right 0.45; top -1.2, bottom 0.2; front 0.0, back 1.0.\nTomb Statuary: left -0.15, right 0.25; top -0.02, bottom 0.52; front 0.0, back 0.75.\nBricks: left -0.3, right 2.1; top -0.3, bottom 0.3; front 0.0, back 2.0.\n```\n\n### Intrinsic3D\nThe Intrinsic3D approach takes the initial SDF volume (```fusion/volume_0.004.tsdf```) and selected keyframes (```fusion/keyframes.txt```) and jointly optimizes the scene geometry and appearance along with the involved image formation model:\n\n```\n../../build/bin/AppIntrinsic3D -s=sensor.yml -i=intrinsic3d.yml\n```\n\nThe output is generated in the subfolder ```intrinsic3d/```, with the final refined 3D mesh stored as ```mesh_g0_p0.ply```. The refined camera poses and color camera intrinsics are stored as ```poses_g0_p0.txt``` and ```intrinsics_g0_p0.txt``` respectively.\nIntermediate results of coarser refinement levels are also output as\n```mesh_g*_p*.ply```, where ```_g*``` specifies the SDF grid level and ```_p*``` stand for the RGB-D pyramid level (0 is always the finest/highest resolution).\n\nThe configuration file ```intrinsic3d.yml``` allows to adjust various parameters of the joint optimization method (e.g. hyperparameters for regularization terms).\nIn addition to only exporting the mesh colors, the ```output_mesh_*``` parameters enable other visualizations such as the refined albedo (set ```output_mesh_albedo``` to 1).\n\nAs the Intrinsic3D approach is computationally very demanding w.r.t. runtime and memory, it may require up to 32GB RAM during the optimization (even for object size reconstructions).\nYou can reduce the finest grid level resolution by decreasing the parameter ```num_grid_levels``` to reduce the memory usage. ```num_grid_levels: \"3\"``` means that the input SDF volume is upsampled twice; for an initial SDF grid with voxel size 0.004, the voxel size of the final refined SDF grid is 0.001.\n\n## Contact\nIf you have any questions, please contact [Robert Maier \u0026lt;robert.maier@tum.de\u003e](mailto:robert.maier@tum.de) or [Kihwan Kim \u0026lt;kihwank@nvidia.com\u003e](mailto:kihwank@nvidia.com).\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FNVlabs%2Fintrinsic3d","html_url":"https://awesome.ecosyste.ms/projects/github.com%2FNVlabs%2Fintrinsic3d","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FNVlabs%2Fintrinsic3d/lists"}