{"id":15033539,"url":"https://github.com/nvlabs/nvdiffrec","last_synced_at":"2025-05-15T13:05:27.694Z","repository":{"id":37452100,"uuid":"431420791","full_name":"NVlabs/nvdiffrec","owner":"NVlabs","description":"Official code for the CVPR 2022 (oral) paper \"Extracting Triangular 3D Models, Materials, and Lighting From Images\".","archived":false,"fork":false,"pushed_at":"2024-05-02T11:08:55.000Z","size":159500,"stargazers_count":2210,"open_issues_count":67,"forks_count":229,"subscribers_count":53,"default_branch":"main","last_synced_at":"2025-05-15T13:05:26.566Z","etag":null,"topics":["deep-learning","pytorch"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"other","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/NVlabs.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE.txt","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2021-11-24T09:25:05.000Z","updated_at":"2025-05-15T06:33:38.000Z","dependencies_parsed_at":"2023-02-12T15:17:07.972Z","dependency_job_id":"4e5ebdb3-78b0-4767-b7e3-c9023c438ba2","html_url":"https://github.com/NVlabs/nvdiffrec","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/NVlabs%2Fnvdiffrec","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/NVlabs%2Fnvdiffrec/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/NVlabs%2Fnvdiffrec/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/NVlabs%2Fnvdiffrec/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/NVlabs","download_url":"https://codeload.github.com/NVlabs/nvdiffrec/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":254346624,"owners_count":22055808,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["deep-learning","pytorch"],"created_at":"2024-09-24T20:21:37.232Z","updated_at":"2025-05-15T13:05:27.670Z","avatar_url":"https://github.com/NVlabs.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# nvdiffrec\n\n![Teaser image](https://nvlabs.github.io/nvdiffrec/assets/system.JPG \"Teaser image\")\n\nJoint optimization of topology, materials and lighting from multi-view image observations \nas described in the paper\n[Extracting Triangular 3D Models, Materials, and Lighting From Images](https://nvlabs.github.io/nvdiffrec/).\n\nFor differentiable marching tetrahedons, we have adapted code from NVIDIA's [Kaolin: A Pytorch Library for Accelerating 3D Deep Learning Research](https://github.com/NVIDIAGameWorks/kaolin).\n\n# News\n\n- **2023-10-20** : We added a version of the renderutils library written in [slangpy](https://shader-slang.com/slang/user-guide/a1-02-slangpy.html) to leverage the autodiff capabilities of slang instead of CUDA extensions with manually crafted forward and backward passes. This simplifies the code substantially, with the same runtime performance as before. This version is available in the `slang` [branch](https://github.com/NVlabs/nvdiffrec/tree/slang) of this repo.\n\n- **2023-09-15** : We added support for the [FlexiCubes](https://research.nvidia.com/labs/toronto-ai/flexicubes/) isosurfacing technique. Please see the config `configs/bob_flexi.json` for a usage example, and refer to the [FlexiCubes documentation](https://github.com/nv-tlabs/FlexiCubes) for details.\n\n# Citation\n\n```\n@inproceedings{Munkberg_2022_CVPR,\n    author    = {Munkberg, Jacob and Hasselgren, Jon and Shen, Tianchang and Gao, Jun and Chen, Wenzheng \n                    and Evans, Alex and M\\\"uller, Thomas and Fidler, Sanja},\n    title     = \"{Extracting Triangular 3D Models, Materials, and Lighting From Images}\",\n    booktitle = {Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)},\n    month     = {June},\n    year      = {2022},\n    pages     = {8280-8290}\n}\n```\n\n# Licenses\n\nCopyright \u0026copy; 2022, NVIDIA Corporation. All rights reserved.\n\nThis work is made available under the [Nvidia Source Code License](https://github.com/NVlabs/nvdiffrec/blob/main/LICENSE.txt).\n\nFor business inquiries, please visit our website and submit the form: [NVIDIA Research Licensing](https://www.nvidia.com/en-us/research/inquiries/).\n\n# Installation\n\nRequires Python 3.6+, VS2019+, Cuda 11.3+ and PyTorch 1.10+\n\nTested in Anaconda3 with Python 3.9 and PyTorch 1.10\n\n## One time setup (Windows)\n\nInstall the [Cuda toolkit](https://developer.nvidia.com/cuda-toolkit) (required to build the PyTorch extensions).\nWe support Cuda 11.3 and above.\nPick the appropriate version of PyTorch compatible with the installed Cuda toolkit.\nBelow is an example with Cuda 11.6\n\n```\nconda create -n dmodel python=3.9\nactivate dmodel\nconda install pytorch torchvision torchaudio cudatoolkit=11.6 -c pytorch -c conda-forge\npip install ninja imageio PyOpenGL glfw xatlas gdown\npip install git+https://github.com/NVlabs/nvdiffrast/\npip install --global-option=\"--no-networks\" git+https://github.com/NVlabs/tiny-cuda-nn#subdirectory=bindings/torch\nimageio_download_bin freeimage\n```\n\n### Every new command prompt\n`activate dmodel`\n\n# Examples\n\n*Our approach is designed for high-end NVIDIA GPUs with large amounts of memory.\nTo run on mid-range GPU's, reduce the batch size parameter in the .json files.*\n\nSimple genus 1 reconstruction example:\n```\npython train.py --config configs/bob.json\n```\nVisualize training progress (only supported on Windows):\n```\npython train.py --config configs/bob.json --display-interval 20\n```\n\nMulti GPU example (Linux only. *Experimental: all results in the paper were generated using a single GPU*),\nusing [PyTorch DDP](https://pytorch.org/docs/stable/elastic/run.html#launcher-api)\n```\ntorchrun --nproc_per_node=4 train.py --config configs/bob.json\n```\n\nBelow, we show the starting point and the final result. References to the right.\n\n![Initial guess](images/start_of_training.jpg \"Intial guess\")\n![Our result](images/end_of_training.jpg \"Our result\")\n\nThe results will be stored in the `out` folder.\nThe [Spot](http://www.cs.cmu.edu/~kmcrane/Projects/ModelRepository/index.html#spot) and\n[Bob](https://www.cs.cmu.edu/~kmcrane/Projects/ModelRepository/index.html) models were\ncreated and released into the public domain by [Keenan Crane](http://www.cs.cmu.edu/~kmcrane/index.html).\n\nIncluded examples\n\n- `spot.json` - Extracting a 3D model of the spot model. Geometry, materials, and lighting from image observations.\n- `spot_fixlight.json` - Same as above but assuming known environment lighting.\n- `spot_metal.json` - Example of joint learning of materials and high frequency environment lighting to showcase split-sum.\n- `bob.json` - Simple example of a genus 1 model.\n\n# Datasets\n\nWe additionally include configs (`nerf_*.json`, `nerd_*.json`) to reproduce the main results of the paper. We rely on third party datasets, which\nare courtesy of their respective authors. Please note\nthat individual licenses apply to each dataset. To automatically download and pre-process all datasets, run the \n`download_datasets.py` script:\n```\nactivate dmodel\ncd data\npython download_datasets.py\n```\nBelow follows more information and instructions on how to manually install the datasets (in case the automated script fails).\n\n**NeRF synthetic dataset** Our view interpolation results use the synthetic dataset from the original [NeRF](https://github.com/bmild/nerf) paper.\nTo manually install it, download the [NeRF synthetic dataset archive](https://drive.google.com/uc?export=download\u0026id=18JxhpWD-4ZmuFKLzKlAw-w5PpzZxXOcG)\nand unzip it into the `nvdiffrec/data` folder. This is required for running any of the `nerf_*.json` configs.\n\n**NeRD dataset** We use datasets from the [NeRD](https://markboss.me/publication/2021-nerd/) paper, which features real-world photogrammetry and inaccurate\n(manually annotated) segmentation masks. Clone the NeRD datasets using git and rescale them to 512 x 512 pixels resolution using the script\n`scale_images.py`. This is required for running any of the `nerd_*.json` configs.\n```\nactivate dmodel\ncd nvdiffrec/data/nerd\ngit clone https://github.com/vork/ethiopianHead.git\ngit clone https://github.com/vork/moldGoldCape.git\npython scale_images.py\n```\n\n# Server usage (through Docker)\n\n- Build docker image.\n```\ncd docker\n./make_image.sh nvdiffrec:v1\n```\n\n- Start an interactive docker container:\n`docker run --gpus device=0 -it --rm -v /raid:/raid -it nvdiffrec:v1 bash`\n\n- Detached docker:\n`docker run --gpus device=1 -d -v /raid:/raid -w=[path to the code] nvdiffrec:v1 python train.py --config configs/bob.json`\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fnvlabs%2Fnvdiffrec","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fnvlabs%2Fnvdiffrec","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fnvlabs%2Fnvdiffrec/lists"}