{"id":13778489,"url":"https://github.com/autonomousvision/differentiable_volumetric_rendering","last_synced_at":"2025-04-04T11:15:24.708Z","repository":{"id":38776343,"uuid":"248476364","full_name":"autonomousvision/differentiable_volumetric_rendering","owner":"autonomousvision","description":" This repository contains the code for the CVPR 2020 paper \"Differentiable Volumetric Rendering: Learning Implicit 3D Representations without 3D Supervision\"","archived":false,"fork":false,"pushed_at":"2021-11-09T07:56:02.000Z","size":23190,"stargazers_count":815,"open_issues_count":12,"forks_count":92,"subscribers_count":30,"default_branch":"master","last_synced_at":"2025-03-28T10:09:32.223Z","etag":null,"topics":["3d-deep-learning","3d-reconstruction","cvpr-2020","cvpr2020","differentiable-rendering","dvr","implicit-representions","mesh-generation","novel-view-synthesis"],"latest_commit_sha":null,"homepage":"http://www.cvlibs.net/publications/Niemeyer2020CVPR.pdf","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/autonomousvision.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2020-03-19T10:41:02.000Z","updated_at":"2025-03-24T09:09:48.000Z","dependencies_parsed_at":"2022-07-14T04:10:32.031Z","dependency_job_id":null,"html_url":"https://github.com/autonomousvision/differentiable_volumetric_rendering","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/autonomousvision%2Fdifferentiable_volumetric_rendering","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/autonomousvision%2Fdifferentiable_volumetric_rendering/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/autonomousvision%2Fdifferentiable_volumetric_rendering/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/autonomousvision%2Fdifferentiable_volumetric_rendering/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/autonomousvision","download_url":"https://codeload.github.com/autonomousvision/differentiable_volumetric_rendering/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":247166168,"owners_count":20894654,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["3d-deep-learning","3d-reconstruction","cvpr-2020","cvpr2020","differentiable-rendering","dvr","implicit-representions","mesh-generation","novel-view-synthesis"],"created_at":"2024-08-03T18:00:54.196Z","updated_at":"2025-04-04T11:15:24.689Z","avatar_url":"https://github.com/autonomousvision.png","language":"Python","funding_links":[],"categories":["Implicit representations of Geometry and Appearance","Python"],"sub_categories":["From 2D supervision only (“inverse graphics”)"],"readme":"# Differentiable Volumetric Rendering\n#### [Paper](http://www.cvlibs.net/publications/Niemeyer2020CVPR.pdf) | [Supplementary](http://www.cvlibs.net/publications/Niemeyer2020CVPR_supplementary.pdf) | [Spotlight Video](https://www.youtube.com/watch?v=lcub1KH-mmk) | [Blog Entry](https://autonomousvision.github.io/differentiable-volumetric-rendering/) | [Presentation](https://www.youtube.com/watch?v=U_jIN3qWVEw) | [Interactive Slides](https://m-niemeyer.github.io/slides/gtc/#/) | [Project Page](https://avg.is.tuebingen.mpg.de/publications/niemeyer2020cvpr)\n\n\u003cdiv style=\"text-align: center\"\u003e\n\u003cimg src=\"media/single-view-reconstruction/ours-combined/airplane1.gif\" width=\"256\"/\u003e\n\u003cimg src=\"media/single-view-reconstruction/ours-combined/bench.gif\" width=\"256\"/\u003e\n\u003cimg src=\"media/single-view-reconstruction/ours-combined/car1.gif\" width=\"256\"/\u003e\n\u003cimg src=\"media/multi-view-reconstruction/birds/shape.gif\" width=\"256\"/\u003e\n\u003cimg src=\"media/multi-view-reconstruction/birds/normals.gif\" width=\"256\"/\u003e\n\u003cimg src=\"media/multi-view-reconstruction/birds/texture.gif\" width=\"256\"/\u003e\n\u003c/div\u003e\n\nThis repository contains the code for the paper\n[Differentiable Volumetric Rendering: Learning Implicit 3D Representations without 3D Supervision](http://www.cvlibs.net/publications/Niemeyer2020CVPR.pdf).\n\nYou can find detailed usage instructions for training your own models and using pre-trained models below.\n\nIf you find our code or paper useful, please consider citing\n\n    @inproceedings{DVR,\n        title = {Differentiable Volumetric Rendering: Learning Implicit 3D Representations without 3D Supervision},\n        author = {Niemeyer, Michael and Mescheder, Lars and Oechsle, Michael and Geiger, Andreas},\n        booktitle = {Proc. IEEE Conf. on Computer Vision and Pattern Recognition (CVPR)},\n        year = {2020}\n    }\n\n## Installation\n\nFirst you have to make sure that you have all dependencies in place.\nThe simplest way to do so, is to use [anaconda](https://www.anaconda.com/).\n\nYou can create an anaconda environment called `dvr` using\n```\nconda env create -f environment.yaml\nconda activate dvr\n```\nNext, compile the extension modules.\nYou can do this via\n```\npython setup.py build_ext --inplace\n```\n\n## Demo\n\u003cdiv style=\"text-align: center\"\u003e\n\u003cimg src=\"media/demo/choy_renderings/f32cfab76f75cbebc6ced373b157c8ba_input.jpg\" width=\"220\"/\u003e \n\u003cimg src=\"media/single-view-reconstruction/ours-rgb/car2.gif\" width=\"260\"/\u003e\n\u003cimg src=\"media/single-view-reconstruction/ours-depth/car2.gif\" width=\"260\"/\u003e\n\nYou can now test our code on the provided input images in the `demo` folder.\nTo this end, start the generation process for one of the config files in the `configs/demo` folder.\nFor example, simply run\n```\npython generate.py configs/demo/demo_combined.yaml\n```\nThis script should create a folder `out/demo/demo_combined` where the output meshes are stored.\nThe script will copy the inputs into the `generation/inputs` folder and creates the meshes in the `generation/meshes` folder.\nMoreover, the script creates a `generation/vis` folder where both inputs and outputs are copied together.\n\n## Dataset\n\n### Download Datasets\n\nTo evaluate a pre-trained model or train a new model from scratch, you have to obtain the respective dataset.\nWe use three different datasets in the DVR project: \n\u003col type=\"a\"\u003e\n\u003cli\u003eShapeNet for 2.5D supervised models (using the \u003ca href=\"https://arxiv.org/abs/1604.00449\"\u003eChoy et. al. renderings\u003c/a\u003e as input and our renderings as supervision) \u003c/li\u003e\n\u003cli\u003eShapeNet for 2D supervised models (using the \u003ca href=\"https://arxiv.org/abs/1711.07566\"\u003eKato et. al. renderings\u003c/a\u003e)\u003c/li\u003e\n\u003cli\u003eA subset of the DTU multi-view dataset \u003c/li\u003e\n\u003c/ol\u003e\n\nYou can download our preprocessed data using\n```\nbash scripts/download_data.sh\n```\nand following the instructions. The sizes of the datasets are 114GB (a), 34GB (b), and 0.5GB (c).\n\nThis script should download and unpack the data automatically into the `data` folder.\n\n### Data Convention\n\nPlease have a look at the [FAQ](https://github.com/autonomousvision/differentiable_volumetric_rendering/blob/master/FAQ.md) for details regarding the type of camera matrices we use.\n\n## Usage\nWhen you have installed all binary dependencies and obtained the preprocessed data, you are ready to run our pre-trained models and train new models from scratch.\n\n### Generation\nTo generate meshes using a trained model, use\n```\npython generate.py CONFIG.yaml\n```\nwhere you replace `CONFIG.yaml` with the correct config file.\n\nThe easiest way is to use a pre-trained model.\nYou can do this by using one of the config files which are indicated with `_pretrained.yaml`. \n\nFor example, for our 2.5D supervised single-view reconstruction model run\n```\npython generate.py configs/single_view_reconstruction/multi_view_supervision/ours_depth_pretrained.yaml\n```\nor for our multi-view reconstruction from RGB images and sparse depth maps for the birds object run\n```\npython generate.py configs/multi_view_reconstruction/birds/ours_depth_mvs_pretrained.yaml\n```\nOur script will automatically download the model checkpoints and run the generation.\nYou can find the outputs in the `out/.../pretrained` folders.\n\nPlease note that the config files  `*_pretrained.yaml` are only for generation, not for training new models: when these configs are used for training, the model will be trained from scratch, but during inference our code will still use the pre-trained model.\n\n### Generation From Your Own Single Images\n\nSimilar to our demo, you can easily generate 3D meshes from your own single images. To this end, create a folder which contains your own images (e.g. `media/my_images`). Next, you can reuse the config file `configs/demo/demo_combined.yaml` and just adjust the [data - path](https://github.com/autonomousvision/differentiable_volumetric_rendering/blob/4ba785a2bfee0cc28324fbbd96c6e5e83f6899fc/configs/demo/demo_combined.yaml#L4) and [training - out_dir](https://github.com/autonomousvision/differentiable_volumetric_rendering/blob/4ba785a2bfee0cc28324fbbd96c6e5e83f6899fc/configs/demo/demo_combined.yaml#L6) arguments to your needs. For example, you can set the config file to\n```\ninherit_from: configs/single_view_reconstruction/multi_view_supervision/ours_combined_pretrained.yaml\ndata:\n  dataset_name: images\n  path: media/my_images\ntraining:\n  out_dir:  out/my_3d_models\n```\nto generate 3D models for the images in `media/my_images`. The models will be saved to `out/my_3d_models`.\nSimilar to before, to start the generation process, run\n```\npython generate.py configs/demo/demo_combined.yaml \n```\n*Note:* You can only expect our model to provide reasonable results on data which is similar to what it was trained on (white background, single object, etc.). \n\n\n### Evaluation\nFor evaluation of the models, we provide the script `eval_meshes.py`. You can run it using\n```\npython eval_meshes.py CONFIG.yaml\n```\nThe script takes the meshes generated in the previous step and evaluates them using a standardized protocol.\nThe output will be written to `.pkl`/`.csv` files in the corresponding generation folder which can be processed using [pandas](https://pandas.pydata.org/).\n\n*Note:* We follow previous works to use \"use 1/10 times the maximal edge length of the current object’s bounding box as unit 1\" (see [Section 4 - Metrics](http://www.cvlibs.net/publications/Mescheder2019CVPR.pdf)). In practise, that means that we multiply the Chamfer-L1 metric by a factor of 10 for reporting the numbers in the paper.\n    \n    \n### Training\nFinally, to train a new network from scratch, run\n```\npython train.py CONFIG.yaml\n```\nwhere you replace `CONFIG.yaml` with the name of the configuration file you want to use.\n\nYou can monitor on \u003chttp://localhost:6006\u003e the training process using [tensorboard](https://www.tensorflow.org/guide/summaries_and_tensorboard):\n```\ncd OUTPUT_DIR\ntensorboard --logdir ./logs\n```\nwhere you replace `OUTPUT_DIR` with the respective output directory.\n\nFor available training options, please take a look at `configs/default.yaml`.\n\n# Futher Information\n\n## More Work on Implicit Representations\nIf you like the DVR project, please check out other works on implicit representions from our group:\n- [Mescheder et. al. - Occupancy Networks: Learning 3D Reconstruction in Function Space (CVPR 2019)](https://avg.is.tuebingen.mpg.de/publications/occupancy-networks)\n- [Oechsle et. al. - Texture Fields: Learning Texture Representations in Function Space (ICCV 2019)](https://avg.is.tuebingen.mpg.de/publications/oechsle2019iccv)\n- [Niemeyer et. al. - Occupancy Flow: 4D Reconstruction by Learning Particle Dynamics (ICCV 2019)](https://avg.is.tuebingen.mpg.de/publications/niemeyer2019iccv)\n- [Peng et. al. - Convolutional Occupancy Networks (ArXiv 2020)](https://arxiv.org/abs/2003.04618)\n- [Oechsle et. al. - Learning Implicit Surface Light Fields (ArXiv 2020)](https://arxiv.org/abs/2003.12406)\n\n## Other Relevant Works\nAlso check out other exciting works on inferring implicit representations without 3D supervision: \n- [Liu et. al. - Learning to Infer Implicit Surfaces without 3D Supervision (NeurIPS 2019)](https://arxiv.org/abs/1911.00767)\n- [Sitzmann et. al. - Scene Representation Networks: Continuous 3D-Structure-Aware Neural Scene Representations (NeurIPS 2019)](https://arxiv.org/abs/1906.01618)\n- [Liu. et. al. - DIST: Rendering Deep Implicit Signed Distance Function with Differentiable Sphere Tracing (CVPR 2020)](http://b1ueber2y.me/projects/DIST-Renderer)\n- [Mildenhall et. al. - NeRF: Representing Scenes as Neural Radiance Fields for View Synthesis (ArXiv 2020)](https://arxiv.org/abs/2003.08934)\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fautonomousvision%2Fdifferentiable_volumetric_rendering","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fautonomousvision%2Fdifferentiable_volumetric_rendering","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fautonomousvision%2Fdifferentiable_volumetric_rendering/lists"}