{"id":13442938,"url":"https://github.com/vLAR-group/DM-NeRF","last_synced_at":"2025-03-20T15:31:40.132Z","repository":{"id":63365560,"uuid":"524079362","full_name":"vLAR-group/DM-NeRF","owner":"vLAR-group","description":"🔥DM-NeRF in PyTorch (ICLR 2023)","archived":false,"fork":false,"pushed_at":"2022-10-11T05:41:09.000Z","size":546450,"stargazers_count":249,"open_issues_count":7,"forks_count":17,"subscribers_count":6,"default_branch":"main","last_synced_at":"2024-10-28T06:00:14.462Z","etag":null,"topics":["3d","3d-computer-vision","3d-editor","3d-reconstruction","3d-segmentation","deep-learning","iclr2023","nerfs"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"other","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/vLAR-group.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2022-08-12T12:28:44.000Z","updated_at":"2024-10-23T09:12:10.000Z","dependencies_parsed_at":"2022-11-17T17:00:54.420Z","dependency_job_id":null,"html_url":"https://github.com/vLAR-group/DM-NeRF","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/vLAR-group%2FDM-NeRF","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/vLAR-group%2FDM-NeRF/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/vLAR-group%2FDM-NeRF/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/vLAR-group%2FDM-NeRF/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/vLAR-group","download_url":"https://codeload.github.com/vLAR-group/DM-NeRF/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":244640071,"owners_count":20485978,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["3d","3d-computer-vision","3d-editor","3d-reconstruction","3d-segmentation","deep-learning","iclr2023","nerfs"],"created_at":"2024-07-31T03:01:53.565Z","updated_at":"2025-03-20T15:31:40.127Z","avatar_url":"https://github.com/vLAR-group.png","language":"Python","funding_links":[],"categories":["Python"],"sub_categories":[],"readme":"[![arXiv](https://img.shields.io/badge/arXiv-2208.07227-b31b1b.svg)](https://arxiv.org/abs/2208.07227)\n![visitors](https://visitor-badge.glitch.me/badge?page_id=vLAR-group/DM-NeRF)\n[![License CC BY-NC-SA 4.0](https://img.shields.io/badge/license-CC4.0-blue.svg)](https://creativecommons.org/licenses/by-nc-sa/4.0/legalcode)\n[![Twitter Follow](https://img.shields.io/twitter/follow/vLAR_Group?style=social)](https://twitter.com/vLAR_Group)\n\n## DM-NeRF: 3D Scene Geometry Decomposition and Manipulation from 2D Images\n[Bing Wang](https://www.cs.ox.ac.uk/people/bing.wang/), [Lu Chen](https://chenlu-china.github.io/), [Bo Yang\u003csup\u003e*\u003c/sup\u003e](https://yang7879.github.io/) \u003cbr /\u003e\n[**Paper**](https://arxiv.org/abs/2208.07227) | [**Video**](https://www.youtube.com/watch?v=iE0RwmdLIzk) | [**DM-SR**](https://www.dropbox.com/s/1k75m38vahizbp9/dmsr.zip?dl=0)\n\n\n\u003cp align=\"center\"\u003e \u003cimg src=\"/figs/architecture.png\" width=\"80%\"\u003e \u003c/p\u003e\n\nThe architecture of our proposed DM-NeRF. Given a 3D point $\\boldsymbol{p}$, we learn an object code through a series of loss functions using both 2D and 3D supervision signals.\n\n## 1. Decomposition and Reconstruction:\n\n\u003cp align=\"center\"\u003e \u003cimg src=\"/figs/mesh.gif\" width=\"70%\"\u003e\u003c/a\u003e \u003c/p\u003e\n\n## 2. Decomposition and Rendering:\n\n\u003cp align=\"center\"\u003e \u003cimg src=\"/figs/decomposition.gif\" width=\"70%\"\u003e\u003c/a\u003e \u003c/p\u003e\n\n## 3. Manipulation:\n\n\u003cp align=\"center\"\u003e \u003cimg src=\"/figs/manipulation.gif\" width=\"70%\"\u003e\u003c/a\u003e \u003c/p\u003e\n\n## 4. Installation\n\nDM-NeRF uses a Conda environment that makes it easy to install all dependencies.\n\n1. Create the `DM-NeRF` Conda environment (Python 3.7) with [miniconda](https://docs.conda.io/en/latest/miniconda.html).\n\n```bash\nconda create --name DM-NeRF python=3.7\nconda activate DM-NeRF\n```\n\n2. Install all dependencies by running:\n\n```bash\npip install -r requirements.txt\n```\n\n### 4.1 Datasets\n\nIn this paper, we consider the following three different datasets:\n\n#### (1) [DM-SR](https://www.dropbox.com/s/1k75m38vahizbp9/dmsr.zip?dl=0)\n\nTo the best of our knowledge, there is no existing 3D scene dataset suitable for quantitative evaluation of geometry manipulation. Therefore, we create a synthetic dataset with 8 types of different and complex indoor rooms, called DM-SR. The room types and designs follow [Hypersim Dataset](https://github.com/apple/ml-hypersim). Overall, we firstly render the static scenes, and then manipulate each scene followed by second round rendering. Each scene has a physical size of about 12x12x3 meters with around 8 objects.  We will keep updating [DM-SR](https://www.dropbox.com/s/1k75m38vahizbp9/dmsr.zip?dl=0) for future research in the community. \n\n#### (2) [Replica](https://www.dropbox.com/s/t1bref0zrmbq1gl/replica.zip?dl=0)\n\nIn this paper, we use 7 scenes `office0, office2, office3, office4, room0, room1, room2` from the [Replica Dataset](https://github.com/facebookresearch/Replica-Dataset). We request the authors of [Semantic-NeRF](https://github.com/Harry-Zhi/semantic_nerf) to generate color images and 2D object masks with camera poses at 640x480 pixels for each of 7 scenes. Each scene has 59~93 objects with very diverse sizes. Details of camera settings and trajectories can be found [here](https://www.dropbox.com/s/t1bref0zrmbq1gl/replica.zip?dl=0).\n\n#### (3) [ScanNet](http://www.scan-net.org/)\n\nIn this paper, we use 8 scenes `scene0010_00, scene0012_00, scene0024_00, scene0033_00, scene0038_00, scene0088_00, scene0113_00, scene0192_00` from the ScanNet Dataset.\n\n### 4.2 Training\n\nFor the training of our standard DM-NeRF , you can simply run the following command with a chosen config file specifying data directory and hyper-params.\n\n```bash\n\nCUDA_VISIBLE_DEVICES=0 python -u train_dmsr.py --config configs/dmsr/train/study.txt\n\n```\nOther working modes and set-ups can be also made via the above command by choosing different config files.\n\n\n### 4.3 Evaluation\n\nIn this paper, we use PSNR, SSIM, LPIPS for rendering evaluation, and mAPs for both decomposition and manipulation evluations.\n\n#### (1) Decomposition\n\n##### Quantitative Evaluation\n\nFor decomposition evaluation, you need choose a specific config file and then run:\n\n```bash \n\nCUDA_VISIBLE_DEVICES=0 python -u test_dmsr.py --config configs/dmsr/test/study.txt\n\n```\n##### Mesh Generation\n\nFor mesh generation, you can change the config file and then run:\n\n```bash\nCUDA_VISIBLE_DEVICES=0 python -u test_dmsr.py --config configs/dmsr/test/meshing.txt\n\n```\n\n#### (2) Manipulation\n\n##### Quantitative Evaluation\n\nWe provide the DM-SR dataset for the quantitative evaluation of geometry manipulation.\n\nSet the target object and desired manipulated settings in a sepcific config file. And then run:\n\n```bash\n\nCUDA_VISIBLE_DEVICES=0 python -u test_dmsr.py --config configs/dmsr/mani/study.txt --mani_mode translation\n\n```\n##### Qualitative Evaluation\n\nFor other qualitative evaluations, you can change the config file and then run:\n\n```bash\nCUDA_VISIBLE_DEVICES=0 python -u test_dmsr.py --config configs/dmsr/mani/demo_deform.txt\n\n```\n\n## 5. [Video (Youtube)](https://www.youtube.com/watch?v=yQtpPfM5dTA)\n\u003cp align=\"center\"\u003e \u003ca href=\"https://www.youtube.com/watch?v=iE0RwmdLIzk\"\u003e\u003cimg src=\"/figs/DM-NeRF.png\" width=\"70%\"\u003e\u003c/a\u003e \u003c/p\u003e\n\n\n### Citation\nIf you find our work useful in your research, please consider citing:\n\n      @article{wang2022dmnerf,\n      title={DM-NeRF: 3D Scene Geometry Decomposition and Manipulation from 2D Images},\n      author={Bing, Wang and Chen, Lu and Yang, Bo},\n      journal={arXiv preprint arXiv:2208.07227},\n      year={2022}\n    }\n\n### License\nLicensed under the CC BY-NC-SA 4.0 license, see [LICENSE](./LICENSE).\n\n### Updates\n* 31/8/2022: Data release！\n* 25/8/2022: Code release！\n* 15/8/2022: Initial release！\n\n## Related Repos\n1. [RangeUDF: Semantic Surface Reconstruction from 3D Point Clouds](https://github.com/vLAR-group/RangeUDF) ![GitHub stars](https://img.shields.io/github/stars/vLAR-group/RangeUDF.svg?style=flat\u0026label=Star)\n2. [GRF: Learning a General Radiance Field for 3D Representation and Rendering](https://github.com/alextrevithick/GRF) ![GitHub stars](https://img.shields.io/github/stars/alextrevithick/GRF.svg?style=flat\u0026label=Star)\n3. [3D-BoNet: Learning Object Bounding Boxes for 3D Instance Segmentation on Point Clouds](https://github.com/Yang7879/3D-BoNet) ![GitHub stars](https://img.shields.io/github/stars/Yang7879/3D-BoNet.svg?style=flat\u0026label=Star)\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FvLAR-group%2FDM-NeRF","html_url":"https://awesome.ecosyste.ms/projects/github.com%2FvLAR-group%2FDM-NeRF","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FvLAR-group%2FDM-NeRF/lists"}