{"id":13474054,"url":"https://liuyuan-pal.github.io/NeRO/","last_synced_at":"2025-03-26T20:31:54.527Z","repository":{"id":169997147,"uuid":"633824322","full_name":"liuyuan-pal/NeRO","owner":"liuyuan-pal","description":"[SIGGRAPH2023] NeRO: Neural Geometry and BRDF Reconstruction of Reflective Objects from Multiview Images","archived":false,"fork":false,"pushed_at":"2024-08-19T00:47:27.000Z","size":64768,"stargazers_count":546,"open_issues_count":15,"forks_count":36,"subscribers_count":11,"default_branch":"main","last_synced_at":"2024-10-30T07:45:54.283Z","etag":null,"topics":["3d-reconstruction","multiview-stereo","neural-fields"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/liuyuan-pal.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null}},"created_at":"2023-04-28T11:12:32.000Z","updated_at":"2024-10-29T02:36:11.000Z","dependencies_parsed_at":"2024-04-15T10:31:02.649Z","dependency_job_id":null,"html_url":"https://github.com/liuyuan-pal/NeRO","commit_stats":null,"previous_names":["liuyuan-pal/nero"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/liuyuan-pal%2FNeRO","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/liuyuan-pal%2FNeRO/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/liuyuan-pal%2FNeRO/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/liuyuan-pal%2FNeRO/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/liuyuan-pal","download_url":"https://codeload.github.com/liuyuan-pal/NeRO/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":245731605,"owners_count":20663213,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["3d-reconstruction","multiview-stereo","neural-fields"],"created_at":"2024-07-31T16:01:09.196Z","updated_at":"2025-03-26T20:31:49.518Z","avatar_url":"https://github.com/liuyuan-pal.png","language":"Python","funding_links":[],"categories":["Uncategorized"],"sub_categories":["Uncategorized"],"readme":"# NeRO\r\nNeRO: Neural Geometry and BRDF Reconstruction of Reflective Objects from Multiview Images\r\n![](assets/teaser.jpg)\r\n\r\n## [Project page](https://liuyuan-pal.github.io/NeRO/) | [Paper](https://arxiv.org/abs/2305.17398)\r\n\r\n### Update\r\n\r\n2023.07.26 Thanks @[ingra14m](https://github.com/ingra14m) for adding support for the NeRF-blender datasets (ShinyObject/NeRF-Synthetic). These codes are included in the `nerf-syn` branch https://github.com/liuyuan-pal/NeRO/tree/nerf-syn. Welcome to try it!\r\n\r\n2024.04.14 Thanks @[bell-one](https://github.com/bell-one) for adding a material extraction module. Now we can extract the materials to UV maps instead of defining them on the vertices. Check out [extract_materials_texture_map.py](extract_materials_texture_map.py) for this function!\r\n\r\n## Usage\r\n### Setup\r\n1. Install basic required packages.\r\n```shell\r\ngit clone https://github.com/liuyuan-pal/NeRO.git\r\ncd NeRO\r\npip install -r requirements.txt\r\n```\r\n2. Install `nvdiffrast`. Please follow instructions here [https://nvlabs.github.io/nvdiffrast/#installation](https://nvlabs.github.io/nvdiffrast/#installation).\r\n3. Install `raytracing`. Please follow instructions here [https://github.com/ashawkey/raytracing](https://github.com/ashawkey/raytracing).\r\n\r\n### Download datasets\r\n\r\nModels and datasets all can be found [here](https://connecthkuhk-my.sharepoint.com/:f:/g/personal/yuanly_connect_hku_hk/EvNz_o6SuE1MsXeVyB0VoQ0B9zL8NZXjQQg0KknIh6RKjQ?e=MaonKe).\r\n\r\n### Stage I: Shape reconstruction\r\n\r\n1. In the `NeRO` directory, ensure that you have the following data:\r\n```\r\nNeRO\r\n|-- data\r\n    |-- GlossyReal\r\n        |-- bear \r\n            ...\r\n    |-- GlossySynthetic\r\n        |-- bell\r\n            ...\r\n```\r\n2. Run the training script\r\n```shell\r\n# reconstructing the \"bell\" of the Glossy Synthetic dataset\r\npython run_training.py --cfg configs/shape/syn/bell.yaml\r\n\r\n# reconstructing the \"bear\" of the Glossy Real dataset\r\npython run_training.py --cfg configs/shape/real/bear.yaml\r\n```\r\nIntermediate results will be saved at `data/train_vis`. Models will be saved at `data/model`.\r\n\r\n3. Extract mesh from the model.\r\n```shell\r\npython extract_mesh.py --cfg configs/shape/syn/bell.yaml\r\npython extract_mesh.py --cfg configs/shape/real/bear.yaml\r\n```\r\nThe extracted meshes will be saved at `data/meshes`.\r\n\r\n### Stage II: Material estimation\r\n\r\n1. In the `NeRO` directory, ensure that you have the following data:\r\n```\r\nNeRO\r\n|-- data\r\n    |-- GlossyReal\r\n        |-- bear \r\n            ...\r\n    |-- GlossySynthetic\r\n        |-- bell\r\n            ...\r\n    |-- meshes\r\n        | -- bell_shape-300000.ply\r\n        | -- bear_shape-300000.ply\r\n             ...\r\n```\r\n2. Run the training script:\r\n```shell\r\n# estimate BRDF of the \"bell\" of the Glossy Synthetic dataset\r\npython run_training.py --cfg configs/material/syn/bell.yaml\r\n\r\n# estimate BRDF of the \"bear\" of the Glossy Real dataset\r\npython run_training.py --cfg configs/material/real/bear.yaml\r\n```\r\nIntermediate results will be saved at `data/train_vis`. Models will be saved at `data/model`.\r\n\r\n3. Extract materials from the model.\r\n```shell\r\npython extract_materials.py --cfg configs/material/syn/bell.yaml\r\npython extract_materials.py --cfg configs/material/real/bear.yaml\r\n```\r\nThe extracted materials will be saved at `data/materials`.\r\n\r\n### Relighting\r\n\r\n1. In the `NeRO` directory, ensure that you have the following data:\r\n```\r\nNeRO\r\n|-- data\r\n    |-- GlossyReal\r\n        |-- bear \r\n            ...\r\n    |-- GlossySynthetic\r\n        |-- bell\r\n            ...\r\n    |-- meshes\r\n        | -- bell_shape-300000.ply\r\n        | -- bear_shape-300000.ply\r\n             ...\r\n    |-- materials\r\n        | -- bell_material-100000\r\n            | -- albedo.npy\r\n            | -- metallic.npy\r\n            | -- roughness.npy\r\n        | -- bear_material-100000\r\n            | -- albedo.npy\r\n            | -- metallic.npy\r\n            | -- roughness.npy\r\n    |-- hdr\r\n        | -- neon_photostudio_4k.exr\r\n```\r\n2. Run relighting script\r\n```shell\r\npython relight.py --blender \u003cpath-to-your-blender\u003e \\\r\n                  --name bell-neon \\\r\n                  --mesh data/meshes/bell_shape-300000.ply \\\r\n                  --material data/materials/bell_material-100000 \\\r\n                  --hdr data/hdr/neon_photostudio_4k.exr \\\r\n                  --trans\r\n                  \r\npython relight.py --blender \u003cpath-to-your-blender\u003e \\\r\n                  --name bear-neon \\\r\n                  --mesh data/meshes/bear_shape-300000.ply \\\r\n                  --material data/materials/bear_material-100000 \\\r\n                  --hdr data/hdr/neon_photostudio_4k.exr\r\n```\r\nThe relighting results will be saved at `data/relight` with the directory name of `bell-neon` or `bear-neon`. This command means that we use `neon_photostudio_4k.exr` to relight the object.\r\n\r\n\r\n### Training on custom objects\r\n\r\nRefer to [custom_object.md](custom_object.md).\r\n\r\n### Evaluation\r\n\r\nRefer to [eval.md](eval.md).\r\n\r\n## Acknowledgements\r\nIn this repository, we have used codes from the following repositories. \r\nWe thank all the authors for sharing great codes.\r\n\r\n- [NeuS](https://github.com/Totoro97/NeuS)\r\n- [NvDiffRast](https://github.com/NVlabs/nvdiffrast)\r\n- [NvDiffRec](https://github.com/NVlabs/nvdiffrec)\r\n- [Ref-NeRF](https://github.com/google-research/multinerf)\r\n- [RayTracing](https://github.com/ashawkey/raytracing)\r\n- [COLMAP](https://colmap.github.io/)\r\n\r\n## Citation\r\n```\r\n@inproceedings{liu2023nero,\r\n  title={NeRO: Neural Geometry and BRDF Reconstruction of Reflective Objects from Multiview Images},\r\n  author={Liu, Yuan and Wang, Peng and Lin, Cheng and Long, Xiaoxiao and Wang, Jiepeng and Liu, Lingjie and Komura, Taku and Wang, Wenping},\r\n  booktitle={SIGGRAPH},\r\n  year={2023}\r\n}\r\n```\r\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/liuyuan-pal.github.io%2FNeRO%2F","html_url":"https://awesome.ecosyste.ms/projects/liuyuan-pal.github.io%2FNeRO%2F","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/liuyuan-pal.github.io%2FNeRO%2F/lists"}