{"id":13958475,"url":"https://github.com/snap-research/NeROIC","last_synced_at":"2025-07-21T00:30:58.016Z","repository":{"id":41238006,"uuid":"445657089","full_name":"snap-research/NeROIC","owner":"snap-research","description":null,"archived":false,"fork":false,"pushed_at":"2023-01-19T22:21:36.000Z","size":8908,"stargazers_count":1001,"open_issues_count":14,"forks_count":134,"subscribers_count":90,"default_branch":"master","last_synced_at":"2025-05-23T17:18:36.497Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"other","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/snap-research.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2022-01-07T21:28:21.000Z","updated_at":"2025-05-22T17:00:28.000Z","dependencies_parsed_at":"2023-02-11T22:01:22.171Z","dependency_job_id":null,"html_url":"https://github.com/snap-research/NeROIC","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/snap-research/NeROIC","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/snap-research%2FNeROIC","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/snap-research%2FNeROIC/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/snap-research%2FNeROIC/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/snap-research%2FNeROIC/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/snap-research","download_url":"https://codeload.github.com/snap-research/NeROIC/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/snap-research%2FNeROIC/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":266221247,"owners_count":23894964,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-08-08T13:01:37.320Z","updated_at":"2025-07-21T00:30:56.681Z","avatar_url":"https://github.com/snap-research.png","language":"Python","funding_links":[],"categories":["其他_机器视觉"],"sub_categories":["网络服务_其他"],"readme":"# NeROIC: Neural Object Capture and Rendering from Online Image Collections\n\nThis repository is the official implementation of  the NeROIC model from [NeROIC: Neural Object Capture and Rendering from Online Image Collections](https://arxiv.org/abs/2201.02533) by [Zhengfei Kuang](https://zhengfeikuang.com), [Kyle Olszewski](https://kyleolsz.github.io/), [Menglei Chai](https://mlchai.com/), [Zeng Huang](https://zeng.science/), [Panos Achlioptas](https://optas.github.io/), and [Sergey Tulyakov](http://stulyakov.com).\n\nThis work has been tested on Ubuntu 20.04. \n\nOur project page is https://zfkuang.github.io/NeROIC.\n\n## Quick Start\n* Install required libraries.\n\n```sh\nconda env create -f environment.yaml\nconda activate neroic\napt-get install graphicsmagick-imagemagick-compat\n```\n\n* Download our example data at [here](https://drive.google.com/drive/folders/1HzxaO9CcQOcUOp32xexVYFtsyKKULR7T?usp=sharing). (figure_dataset, milkbox_dataset and television_dataset)\n\n* Optimize the geometry network.\n\n```bash\npython train.py \\\n--config \u003cpath_of_the_geometry_stage_config_file\u003e \\\n--datadir \u003cpath of the data\u003e \n```\n\nfor example:\n\n```bash\npython train.py \\\n--config configs/milkbox_geometry.yaml \\\n--datadir ./data/milkbox_dataset\n```\n\n(Optional) running the script with multiple GPU is a bit tricky: you should first run it a single GPU (which will generate the resized images), then stop the process before the training starts. Then run:\n\n```bash\nCUDA_VISIBLE_DEVICES=0,1,2,3 python train.py \\\n--config \u003cpath_of_the_geometry_stage_config_file\u003e \\\n--datadir \u003cpath of the data\u003e \\\n--num_gpus 4\n```\n\n* Extract the normal from learned geometry.\n\n```bash\npython generate_normal.py \\\n --config \u003cpath_of_the_geometry_stage_config_file\u003e \\\n --ft_path \u003cpath_of_the_geometry_network_weight_file\u003e \\\n--datadir \u003cpath of the data\u003e \n```\n\nfor example: \n\n```bash\npython generate_normal.py \\\n--config configs/milkbox_geometry.yaml \\\n--ft_path logs/milkbox_geometry/epoch=29.ckpt \\\n--datadir data/milkbox_dataset\n```\n\n* Optimize the rendering network.\n\n```bash\npython train.py \\\n--config \u003cpath_of_the_rendering_stage_config_file\u003e \\\n--ft_path \u003cpath_of_the_geometry_network_weight_file\u003e \\\n--datadir \u003cpath of the data\u003e \n```\nfor example (training with 4 GPUs): \n\n```bash\nCUDA_VISIBLE_DEVICES=0,1,2,3 python train.py \\\n--config configs/milkbox_rendering.yaml \\\n--ft_path logs/milkbox_geometry/epoch=29.ckpt \\\n--datadir data/milkbox_dataset \\\n--num_gpus 4\n```\n\ntraining logs \u0026 results will be saved at `logs/`.\n\n## Video Generation\nthe video of novel testing views is generated on-the-fly during training, and saved under `logs/`. To render video with a pre-trained model, run:\n\n```bash\npython train.py \\\n--config \u003cpath_of_the_config_file\u003e \\\n--ft_path \u003cpath_of_the_model_weight_file\u003e \\\n--datadir \u003cpath of the data\u003e \\\n--i_video 1 \\ \n(optional) --test_img_id \u003cindex_of_the_reference_image\u003e\n```\n\n## Testing\nWe provide additional testing scripts for material decomposition and relighting. \n\nTo decompose materials:\n\n```bash\npython test_material.py \\\n--config \u003cpath_of_the_rendering_stage_config_file\u003e \\\n--ft_path \u003cpath_of_the_rendering_network_weight_file\u003e \\\n--datadir \u003cpath of the data\u003e \n```\n\nTo relight the model (panoramic exr HDRI maps are prefered):\n\n```bash\npython test_relighting.py \\\n--config \u003cpath_of_the_rendering_stage_config_file\u003e \\\n--ft_path \u003cpath_of_the_rendering_network_weight_file\u003e \\\n--datadir \u003cpath of the data\u003e \\\n--test_env_filename \u003cpath_of_the_envrionment_map\u003e \n```\n\n## Bring Your Own Data\n\nTo train NeROIC with your own collected data, click [here](scripts/README.md).\n\n## Model Overview\n\nOur two-stage model takes images of an object from different conditions as input. \nWith the camera poses of images and object foreground masks acquired by other state-of-the-art methods, \nWe first optimize the geometry of scanned object and refine camera poses by training a NeRF-based network; \nWe then compute the surface normal from the geometry (represented by density function) using our normal extraction layer;\nFinally, our second stage model decomposes the material properties of the object and solves for the lighting conditions for each image. \n\n![Screenshot](assets/resources/framework.png)\n\n## Novel View Synthesis\n\nGiven online images from a common object, our model can synthesize novel views of the object with the lighting conditions from the training images.\n\nhttps://user-images.githubusercontent.com/8952528/148708746-14c2db49-2516-4e31-a1be-559be4480b01.mp4\n\n![Screenshot](assets/resources/nvs.png)\n\n## Material Decomposition\n\nhttps://user-images.githubusercontent.com/8952528/148708751-7fb1c820-57d3-454f-a411-635c141fab18.mp4\n\n![Screenshot](assets/resources/material.png)\n\n## Relighting\n\nhttps://user-images.githubusercontent.com/8952528/148708757-f7c981f0-3963-49cb-a492-4fae73429105.mp4\n\n## Citation\n\nIf you find this useful, please cite the following:\n```bibtex\n@article{10.1145/3528223.3530177,\nauthor = {Kuang, Zhengfei and Olszewski, Kyle and Chai, Menglei and Huang, Zeng and Achlioptas, Panos and Tulyakov, Sergey},\ntitle = {NeROIC: Neural Rendering of Objects from Online Image Collections},\nyear = {2022},\nissue_date = {July 2022},\npublisher = {Association for Computing Machinery},\naddress = {New York, NY, USA},\nvolume = {41},\nnumber = {4},\nissn = {0730-0301},\nurl = {https://doi.org/10.1145/3528223.3530177},\ndoi = {10.1145/3528223.3530177},\njournal = {ACM Trans. Graph.},\nmonth = {jul},\narticleno = {56},\nnumpages = {12},\nkeywords = {neural rendering, reflectance \u0026 shading models, multi-view \u0026 3D}\n}\n```\n\nAcknowledgements: This work borrows many code from [NeRF-pl](https://github.com/kwea123/nerf_pl). We thank the author for releasing his code.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsnap-research%2FNeROIC","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fsnap-research%2FNeROIC","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsnap-research%2FNeROIC/lists"}