{"id":18614404,"url":"https://github.com/ywq/psnerf","last_synced_at":"2025-04-11T00:30:42.183Z","repository":{"id":49761910,"uuid":"517896951","full_name":"ywq/psnerf","owner":"ywq","description":"PS-NeRF: Neural Inverse Rendering for Multi-view Photometric Stereo (ECCV 2022)","archived":false,"fork":false,"pushed_at":"2022-10-04T13:47:01.000Z","size":6118,"stargazers_count":136,"open_issues_count":9,"forks_count":11,"subscribers_count":10,"default_branch":"main","last_synced_at":"2024-11-07T03:31:24.476Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/ywq.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2022-07-26T03:16:52.000Z","updated_at":"2024-11-01T07:15:35.000Z","dependencies_parsed_at":"2022-08-28T17:22:52.514Z","dependency_job_id":null,"html_url":"https://github.com/ywq/psnerf","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ywq%2Fpsnerf","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ywq%2Fpsnerf/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ywq%2Fpsnerf/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ywq%2Fpsnerf/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/ywq","download_url":"https://codeload.github.com/ywq/psnerf/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248322216,"owners_count":21084333,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-07T03:25:56.105Z","updated_at":"2025-04-11T00:30:39.020Z","avatar_url":"https://github.com/ywq.png","language":"Python","funding_links":[],"categories":["Papers"],"sub_categories":["NeRF Related Tasks"],"readme":"# PS-NeRF: Neural Inverse Rendering for Multi-view Photometric Stereo (ECCV 2022)\n\n### [Project Page](https://ywq.github.io/psnerf/) | [Paper](http://arxiv.org/abs/2207.11406) \n\n\u003cp align=\"center\"\u003e\n    \u003cimg src='docs/demo.gif' width=\"800\"\u003e\n\u003c/p\u003e\n\n\nThis repository contains the implementation of the paper:\n\n**PS-NeRF: Neural Inverse Rendering for Multi-view Photometric Stereo**  \n[Wenqi Yang](https://ywq.github.io/),\n[Guanying Chen](https://guanyingc.github.io/), \n[Chaofeng Chen](http://chaofengc.github.io/),\n[Zhenfang Chen](https://zfchenunique.github.io/),\n[Kwan-Yee K. Wong](http://i.cs.hku.hk/~kykwong/)  \nEuropean Conference on Computer Vision (ECCV), 2022\n\n## Abstract\nTraditional multi-view photometric stereo (MVPS) methods are often composed of multiple disjoint stages, resulting in noticeable accumulated errors. In this paper, we present a neural inverse rendering method for MVPS based on implicit representation. Given multi-view images of a non-Lambertian object illuminated by multiple unknown directional lights, our method jointly estimates the geometry, materials, and lights. Our method first employs multi-light images to estimate per-view surface normal maps, which are used to regularize the normals derived from the neural radiance field. It then jointly optimizes the surface normals, spatially-varying BRDFs, and lights based on a shadow-aware differentiable rendering layer. After optimization, the reconstructed object can be used for novel-view rendering, relighting, and material editing. Experiments on both synthetic and real datasets demonstrate that our method achieves far more accurate shape reconstruction than existing MVPS and neural rendering methods. Our code and model will be made publicly available.\n\n\u003cp align=\"center\"\u003e\n    \u003cimg src='docs/intro.png' width=\"800\"\u003e\n\u003c/p\u003e\n\n## Setup\nOur work is implemented in PyTorch and tested with Ubuntu 18.04/20.04.\n- Python 3.9\n- PyTorch 1.8.0\n\nCreate an anaconda environment called `psnerf` using\n```\nconda env create -f environment.yaml\nconda activate psnerf\n```\n\n\n## Download\n\nDownload the dataset and trained models using\n```\nsh download.sh\n```\nIf the above command is not working, please manually download the files:\n- Released models of SDPS-Net from Google Drive ([LCNet and NENet](https://drive.google.com/drive/folders/1HCW9YDfsFoPxda3GDTjj3L5yuQUXw8a3?usp=sharing)) and put them in `preprocessing/data/models/`;\n- Trained models of our method from OneDrive ([data.tgz](https://connecthkuhk-my.sharepoint.com/:f:/g/personal/wenqi_connect_hku_hk/Et3Qh3fV3H9Lk71Bln0YEQABBzu8eBUfTp44dcqtaz18sQ?e=WUCR12)) and put the extracted folder `data/` under the project path;\n- Preprocessed dataset of both synthetic and real objects from OneDrive ([dataset.tgz](https://connecthkuhk-my.sharepoint.com/:f:/g/personal/wenqi_connect_hku_hk/Et3Qh3fV3H9Lk71Bln0YEQABBzu8eBUfTp44dcqtaz18sQ?e=WUCR12)) and put the extracted folder `dataset/` under the project path;\n- Environment maps for relighting application from OneDrive ([envmap.tgz](https://connecthkuhk-my.sharepoint.com/:f:/g/personal/wenqi_connect_hku_hk/Et3Qh3fV3H9Lk71Bln0YEQABBzu8eBUfTp44dcqtaz18sQ?e=WUCR12)) and put the extracted folder `envmap/` under `stage2/`.\n\n### Dataset\nThe real dataset was processed from [DiLiGenT-MV Dataset](https://sites.google.com/site/photometricstereodata/mv?authuser=0), which contains 5 objects `BEAR`, `BUDDHA`, `COW`, `POT2`, `READING`.\n\nThe synthetic dataset was rendered using [Mitsuba](https://www.mitsuba-renderer.org/), which contains 2 objects `BUNNY`, `ARMADILLO`.\n\nAfter downloaded and extracted, you can find the processed datasets in `./dataset` folder.\n### Model\nWe release the pretrained models of the 5 real scenes. After downloaded and extracted, you can find them in `./data` folder.\n\n### Objects\nAdditionally, if you want to evaluate the mesh, you may run the following command or manually download the rescaled ground-truth objects from OneDrive ([objects.tgz](https://connecthkuhk-my.sharepoint.com/:f:/g/personal/wenqi_connect_hku_hk/Et3Qh3fV3H9Lk71Bln0YEQABBzu8eBUfTp44dcqtaz18sQ?e=WUCR12)).\n```bash\n# Download objects\nwget http://www.visionlab.cs.hku.hk/data/psnerf/objects.tgz\ntar -xzvf dataset.tgz\nrm objects.tgz\n```\nThe file `trans.json` stores the transformation parameters. You may also get the rescaled objects from [DiLiGenT-MV Dataset](https://sites.google.com/site/photometricstereodata/mv?authuser=0) by `vertices_new = (vertices - center) / scale`. (Objects in `objects.tgz` are already rescaled.)\n\n\n\n##  Test \u0026 Evaluation\nAfter download the pretrained models, you may run the following code to test and evalauate the results. (You may specify the `OBJ_NAME` from  `bear`, `buddha`, `cow`, `pot2`, `reading`.) \n```bash\n## Please replace the `GPU_ID` and `OBJ_NAME` with your choices.\ncd stage2\npython eval.py --gpu GPU_ID --obj_name OBJ_NAME --expname test_1 --exps_folder ../data/stage2\ncd ..\n\npython evaluation.py --obj OBJ_NAME --expname test_1 --test_out_dir stage2/test_out\n```\nFor testing and evaluating on your own trained models:\n```bash\n## Please replace the `GPU_ID`, `OBJ_NAME` and `EXPNAME` with your choices.\ncd stage2\npython eval.py --gpu GPU_ID --obj_name OBJ_NAME --expname EXPNAME\ncd ..\n## other optional arguments\n# --exps_folder EXP_FOLDER        # specify exp_folder (default: ./out)\n# --test_out_dir TEST_OUT_DIR     # test_out_dir (default: ./test_out)\n# --save_npy                      # save npy files\n# --timestamp TIMESTAMP           # specify the timestamp (default: latest)\n# --checkpoint CHECKPOINT         # specify the checkpoint (default: latest)\n# --light_batch N_LIGHT           # modify light batch according to your GPU memory (default: 64)\n\n## Please replace the `OBJ_NAME`, `EXPNAME` and `TEST_OUT_DIR` with your choices.\npython evaluation.py --obj OBJ_NAME --expname EXPNAME --test_out_dir TEST_OUT_DIR\n```\n\n\n\n##  Train\nTo train a model from scratch, you need to first prepare the training data from [Preprocessing](./preprocessing) with pretrained [SDPS-Net](https://github.com/guanyingc/SDPS-Net). After that you will obtain the coarse normal/light direction/light intensity estimations from the dataset.  \nThen you need to calculate the light-averaged images (see below) for training [Stage I](./stage1). For convenience, we extract the surface/normal for each view and visibility for each light direction to accelerate the [Stage II](./stage2) training process.  \nAfter [Stage II](./stage2) training phase is finished, you may test and evaluate the final results. Please go to the subpages and follow the detailed instructions there.\n- Prepare light-averaged images for stage I\n    ```bash\n    ## replace `OBJ_NAME` with your choice.\n    python light_avg.py --obj OBJ_NAME --path dataset\n    ## other optional arguments\n    # --train_light          # specify the train_light\n    # --light_intnorm        # enable it if use GT/SDPS-Net light intensity to normalize the images\n    # --sdps                 # enable it if use light intensity predicted by SDPS-Net (you should also enable `--light_intnorm`)\n    ```\n\n- The overall workflow is as follows:\n    ```bash\n    cd preprocessing\n    python test.py xxxx\n    cd ..\n    python light_avg.py xxxx\n    cd stage1\n    python train.py xxxx\n    python shape_extract.py xxxx\n    cd ../stage2\n    python train.py xxxx\n    python eval.py xxxx\n    cd ..\n    python evaluation.py xxxx\n    ```\n\n## Relighting and Material Editing\nOur method jointly estimates surface normals, spatially-varying BRDFs, and lights. After optimization, the reconstructed objects can be used for novel-view rendering, relighting, and material editing. You may render under environmental lighting and edit materials with the trained models for applications. Please follow the instructions in [Stage II](./stage2#relighting-and-material-editing).\n\n\n## Prepare your own data\nWe provide preprocessed datasets, if you want to try with your own data, please prepare your dataset and parameters as follows.\n- Data Structure\n```\n└── OBJ_NAME\n    ├── params.json\n    ├── img\n    │   ├── view_01\n    │   │   ├── 001.png\n    │   │   └── ...\n    │   └── ...\n    │       ├── 001.png\n    │       └── ...\n    ├── mask\n    │   ├── view_01.png\n    │   └── ...\n    ├── norm_mask\n    │   ├── view_01.png\n    │   └── ...\n    ├── normal (optional)\n    │   ├── img\n    │   │   ├── view_01.png\n    │   │   └── ...\n    │   └── npy\n    │       ├── view_01.png\n    │       └── ...\n    └── visibility (optional)\n        ├── view_01\n        │   ├── 001.png\n        │   └── ...\n        └── ...\n            ├── 001.png\n            └── ...\n```\n- Parameters in `params.json`\n```\n== PARAM ==             == HELP ==\nobj_name                name of the object\nn_view                  total number of views (train + test)            \nimhw                    resolution of images as [H,W]\ngt_normal_world         whether normal is in world coordinate system\nview_train              index of training views         \nview_test               index of testing views         \nK                       K matrix \npose_c2w                camera-to-world transformation matrix of all views         \nlight_is_same           whether lights are the same among all views             \nlight_direction         list of light directions for each view as [L1*3, L2*3, ...], (if `light_is_same`, only one view is provided as L*3)         \nlight_intensity         (optional) list of light intensity for each view as [L1*3, L2*3, ...], (if `light_is_same`, only one view is provided as L*3)         \nview_slt_N              (optional) index of selected N views for training         \nlight_slt_N             (optional) index of selected N lights for training\n```\n\n\n## Citation\nIf you find this code or the provided models useful in your research, please consider cite: \n```bibtex\n@inproceedings{yang2022psnerf,\n    title={PS-NeRF: Neural Inverse Rendering for Multi-view Photometric Stereo},\n    author={Yang, Wenqi and Chen, Guanying and Chen, Chaofeng and \n            Chen, Zhenfang and Wong, Kwan-Yee K.},\n    booktitle={European Conference on Computer Vision (ECCV)},\n    year={2022}\n}\n```\n\n\n\n## Acknowledgement\nPart of our code is based on the awesome [SDPS-Net](https://github.com/guanyingc/SDPS-Ne), [UNISURF](https://github.com/autonomousvision/unisurf), and [PhySG](https://github.com/Kai-46/PhySG).\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fywq%2Fpsnerf","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fywq%2Fpsnerf","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fywq%2Fpsnerf/lists"}