{"id":34716145,"url":"https://github.com/cyberagentailab/mvscps","last_synced_at":"2026-05-21T21:38:12.805Z","repository":{"id":318776618,"uuid":"1067752444","full_name":"CyberAgentAILab/MVSCPS","owner":"CyberAgentAILab","description":"[ICCV 2025] Neural Multi-View Self-Calibrated Photometric Stereo without Photometric Stereo Cues","archived":false,"fork":false,"pushed_at":"2025-10-15T00:52:00.000Z","size":3577,"stargazers_count":1,"open_issues_count":0,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2025-10-15T05:51:20.216Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"other","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/CyberAgentAILab.png","metadata":{"files":{"readme":"readme.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2025-10-01T10:38:15.000Z","updated_at":"2025-10-15T00:52:04.000Z","dependencies_parsed_at":"2025-10-15T16:08:46.407Z","dependency_job_id":"a24dd168-389c-4c7d-aa6b-e83fe17607f1","html_url":"https://github.com/CyberAgentAILab/MVSCPS","commit_stats":null,"previous_names":["cyberagentailab/mvscps"],"tags_count":null,"template":false,"template_full_name":null,"purl":"pkg:github/CyberAgentAILab/MVSCPS","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/CyberAgentAILab%2FMVSCPS","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/CyberAgentAILab%2FMVSCPS/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/CyberAgentAILab%2FMVSCPS/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/CyberAgentAILab%2FMVSCPS/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/CyberAgentAILab","download_url":"https://codeload.github.com/CyberAgentAILab/MVSCPS/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/CyberAgentAILab%2FMVSCPS/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":33315945,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-21T12:23:38.849Z","status":"ssl_error","status_checked_at":"2026-05-21T12:22:11.673Z","response_time":62,"last_error":"SSL_read: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2025-12-25T00:56:50.635Z","updated_at":"2026-05-21T21:38:12.800Z","avatar_url":"https://github.com/CyberAgentAILab.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"\u003ch2 align=\"center\"\u003eNeural Multi-View Self-Calibrated Photometric Stereo without Photometric Stereo Cues\u003c/h2\u003e\n\u003ch4 align=\"center\"\u003e\n    \u003ca href=\"https://xucao-42.github.io/homepage/\"\u003e\u003cstrong\u003eXu Cao\u003c/strong\u003e\u003c/a\u003e\n    ·\n    \u003ca href=\"https://taketomitakafumi.sakura.ne.jp/web/en/\"\u003e\u003cstrong\u003eTakafumi Taketomi\u003c/strong\u003e\u003c/a\u003e\n\u003c/h3\u003e\n\u003ch4 align=\"center\"\u003eICCV 2025 \u003c/h3\u003e\n\u003cp align=\"center\"\u003e\n  \u003cbr\u003e\n    \u003ca href=\"https://www.arxiv.org/abs/2507.23162\"\u003e\n      \u003cimg src='https://img.shields.io/badge/arXiv-Paper-981E32?style=for-the-badge\u0026Color=B31B1B' alt='arXiv PDF'\u003e\n    \u003c/a\u003e\n    \u003ca href='https://huggingface.co/datasets/cyberagent/mvscps'\u003e\n      \u003cimg src='https://img.shields.io/badge/HuggingFace-Dataset-5468FF?style=for-the-badge' alt='Dataset'\u003e\u003c/a\u003e\n    \u003ca href=\"https://www.youtube.com/watch?v=vivhD-jspmE\"\u003e\n      \u003cimg src='https://img.shields.io/badge/Youtube-Video-981E32?style=for-the-badge\u0026Color=B31B1B' alt='video'\u003e\n    \u003c/a\u003e\n\n## Overview\n\u003cdiv align=\"center\"\u003e\n\u003cimg src=\"./media/teaser.png\" alt=\"Teaser\" width=\"100%\"\u003e\n\u003c/div\u003e\n\nMVSCPS jointly recovers geometry, reflectance, and lighting from multi-view one-light-at-a-time (OLAT) images, featuring:\n- No light calibration required\n- Single-stage, end-to-end optimization; thereby no intermediate photometric stereo step\n- Flexible camera-light configurations. In the extreme case, the camera and light source can move independently for each shot.\n\n## Getting Started\n\n### Environment Setup\nInstallation (Linux, `uv`, CUDA 11.8)\n```\nchmod +x install_env.sh\n./install_env.sh\n```\nWhat this installs\n- A Python **3.9** virtual environment: `.venv-mvscps`\n- If `uv` is not detected, the script automatically installs it.\n- **PyTorch 2.5.1**, **torchvision 0.20.1** (CUDA **11.8** wheels)\n- Source-built extensions: **tiny-cuda-nn** (Torch bindings) and **nerfacc**\n- Training/config stack: **pytorch-lightning 1.9.5**, **hydra-core**, **omegaconf**\n- Visualization \u0026 scientific libs: **matplotlib**, **pyvista**, **open3d**, **opencv-python**, **imageio[ffmpeg]**, **scipy**, **scikit-image**, **trimesh**, **lpips**, **tensorboard**, **wandb**, **huggingface_hub**, etc.\n\n\nAfter successfully setting up the environment, you should see the following output:\n```\nPython: 3.9.23\n[OK] Torch 2.5.1+cu118 | CUDA: True\n[OK] VTK 9.2.6 | PyVista 0.37.0\n[OK] PyVista import: OK\n[OK] Trimesh ray engine: trimesh.ray.ray_pyembree.RayMeshIntersector\n[OK] nerfacc 0.3.3 | path: /MVSCPS/.venv-mvscps/lib/python3.9/site-packages/nerfacc\n[OK] nerfacc CUDA extension import: nerfacc.cuda\n[OK] tinycudann forward OK on cuda: output shape (128, 16)\n```\n\nIf you are using Docker, including the following lines in your Dockerfile should be sufficient:\n```Dockerfile\nFROM --platform=linux/amd64 docker.io/nvidia/cuda:11.8.0-cudnn8-devel-ubuntu22.04\n\n# Install dependencies\nRUN apt-get update \u0026\u0026 apt-get -y install python3-pip vim cmake openssh-server build-essential htop nvtop git wget curl unzip zip bash-completion sudo libgl1-mesa-glx xvfb rsync tmux libglib2.0-0 libbz2-dev ffmpeg libsm6 libxext6 \u0026\u0026 \\\n    ln -s /usr/bin/python3 /usr/bin/python\n```\n### Data Preparation\n\nDownload and preprocess [DiLiGenT-MV](https://sites.google.com/site/photometricstereodata/mv) dataset (~7GB):\n\n```\n. data/prepare_data_diligentmv.sh\n```\nThis script downloads the DiLiGenT-MV dataset in `data/DiLiGenT-MV_origin`, reorganize the file structures in `data/DiLiGenT-MV`, and calculate the scene normalization parameters for training.\n\n\nDownload and preprocess our self-collected dataset (~60GB):\n\n```\n. data/prepare_data_mvscps.sh\n```\nThis script downloads our self-collected data from [HuggingFace Dataset](https://huggingface.co/datasets/cyberagent/mvscps) and calculate the scene normalization parameters for training.\n\n\n### Training\n```\n. ./launch_diligentmv.sh\n```\nThis script trains MVSCPS on all 5 scenes in DiLiGenT-MV dataset sequentially. \nOLAT images captured from 18 views under 32 lights per view are used for training for each scene. \nThe training takes about 15 minutes per scene on a single NVIDIA A100 GPU.\nTesting, BRDF map rendering, and relighting are performed after training, and takes about 20~30 minutes per scene.\nThe trained models and results are saved in `exp/diligentmv`.\n\nEmpirical evidences that the training is converging:\n- `test/mae_light` drops below 3 degrees within 800 iterations.\n- `test/mae_normal` drops below 10 degrees within 3000 iterations.\n- `train/inv_s` increases to above 2000 within 10000 iterations.\n\nFor training on our self-collected dataset, run the following command:\n```\n. ./launch_mvscps.sh\n```\nThis script trains MVSCPS on all 6 scenes in our self-collected dataset sequentially.\nThe training takes about 80 minutes per scene on a single NVIDIA A100 GPU.\nThe trained models and results are saved in `exp/mvscps`.\n\n### Tips\n- **Coordinate system.** We follow the OpenCV convention: x → right, y → down, z → forward.\n- **Controlling train/val/test subsets.** We use a plain-text index file to specify which image subsets are used for training/validation/testing. Sample files are provided under `configs/view_light_indices`. You can prepare your own file and set `dataset.train.view_light_index_fname` and `dataset.train.view_light_index_file` in the config file.\n- **Using your own data.** Please refer to the folder structure in our [HuggingFace Dataset](https://huggingface.co/datasets/cyberagent/mvscps) and the preprocessing script `data/preprocess_data_mvscps.py`. After your data is prepared, implement your custom image loader in `dataloader/load_fn`, then configure `dataset.img_load_fn`, `dataset.img_ext`, and `dataset.img_dirname` in `configs/conf/mvscps.yaml`. This config file should be fine for your custom data.\n- **RAW size mismatch \u0026 cropping.** When loading RAW images with [RawPy](https://pypi.org/project/rawpy/), the resulting image can be slightly larger than the size recorded in EXIF. For this reason our loader applies a small crop ([code reference](https://github.com/CyberAgentAILab/MVSCPS/blob/91f847795adcb8ab8400c77583d3663716bd937d/dataloader/load_fn.py#L37)). Note that the required crop region varies by camera vendor (we confirmed differences between Sony and Canon). A practical way to determine the correct crop for your camera is to compare the RAW-rendered image with its in-camera JPEG (e.g., visualize their difference) and adjust the crop until edge discrepancies disappear.\n## Acknowledgements\nWe thank the open-source project [instant-nsr-pl](https://github.com/bennyguo/instant-nsr-pl), distributed under the MIT license. \n\n```\nCopyright (c) 2022 Yuanchen Guo\n```\n\n## License\nThis project is licensed under the [CC BY-NC 4.0](https://creativecommons.org/licenses/by-nc/4.0/) license.\nYou may use, share, and adapt the material for non-commercial purposes with appropriate credit.\n\n## Citation\n```\n@inproceedings{mvscps2025cao,\n  title = {Neural Multi-View Self-Calibrated Photometric Stereo without Photometric Stereo Cues},\n  author = {Cao, Xu and Taketomi, Takafumi},\n  year = {2025},\n  booktitle = {Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV)},\n}\n```","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcyberagentailab%2Fmvscps","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fcyberagentailab%2Fmvscps","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcyberagentailab%2Fmvscps/lists"}