{"id":33210060,"url":"https://postech-cvlab.github.io/SCNeRF/","last_synced_at":"2025-11-21T05:01:50.011Z","repository":{"id":41095132,"uuid":"393267665","full_name":"POSTECH-CVLab/SCNeRF","owner":"POSTECH-CVLab","description":"[ICCV21] Self-Calibrating Neural Radiance Fields","archived":false,"fork":false,"pushed_at":"2022-08-04T05:41:04.000Z","size":22872,"stargazers_count":470,"open_issues_count":4,"forks_count":43,"subscribers_count":15,"default_branch":"master","last_synced_at":"2025-07-21T06:49:03.209Z","etag":null,"topics":["calibration","computer-vision","deep-learning","implicit-representions","nerf","pytorch","self-calibration"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/POSTECH-CVLab.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2021-08-06T05:54:31.000Z","updated_at":"2025-06-26T16:45:49.000Z","dependencies_parsed_at":"2022-07-08T19:49:02.869Z","dependency_job_id":null,"html_url":"https://github.com/POSTECH-CVLab/SCNeRF","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/POSTECH-CVLab/SCNeRF","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/POSTECH-CVLab%2FSCNeRF","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/POSTECH-CVLab%2FSCNeRF/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/POSTECH-CVLab%2FSCNeRF/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/POSTECH-CVLab%2FSCNeRF/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/POSTECH-CVLab","download_url":"https://codeload.github.com/POSTECH-CVLab/SCNeRF/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/POSTECH-CVLab%2FSCNeRF/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":285560043,"owners_count":27192467,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-11-21T02:00:06.175Z","response_time":61,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["calibration","computer-vision","deep-learning","implicit-representions","nerf","pytorch","self-calibration"],"created_at":"2025-11-16T11:00:26.468Z","updated_at":"2025-11-21T05:01:50.006Z","avatar_url":"https://github.com/POSTECH-CVLab.png","language":"Python","funding_links":[],"categories":["Papers","Tools, Pipeline \u0026 Utilities"],"sub_categories":["Photogrammetry \u0026 3D Scanning"],"readme":"# Self-Calibrating Neural Radiance Fields, ICCV, 2021\n\n[Project Page](https://postech-cvlab.github.io/SCNeRF/) | [Paper](https://arxiv.org/abs/2108.13826) | [Video](https://www.youtube.com/watch?v=wsjx6geduvk)\n\n## Author Information\n\n- [Yoonwoo Jeong](https://yoonwooinfo.notion.site) [[Google Scholar]](https://scholar.google.com/citations?user=HQ1PMggAAAAJ\u0026hl=en)\n- Seokjun Ahn [[Google Scholar]](https://scholar.google.com/citations?user=JDEvLgwAAAAJ\u0026hl=en)\n- [Chris Choy](https://chrischoy.org) [[Google Scholar]](https://scholar.google.com/citations?user=2u8G5ksAAAAJ\u0026hl=en\u0026oi=ao)\n- [Anima Anandkumar](http://tensorlab.cms.caltech.edu/users/anima/) [[Google Scholar]](https://scholar.google.com/citations?user=bEcLezcAAAAJ\u0026hl=en\u0026oi=ao)\n- [Minsu Cho](http://cvlab.postech.ac.kr/~mcho/) [[Google Scholar]](https://scholar.google.com/citations?user=5TyoF5QAAAAJ\u0026hl=en\u0026oi=ao)\n- [Jaesik Park](http://jaesik.info/) [[Google Scholar]](https://scholar.google.com/citations?user=_3q6KBIAAAAJ\u0026hl=en\u0026oi=ao)\n\n\u003cdiv style=\"text-align:center\"\u003e\n\u003cimg src=\"assets/scnerf_teaser.png\" alt=\"Types of camera parameters\"/\u003e\n\u003c/div\u003e\n\n## News\n\n- 2021-09-02: The first version of Self-Calibrating Neural Radiance Fields is published\n\n## Overview\n\nIn this work, we propose a camera self-calibration algorithm for generic cameras with arbitrary non-linear distortions. We jointly learn the geometry of the scene and the accurate camera parameters without any calibration objects. Our camera model consists a pinhole model, radial distortion, and a generic noise model that can learn arbitrary non-linear camera distortions. While traditional self-calibration algorithms mostly rely on geometric constraints, we additionally incorporate photometric consistency. This requires learning the geometry of the scene and we use Neural Radiance Fields (NeRF).\nWe also propose a new geometric loss function, viz., projected ray distance loss, to incorporate geometric consistency for complex non-linear camera models. We validate our approach on standard real image datasets and demonstrate our model can learn the camera intrinsics and extrinsics (pose) from scratch without COLMAP initialization. Also, we show that learning accurate camera models in differentiable manner allows us to improves PSNR over NeRF. We experimentally demonstrate that our proposed method is applicable to variants of NeRF. In addition, we use a set of images captured with a fish-eye lens to demonstrate that learning camera model jointly improves the performance significantly over the COLMAP initialization.\n\n## Method\n\n### Generic Camera Model\n\nWe provide the definition of our differentiable camera model that combines the pinhole camera model, radial distortion, and a generic non-linear camera distortion for self-calibration. Our differentiable generic camera model consists of four components: intrinsic, extrinsic, radial distortion, and non-linear distortion parameters. We show that modeling the rays more accurately (camera model) results in better neural rendering. The following figure shows the computational steps to generate rays of our proposed learnable generic camera model.\n\n\u003cp align=\"center\"\u003e\n\u003cimg src=\"assets/scnerf_computation.png\" alt=\"computational graph for rays\" width=\"600\"/\u003e\n\u003c/p\u003e\n\n\n### Projected Ray Distance\n\nThe generic camera model poses a new challenge defining a geometric loss. In most traditional work, the geometric loss is defined as an epipolar constraint that measures the distance between an epipolar line and the corresponding point, or reprojection error where a 3D point for a correspondence is defined first which is then projected to an image plane to measure the distance between the projection and the correspondence. In this work, rather than requiring a 3D reconstruction to compute an indirect loss like the reprojection error, we propose the projected ray distance loss that directly measures the discrepancy between rays using our generic camera model.\n\n\u003cp align=\"center\"\u003e\n\u003cimg src=\"assets/scnerf_prd.png\" alt=\"projected ray distance\" width=\"400\"/\u003e\n\u003c/p\u003e\n\n### Curriculum Learning\n\nThe camera parameters determine the positions and directions of the rays for NeRF learning, and unstable values often result in divergence or sub-optimal results. Thus, we incrementally add a subset of learning parameters to the optimization process to reduce the complexity of learning cameras and geometry jointly. First, we learn the NeRF network while initializing the camera focal lengths and camera centers to half the image width and height. Learning coarse geometry first is crucial since it initializes the network parameters suitable for learning better camera parameters. Next, we sequentially add camera parameters from the linear camera model, radial distortion, to nonlinear noise of ray direction, ray origin to the learning. We progressively make the camera model more complex to prevent the camera parameters from overfitting and also allows faster training.\n\n\n\u003cp align=\"center\"\u003e\n\u003cimg src=\"assets/scnerf_curriculum.png\" alt=\"curriculum learning\" width=\"400\"/\u003e\n\u003c/p\u003e\n\n\n## Installation\n\n### Requirements\n\n- Ubuntu 16.04 or higher\n- CUDA 11.1 or higher\n- Python v3.7 or higher\n- Pytorch v1.7 or higher\n- Hardware Spec\n    - GPUs 11GB (2080ti) or larger capacity\n    - For NeRF++, 2GPUs(2080ti) are required to reproduce the result\n    - For FishEyeNeRF experiments, we have used 4GPUs(V100). \n\n### Environment Setup\n\n- We recommend to conda for installation. All the requirements for two codes, NeRF and NeRF++, are included in the requirements.txt\n\n    ```\n    conda create -n icn python=3.8\n    conda activate icn\n    pip install -r requirements.txt\n    git submodule update --init --recursive\n    ```\n\n\n### Generating COLMAP poses of custom image set\n\n- We further provide a COLMAP pose generator that can be applied to custom image sets. Run the code below if you should acquire camera information for custome image sets. \n```\nbash colmap_utils/colmap.sh [path to image set]\n```\nThe image collection should be inside the directory \"images.\" Checkout the COLMAP document\n```\n[path]\n    |--- images\n```\n\n\n## Pretrained Weights \u0026 Qualitative Results\n\nHere, we provide pretrained weights for users to easily reproduce results in the paper. You can download the pretrained weight in the following link. In the link, we provide all the weights of experiments, reported in our paper. To load the pretrained weight, add the following argument at the end of argument in each script. In the zip file, we have also included qualitative results that are used in our paper.\n\nLink to download the pretrained weight: [[link]](https://drive.google.com/file/d/13TjJUnRPA5UjTfsTBkx18GGXeYEqxTJf/view?usp=sharing)\n\n\n## Datasets\n\nWe use three datasets for evaluation: LLFF dataset, tanks and temples dataset, and FishEyeNeRF dataset (Images captured with a fish-eye lens).\n- LLFF dataset: [[link]](https://drive.google.com/drive/folders/128yBriW1IG_3NJ5Rp7APSTZsJqdJdfc1)\n- Tanks and Temples dataset: [[link]](https://drive.google.com/file/d/11KRfN91W1AxAW6lOFs4EeYDbeoQZCi87/view?usp=sharing)\n- FishEyeNeRF: [[link]](https://drive.google.com/file/d/1VhnpMUIKEak4TBpKY4H8vMJJe--FM2k5/view?usp=sharing)\n\nPut the data in the directory \"data/\" then add soft link with one of the following:\n\n```\nln -s data NeRF/data\nln -s data nerfplusplus/data\nln -s data nerfplusplus/data/fisheyenerf\n```\n\n## Demo Code\n\nThe demo code is available at \"demo.sh\" file. This code runs curriculum learning in NeRF architecture. Please install the aforementioned requirements before running the code. To run the demo code, run:\n\n```\nsh demo.sh\n```\n\nIf you want to reproduce the results that are reported in our main paper, run the scripts in the \"scripts\" directory.\n\n```\nMain Table 1: Self-Calibration Experiment (LLFF)\nMain Table 2: Improvement over NeRF (LLFF)\nMain Table 3: Improvement over NeRF++ (Tanks and Temples)\nMain Table 4: Improvement over NeRF++ (Images with a fish-eye lens)\n```\n\nCode Example:\n\n```\nsh scripts/main_table_1/fern/main1_fern_ours.sh\nsh scripts/main_table_2/fern/main2_fern_ours.sh\nsh scripts/main_table_3/main3_m60.sh\nsh scripts/main_table_4/globe_ours.sh\n```\n\n## Citing Self-Calibrating Neural Radiance Fields\n\n```\n@inproceedings{SCNeRF2021,\n    author = {Yoonwoo Jeong, Seokjun Ahn, Christopher Choy, Animashree Anandkumar, \n    Minsu Cho, and Jaesik Park},\n    title = {Self-Calibrating Neural Radiance Fields},\n    booktitle = {ICCV},\n    year = {2021},\n}\n```\n\n## Concurrent Work\n\nWe list a few recent concurrent projects that tackle camera extrinsics (pose) optimization in NeRF. Note that our Self-Calibrating NeRF optimizes an extensive set of camera parameters for intrinsics, extrinsics, radial distortion, and non-linear distortion.\n\n- [NeRF--: Neural Radiance Fields Without Known Camera Parameters](https://arxiv.org/abs/2102.07064)\n- [BARF: Bundle-Adjusting Neural Radiance Fields](https://arxiv.org/abs/2104.06405)\n\n\n## Acknowledgements\n\nWe appreciate all ICCV reviewers for valuable comments. Their valuable suggestions have helped us to improve our paper.\nWe also acknowledge amazing implementations of NeRF++(https://github.com/Kai-46/nerfplusplus) and NeRF-pytorch(https://github.com/yenchenlin/nerf-pytorch).\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/postech-cvlab.github.io%2FSCNeRF%2F","html_url":"https://awesome.ecosyste.ms/projects/postech-cvlab.github.io%2FSCNeRF%2F","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/postech-cvlab.github.io%2FSCNeRF%2F/lists"}