{"id":20663695,"url":"https://github.com/vita-group/sinnerf","last_synced_at":"2025-09-09T22:49:54.987Z","repository":{"id":42518334,"uuid":"476138280","full_name":"VITA-Group/SinNeRF","owner":"VITA-Group","description":"[ECCV 2022] \"SinNeRF: Training Neural Radiance Fields on Complex Scenes from a Single Image\", Dejia Xu, Yifan Jiang, Peihao Wang, Zhiwen Fan, Humphrey Shi, Zhangyang Wang","archived":false,"fork":false,"pushed_at":"2022-07-17T19:11:26.000Z","size":55960,"stargazers_count":331,"open_issues_count":7,"forks_count":26,"subscribers_count":14,"default_branch":"master","last_synced_at":"2025-05-20T06:05:44.882Z","etag":null,"topics":["gan","nerf","neural-radiance-fields","pytorch","pytorch-lightning"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/VITA-Group.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2022-03-31T03:35:34.000Z","updated_at":"2025-04-29T06:16:11.000Z","dependencies_parsed_at":"2022-08-12T10:01:17.810Z","dependency_job_id":null,"html_url":"https://github.com/VITA-Group/SinNeRF","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/VITA-Group/SinNeRF","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/VITA-Group%2FSinNeRF","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/VITA-Group%2FSinNeRF/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/VITA-Group%2FSinNeRF/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/VITA-Group%2FSinNeRF/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/VITA-Group","download_url":"https://codeload.github.com/VITA-Group/SinNeRF/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/VITA-Group%2FSinNeRF/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":274375468,"owners_count":25273829,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-09-09T02:00:10.223Z","response_time":80,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["gan","nerf","neural-radiance-fields","pytorch","pytorch-lightning"],"created_at":"2024-11-16T19:19:20.076Z","updated_at":"2025-09-09T22:49:54.944Z","avatar_url":"https://github.com/VITA-Group.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# SinNeRF: Training Neural Radiance Fields on Complex Scenes from a Single Image\n\n[[Paper]](https://arxiv.org/abs/2204.00928) [[Website]](https://vita-group.github.io/SinNeRF/)\n\n\u003cdiv\u003e\n\u003cimg src=\"https://user-images.githubusercontent.com/10709657/161841696-b88f5797-6fd0-4f5c-a661-dd2802a12ac7.gif\" height=\"150\"/\u003e\n\u003cimg src=\"https://user-images.githubusercontent.com/10709657/161841718-047678ab-4511-41ee-aed6-e716d370bfd9.gif\" height=\"150\"/\u003e\n\u003cimg src=\"https://user-images.githubusercontent.com/10709657/161841788-0baebb06-0375-48c6-8cc1-b00c796967db.gif\" height=\"150\"/\u003e\n\u003c/div\u003e\n\n## Pipeline\n\n![](./docs/static/media/SinNeRF.drawio.01f837d9d69b1db62c00.jpg)\n\n## Code\n\n### Environment\n\n```\npip install -r requirements.txt\n```\n\n### Dataset Preparation\n\nPlease download the datasets from these links:\n\n- NeRF synthetic: Download `nerf_synthetic.zip` from https://drive.google.com/drive/folders/128yBriW1IG_3NJ5Rp7APSTZsJqdJdfc1\n- LLFF: Download `nerf_llff_data.zip` from https://drive.google.com/drive/folders/128yBriW1IG_3NJ5Rp7APSTZsJqdJdfc1\n- DTU: Download the preprocessed DTU training data from https://drive.google.com/file/d/1eDjh-_bxKKnEuz5h-HXS7EDJn59clx6V/view\n\nPlease download the depth from here: https://drive.google.com/drive/folders/13Lc79Ox0k9Ih2o0Y9e_g_ky41Nx40eJw?usp=sharing\n\n### Training\n\nIf you meet OOM issue, try:\n\n1. enable `precision=16`\n2. reduce the patch size `--patch_size` (or `--patch_size_x`, `--patch_size_y`) and enlarge the stride size `--sH`, `--sW`\n\n\u003cdetails\u003e\n  \u003csummary\u003eNeRF synthetic\u003c/summary\u003e\n\n\n- Step 1\n  ```\n  python train.py  --dataset_name blender_ray_patch_1image_rot3d  --root_dir  ../../dataset/nerf_synthetic/lego   --N_importance 64 --img_wh 400 400 --num_epochs 2000 --batch_size 1  --optimizer adam --lr 2e-4  --lr_scheduler steplr --decay_step 500 1000 --decay_gamma 0.5  --exp_name lego_s6 --with_ref --patch_size 64 --sW 6 --sH 6 --proj_weight 1 --depth_smooth_weight 0  --dis_weight 0 --num_gpus 4 --load_depth --depth_type nerf --model sinnerf --depth_weight 8 --vit_weight 10 --scan 4\n  ```\n\n- Step 2\n  ```\n  python train.py  --dataset_name blender_ray_patch_1image_rot3d  --root_dir  ../../dataset/nerf_synthetic/lego   --N_importance 64 --img_wh 400 400 --num_epochs 2000 --batch_size 1  --optimizer adam --lr 5e-5  --lr_scheduler steplr --decay_step 500 1000 --decay_gamma 0.5  --exp_name lego_s6_4ft --with_ref --patch_size 64 --sW 4 --sH 4 --proj_weight 1 --depth_smooth_weight 0  --dis_weight 0.01 --num_gpus 4 --load_depth --depth_type nerf --model sinnerf --depth_weight 8 --vit_weight 0 --pt_model xxx.ckpt --nerf_only  --scan 4\n  ```\n\n\u003c/details\u003e\n\n\u003cdetails\u003e\n  \u003csummary\u003eLLFF\u003c/summary\u003e\n\n- Step 1\n  ```\n  python train.py  --dataset_name llff_ray_patch_1image_proj  --root_dir  ../../dataset/nerf_llff_data/room   --N_importance 64 --img_wh 504 378 --num_epochs 2000 --batch_size 1  --optimizer adam --lr 2e-4  --lr_scheduler steplr --decay_step 500 1000 --decay_gamma 0.5  --exp_name llff_room_s4 --with_ref --patch_size_x 63 --patch_size_y 84 --sW 4 --sH 4 --proj_weight 1 --depth_smooth_weight 0  --dis_weight 0 --num_gpus 4 --load_depth --depth_type nerf --model sinnerf --depth_weight 8 --vit_weight 10\n  ```\n\n- Step 2\n  ```\n  python train.py  --dataset_name llff_ray_patch_1image_proj  --root_dir  ../../dataset/nerf_llff_data/room   --N_importance 64 --img_wh 504 378 --num_epochs 2000 --batch_size 1  --optimizer adam --lr 5e-5  --lr_scheduler steplr --decay_step 500 1000 --decay_gamma 0.5  --exp_name llff_room_s4_2ft --with_ref --patch_size_x 63 --patch_size_y 84 --sW 2 --sH 2 --proj_weight 1 --depth_smooth_weight 0  --dis_weight 0.01 --num_gpus 4 --load_depth --depth_type nerf --model sinnerf --depth_weight 8 --vit_weight 0 --pt_model xxx.ckpt --nerf_only\n  ```\n\n\u003c/details\u003e\n\n\u003cdetails\u003e\n  \u003csummary\u003eDTU\u003c/summary\u003e\n\n- Step 1\n  ```\n  python train.py  --dataset_name dtu_proj  --root_dir  ../../dataset/mvs_training/dtu   --N_importance 64 --img_wh 640 512 --num_epochs 2000 --batch_size 1  --optimizer adam --lr 2e-4  --lr_scheduler steplr --decay_step 500 1000 --decay_gamma 0.5  --exp_name dtu_scan4_s8 --with_ref --patch_size_y 70 --patch_size_x 56 --sW 8 --sH 8 --proj_weight 1 --depth_smooth_weight 0  --dis_weight 0 --num_gpus 4 --load_depth --depth_type nerf --model sinnerf --depth_weight 8 --vit_weight 10 --scan 4\n  ```\n\n- Step 2\n  ```\n  python train.py  --dataset_name dtu_proj  --root_dir  ../../dataset/mvs_training/dtu   --N_importance 64 --img_wh 640 512 --num_epochs 2000 --batch_size 1  --optimizer adam --lr 5e-5  --lr_scheduler steplr --decay_step 500 1000 --decay_gamma 0.5  --exp_name dtu_scan4_s8_4ft --with_ref --patch_size_y 70 --patch_size_x 56 --sW 4 --sH 4 --proj_weight 1 --depth_smooth_weight 0  --dis_weight 0.01 --num_gpus 4 --load_depth --depth_type nerf --model sinnerf --depth_weight 8 --vit_weight 0 --pt_model xxx.ckpt --nerf_only  --scan 4\n  ```\n\nMore finetuning with smaller strides benefits reconstruction quality.\n\n\u003c/details\u003e\n\n\n### Testing\n\n```\npython eval.py  --dataset_name llff  --root_dir /dataset/nerf_llff_data/room --N_importance 64 --img_wh 504 378 --model nerf --ckpt_path ckpts/room.ckpt --timestamp test\n```\n\nPlease use `--split val` for NeRF synthetic dataset.\n\n## Acknowledgement\n\nCodebase based on https://github.com/kwea123/nerf_pl . Thanks for sharing!\n\n## Citation\n\nIf you find this repo is helpful, please cite:\n\n```\n\n@InProceedings{Xu_2022_SinNeRF,\nauthor = {Xu, Dejia and Jiang, Yifan and Wang, Peihao and Fan, Zhiwen and Shi, Humphrey and Wang, Zhangyang},\ntitle = {SinNeRF: Training Neural Radiance Fields on Complex Scenes from a Single Image},\njournal={arXiv preprint arXiv:2204.00928},\nyear={2022}\n}\n\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fvita-group%2Fsinnerf","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fvita-group%2Fsinnerf","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fvita-group%2Fsinnerf/lists"}