{"id":20515393,"url":"https://github.com/snap-research/r2l","last_synced_at":"2026-03-14T14:03:36.709Z","repository":{"id":42498772,"uuid":"468126849","full_name":"snap-research/R2L","owner":"snap-research","description":"[ECCV 2022] R2L: Distilling Neural Radiance Field to Neural Light Field for Efficient Novel View Synthesis","archived":false,"fork":false,"pushed_at":"2023-08-15T04:59:36.000Z","size":74677,"stargazers_count":192,"open_issues_count":3,"forks_count":23,"subscribers_count":16,"default_branch":"main","last_synced_at":"2024-12-10T19:23:07.692Z","etag":null,"topics":["deep-learning","distillation","mlp","nerf","neural-light-field","novel-view-synthesis","rendering"],"latest_commit_sha":null,"homepage":"https://snap-research.github.io/R2L/","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"other","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/snap-research.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2022-03-09T23:37:27.000Z","updated_at":"2024-12-10T06:21:59.000Z","dependencies_parsed_at":"2024-11-07T03:40:31.402Z","dependency_job_id":null,"html_url":"https://github.com/snap-research/R2L","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/snap-research%2FR2L","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/snap-research%2FR2L/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/snap-research%2FR2L/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/snap-research%2FR2L/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/snap-research","download_url":"https://codeload.github.com/snap-research/R2L/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":230382858,"owners_count":18216854,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["deep-learning","distillation","mlp","nerf","neural-light-field","novel-view-synthesis","rendering"],"created_at":"2024-11-15T21:21:25.177Z","updated_at":"2026-03-14T14:03:31.659Z","avatar_url":"https://github.com/snap-research.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# R2L: Distilling NeRF to NeLF\n\n### [Project](https://snap-research.github.io/R2L/) | [ArXiv](https://arxiv.org/abs/2203.17261) | [PDF](https://arxiv.org/pdf/2203.17261.pdf) \n\n\u003cdiv align=\"center\"\u003e\n    \u003ca\u003e\u003cimg src=\"figs/snap.svg\"  height=\"120px\" \u003e\u003c/a\u003e\n    \u0026nbsp\n    \u003ca\u003e\u003cimg src=\"figs/smile.png\"  height=\"100px\" \u003e\u003c/a\u003e\n\u003c/div\u003e\n\nThis repository is for the new neral light field (NeLF) method introduced in the following ECCV'22 paper:\n\u003e **[R2L: Distilling Neural Radiance Field to Neural Light Field for Efficient Novel View Synthesis](https://snap-research.github.io/R2L/)** \\\n\u003e [Huan Wang](http://huanwang.tech/) \u003csup\u003e1,2\u003c/sup\u003e, [Jian Ren](https://alanspike.github.io/) \u003csup\u003e1\u003c/sup\u003e, [Zeng Huang](https://zeng.science/) \u003csup\u003e1\u003c/sup\u003e, [Kyle Olszewski](https://kyleolsz.github.io/) \u003csup\u003e1\u003c/sup\u003e, [Menglei Chai](https://mlchai.com/) \u003csup\u003e1\u003c/sup\u003e, [Yun Fu](http://www1.ece.neu.edu/~yunfu/) \u003csup\u003e2\u003c/sup\u003e, and [Sergey Tulyakov](http://www.stulyakov.com/) \u003csup\u003e1\u003c/sup\u003e \\\n\u003e \u003csup\u003e1\u003c/sup\u003e Snap Inc. \u003csup\u003e2\u003c/sup\u003e Northeastern University \\\n\u003e Work done when Huan was an intern at Snap Inc.\n\n**[TL;DR]** We present R2L, a deep (88-layer) residual MLP network that can represent the neural *light* field (NeLF) of complex synthetic and real-world scenes. It is featured by compact representation size (~20MB storage size), faster rendering speed (~30x speedup than NeRF), significantly improved visual quality (1.4dB boost than NeRF), with no whistles and bells (no special data structure or parallelism required).\n\n\u003cdiv align=\"center\"\u003e\n    \u003ca\u003e\u003cimg src=\"figs/frontpage.png\"  width=\"700\" \u003e\u003c/a\u003e\n\u003c/div\u003e\n\n\n## Reproducing Our Results\nBelow we only show the example of scene `lego`. You may test on other scenes simply by changing all the `lego` word segment to other scene names. Scripts have been doubled-checked. You should be able to **run them simply by copy-paste**.  \n\n### 0. Download the code\n```\ngit clone git@github.com:snap-research/R2L.git \u0026\u0026 cd R2L\n```\n\n\n### 1. Set up (original) data\n```bash\nsh scripts/download_example_data.sh\n```\n\n### 2. Set up environment with Anaconda\n- `conda create --name R2L python=3.9.6`\n- `conda activate R2L`\n- `pip install -r requirements.txt` (We use torch 1.9.0, torchvision 0.10.0)\n\n### 3. Quick start: test our trained models\n- Download models:\n```\nsh scripts/download_R2L_models.sh\n```\n\n- Run\n```bash\nCUDA_VISIBLE_DEVICES=0 python main.py --model_name R2L --config configs/lego_noview.txt --n_sample_per_ray 16 --netwidth 256 --netdepth 88 --use_residual --trial.ON --trial.body_arch resmlp --pretrained_ckpt R2L_Blender_Models/lego.tar --render_only --render_test --testskip 1 --experiment_name Test__R2L_W256D88__blender_lego\n```  \n \n### 4. Train R2L models\nThere are two major steps in R2L training. (1) Use *pretrained* NeRF model to generate synthetic data and train R2L network on the synthetic data -- this step can make our R2L model perform *comparably* to the NeRF teacher; (2) Finetune the R2L model in (1) with the *real* data -- this step will further boost the performance and make our R2L model *outperform* the NeRF teacher.\n\nThe detailed step-by-step training pipeline is as follows.\n\n#### Step 1. \nTrain a NeRF model:\n```bash\nCUDA_VISIBLE_DEVICES=0 python main.py --model_name nerf --config configs/lego.txt --experiment_name NeRF__blender_lego\n```\n\nYou can also download the teachers we trained to continue first:\n```bash\nsh scripts/download_NeRF_models.sh\n```\n\nTo test the download teachers, you can use\n```bash\nCUDA_VISIBLE_DEVICES=0 python main.py --model_name nerf --config configs/lego.txt --pretrained_ckpt NeRF_Blender_Models/lego.tar --render_only --render_test --testskip 1 --experiment_name Test__NeRF__blender_lego\n```\n\n\n#### Step 2. \nUse the pretrained NeRF model to generate synthetic data (saved in `.npy` format):\n```bash\nCUDA_VISIBLE_DEVICES=0 python utils/create_data.py --create_data rand --config configs/lego.txt --teacher_ckpt Experiments/NeRF__blender_lego*/weights/200000.tar --n_pose_kd 10000 --datadir_kd data/nerf_synthetic/lego:data/nerf_synthetic/lego_pseudo_images10k --experiment_name NeRF__blender_lego__create_pseudo\n```\n\nIf you are using the downloaded teachers, please use this snippet:\n```bash\nCUDA_VISIBLE_DEVICES=0 python utils/create_data.py --create_data rand --config configs/lego.txt --teacher_ckpt NeRF_Blender_Models/lego.tar --n_pose_kd 10000 --datadir_kd data/nerf_synthetic/lego:data/nerf_synthetic/lego_pseudo_images10k --experiment_name NeRF__blender_lego__create_pseudo\n```\n\nThe pseudo data will be saved in `data/nerf_synthetic/lego_pseudo_images10k`. Every 4096 rays are saved in one .npy file. For 10k images (400x400 resoltuion), there will be 309600 .npy files. On our RTX 2080Ti GPU, rendering 1 image with NeRF takes around 8.5s, so 10k images would take around 24hrs. **If you want to try our method quicker, you may download the lego data we synthesized** (500 images, 2.8GB) and go to Step 3:\n```bash\nsh scripts/download_lego_pseudo_images500.sh\n```\nThe data will be extracted under `data/nerf_synthetic/lego_pseudo_images500`. Using only 500 pseudo images for training would lead to degraded quality, but based on our ablation study (see Fig. 6 in our paper), it works farily good.\n\n\n#### Step 3.\nTrain R2L model on the synthetic data:\n```bash\nCUDA_VISIBLE_DEVICES=0 python main.py --model_name R2L --config configs/lego_noview.txt --n_sample_per_ray 16 --netwidth 256 --netdepth 88 --datadir_kd data/nerf_synthetic/lego_pseudo_images10k --n_pose_video 20,1,1 --N_iters 1200000 --N_rand 20 --data_mode rays --hard_ratio 0.2 --hard_mul 20 --use_residual --trial.ON --trial.body_arch resmlp --num_worker 8 --warmup_lr 0.0001,200 --experiment_name R2L__blender_lego\n```\n\nIf you are using the downloaded `lego_pseudo_images500` data, please use this snippet:\n```bash\nCUDA_VISIBLE_DEVICES=0 python main.py --model_name R2L --config configs/lego_noview.txt --n_sample_per_ray 16 --netwidth 256 --netdepth 88 --datadir_kd data/nerf_synthetic/lego_pseudo_images500 --n_pose_video 20,1,1 --N_iters 1200000 --N_rand 20 --data_mode rays --hard_ratio 0.2 --hard_mul 20 --use_residual --trial.ON --trial.body_arch resmlp --num_worker 8 --warmup_lr 0.0001,200 --experiment_name R2L__blender_lego\n```\n\n#### Step 4. \nConvert original real data (images) to our `.npy` format:\n* For blender data:\n```bash\npython utils/convert_original_data_to_rays_blender.py --splits train --datadir data/nerf_synthetic/lego\n```\nThe converted data will be saved in `data/nerf_synthetic/lego_real_train`.\n\n* For llff data:\n```bash\npython utils/convert_original_data_to_rays_llff.py --splits train --datadir data/nerf_llff_data/flower\n```\nThe converted data will be saved in `data/nerf_llff_data/room_real_train`.\n\n\n#### Step 5. \nFinetune the R2L model in Step 3 on the data in Step 4:\n```bash\nCUDA_VISIBLE_DEVICES=0 python main.py --model_name R2L --config configs/lego_noview.txt --n_sample_per_ray 16 --netwidth 256 --netdepth 88 --datadir_kd data/nerf_synthetic/lego_real_train --n_pose_video 20,1,1 --N_iters 1600000 --N_rand 20 --data_mode rays --hard_ratio 0.2 --hard_mul 20 --use_residual --trial.ON --trial.body_arch resmlp --num_worker 8 --warmup_lr 0.0001,200 --save_intermediate_models --pretrained_ckpt Experiments/R2L__blender_lego_SERVER*/weights/ckpt_1200000.tar --resume --experiment_name R2L__blender_lego__ft\n```\nNote, this step is pretty fast and prone to overfitting, so do not finetune it too much. We simply set the finetuning steps based on our validation.\n\n\n## Results\nThe quantitative and qualitative comparison are shown below. See more results and videos on our [webpage](https://snap-research.github.io/R2L/).\n\u003cdiv align=\"center\"\u003e\n    \u003ca\u003e\u003cimg src=\"figs/blender_psnr_comparison.png\"  width=\"700\" \u003e\u003c/a\u003e\u003cbr\u003e\n    \u003ca\u003e\u003cimg src=\"figs/blender_visual_comparison.png\"  width=\"700\"\u003e\u003c/a\u003e\n\u003c/div\u003e\n\n\n## Acknowledgments\nIn this code we refer to the following implementations: [nerf-pytorch](https://github.com/yenchenlin/nerf-pytorch) and [smilelogging](https://github.com/MingSun-Tse/smilelogging). Great thanks to them! We especially thank [nerf-pytorch](https://github.com/yenchenlin/nerf-pytorch). Our code is largely built upon their wonderful implementation. We also greatly thank the anounymous ECCV'22 reviewers for the constructive comments to help us improve the paper.\n\n## Reference\n\nIf our work or code helps you, please consider to cite our paper. Thank you!\n```BibTeX\n@inproceedings{wang2022r2l,\n  author = {Huan Wang and Jian Ren and Zeng Huang and Kyle Olszewski and Menglei Chai and Yun Fu and Sergey Tulyakov},\n  title = {R2L: Distilling Neural Radiance Field to Neural Light Field for Efficient Novel View Synthesis},\n  booktitle = {ECCV},\n  year = {2022}\n}\n```\n\n\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsnap-research%2Fr2l","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fsnap-research%2Fr2l","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsnap-research%2Fr2l/lists"}