{"id":18614335,"url":"https://github.com/sxyu/pixel-nerf","last_synced_at":"2025-04-08T08:14:57.637Z","repository":{"id":37717159,"uuid":"317389846","full_name":"sxyu/pixel-nerf","owner":"sxyu","description":"PixelNeRF Official Repository","archived":false,"fork":false,"pushed_at":"2024-06-30T21:29:35.000Z","size":2864,"stargazers_count":1425,"open_issues_count":55,"forks_count":204,"subscribers_count":22,"default_branch":"master","last_synced_at":"2025-04-01T05:34:17.274Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"https://alexyu.net/pixelnerf","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"bsd-2-clause","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/sxyu.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2020-12-01T01:19:44.000Z","updated_at":"2025-03-28T08:14:33.000Z","dependencies_parsed_at":"2024-12-05T19:02:08.530Z","dependency_job_id":"0dff3245-c0a4-4d89-acc1-b5877181d1dd","html_url":"https://github.com/sxyu/pixel-nerf","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sxyu%2Fpixel-nerf","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sxyu%2Fpixel-nerf/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sxyu%2Fpixel-nerf/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sxyu%2Fpixel-nerf/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/sxyu","download_url":"https://codeload.github.com/sxyu/pixel-nerf/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":247801175,"owners_count":20998339,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-07T03:25:54.768Z","updated_at":"2025-04-08T08:14:57.607Z","avatar_url":"https://github.com/sxyu.png","language":"Python","funding_links":[],"categories":["Papers"],"sub_categories":["NeRF"],"readme":"# pixelNeRF: Neural Radiance Fields from One or Few Images\n\nAlex Yu, Vickie Ye, Matthew Tancik, Angjoo Kanazawa\u003cbr\u003e\nUC Berkeley\n\n![Teaser](https://raw.github.com/sxyu/pixel-nerf/master/readme-img/paper_teaser.jpg)\n\narXiv: http://arxiv.org/abs/2012.02190\n\nThis is the official repository for our paper, pixelNeRF, pending final release.\nThe two object experiment is still missing.\nSeveral features may also be added.\n\n# Environment setup\n\nTo start, we prefer creating the environment using conda:\n```sh\nconda env create -f environment.yml\nconda activate pixelnerf\n```\nPlease make sure you have up-to-date NVIDIA drivers supporting CUDA 10.2 at least.\n\nAlternatively use `pip -r requirements.txt`.\n\n# Getting the data\n\n- For the main ShapeNet experiments, we use the ShapeNet 64x64 dataset from NMR\nhttps://s3.eu-central-1.amazonaws.com/avg-projects/differentiable_volumetric_rendering/data/NMR_Dataset.zip\n(Hosted by DVR authors)\n    - For novel-category generalization experiment, a custom split is needed.\n      Download the following script:\n      https://drive.google.com/file/d/1Uxf0GguAUTSFIDD_7zuPbxk1C9WgXjce/view?usp=sharing\n      place the said file under `NMR_Dataset` and run `python genlist.py` in the said directory.\n      This generates train/val/test lists for the experiment. Note for evaluation performance reasons,\n      test is only 1/4 of the unseen categories.\n\n- The remaining datasets may be found in\nhttps://drive.google.com/drive/folders/1PsT3uKwqHHD2bEEHkIXB99AlIjtmrEiR?usp=sharing\n    - Custom two-chair `multi_chair_{train/val/test}.zip`. Download splits into a parent directory and pass the parent directory path to training command.\n        - To render out your own dataset, feel free to use our script in `scripts/render_shapenet.py`. See `scripts/README.md` for installation instructions.\n    - DTU (4x downsampled, rescaled) in DVR's DTU format `dtu_dataset.zip`\n    - SRN chair/car (128x128) `srn_*.zip` needed for single-category exps.\n      Note the car set is a re-rendered version provided by Vincent Sitzmann\n\nWhile we could have used a common data format, we chose to keep\nDTU and ShapeNet (NMR) datasets in DVR's format and SRN data in the original SRN format.\nOur own two-object data is in NeRF's format.\nData adapters are built into the code.\n\n# Running the model (video generation)\n\nThe main implementation is in the `src/` directory,\nwhile evalutation scripts are in `eval/`.\n\nFirst, download all pretrained weight files from\n\u003chttps://drive.google.com/file/d/1UO_rL201guN6euoWkCOn-XpqR2e8o6ju/view?usp=sharing\u003e.\nExtract this to `\u003cproject dir\u003e/checkpoints/`, so that `\u003cproject dir\u003e/checkpoints/dtu/pixel_nerf_latest` exists.\n\n\n## ShapeNet Multiple Categories (NMR)\n\n1. Download NMR ShapeNet renderings (see Datasets section, 1st link)\n2. Run using\n    - `python eval/gen_video.py  -n sn64 --gpu_id \u003cGPU(s)\u003e --split test -P '2'  -D \u003cdata_root\u003e/NMR_Dataset -S 0`\n    - For unseen category generalization:\n      `python eval/gen_video.py  -n sn64_unseen --gpu_id=\u003cGPU(s)\u003e --split test -P '2'  -D \u003cdata_root\u003e/NMR_Dataset -S 0`\n\nReplace `\u003cGPU(s)\u003e` with desired GPU id(s), space separated for multiple.  Replace `-S 0` with `-S \u003cobject_id\u003e` to run on a different ShapeNet object id.\nReplace `-P '2'` with `-P '\u003cnumber\u003e'` to use a different input view.\nReplace `--split test` with `--split train | val` to use different data split.\nAppend `-R=20000` if running out of memory.\n\n**Result will be at** `visuals/sn64/videot\u003cobject_id\u003e.mp4` or `visuals/sn64_unseen/videot\u003cobject_id\u003e.mp4`.\nThe script will also print the path.\n\n\nPre-generated results for all ShapeNet objects with comparison may be found at \u003chttps://www.ocf.berkeley.edu/~sxyu/ZG9yaWF0aA/pixelnerf/cross_v2/\u003e\n\n## ShapeNet Single-Category (SRN)\n\n1. Download SRN car (or chair) dataset from Google drive folder in Datasets section.\nExtract to `\u003csrn data dir\u003e/cars_\u003ctrain | test | val\u003e`\n2. `python eval/gen_video.py -n srn_car --gpu_id=\u003cGPU (s)\u003e --split test -P '64 104' -D \u003csrn data dir\u003e/cars -S 1`\n\nUse `-P 64` for 1-view (view numbers are from SRN).\nThe chair set case is analogous (replace car with chair).\nOur models are trained with random 1/2 views per batch during training.\nThis seems to degrade performance especially for 1-view. It may be preferrable to use \na fixed number of views instead.\n\n## DTU\n\nMake sure you have downloaded the pretrained weights above.\n\n1. Download DTU dataset from Google drive folder in Datasets section. Extract to some directory, to get: `\u003cdata_root\u003e/rs_dtu_4`\n2. Run using `python eval/gen_video.py  -n dtu --gpu_id=\u003cGPU(s)\u003e --split val -P '22 25 28'  -D \u003cdata_root\u003e/rs_dtu_4 -S 3 --scale 0.25`\n\nReplace `\u003cGPU(s)\u003e` with desired GPU id(s). Replace `-S 3` with `-S \u003cscene_id\u003e` to run on a different scene. This is not DTU scene number but 0-14 in the val set.\nRemove `--scale 0.25` to render at full resolution (quite slow).\n\n**Result will be at** visuals/dtu/videov\u003cscene_id\u003e.mp4.\nThe script will also print the path.\n\nNote that for DTU, I only use train/val sets, where val is used for test. This is due to the very small size of the dataset.\nThe model overfits to the train set significantly during training.\n\n## Real Car Images\n\n**Note: requires PointRend from detectron2.**\nInstall detectron2 by following https://github.com/facebookresearch/detectron2/blob/master/INSTALL.md.\n\nMake sure you have downloaded the pretrained weights above.\n\n1. Download any car image.\nPlace it in `\u003cproject dir\u003e/input`. Some example images are shipped with the repo.\nThe car should be fully visible.\n2. Run the preprocessor script: `python scripts/preproc.py`. This saves `input/*_normalize.png`.\nIf the result is not reasonable, PointRend didn't work; please try another imge.\n3. Run `python eval/eval_real.py`. Outputs will be in `\u003cproject dir\u003e/output`\n\nThe Stanford Car dataset contains many example car images:\n\u003chttps://ai.stanford.edu/~jkrause/cars/car_dataset.html\u003e.\nNote the normalization heuristic has been slightly modified compared to the paper. There may be some minor differences.\nYou can pass `-e -20` to `eval_real.py` to set the elevation higher in the generated video.\n\n# Overview of flags\n\nGenerally, all scripts in the project take the following flags\n- `-n \u003cexpname\u003e`: experiment name, matching checkpoint directory name\n- `-D \u003cdatadir\u003e`: dataset directory. \n    To save typing, you can set a default data directory for each expname in `expconf.conf` under `datadir`.\n    For SRN/multi_obj datasets with\n    separate directories e.g. `path/cars_train`, `path/cars_val`,\n    put `-D path/cars`.\n- `--split \u003ctrain | val | test\u003e`: data set split\n- `-S \u003csubset_id\u003e`: scene or object id to render\n- `--gpu_id \u003cGPU(s)\u003e`: GPU id(s) to use, space delimited. All scripts except `calc_metrics.py`\nare parallelized. If not specified, uses GPU 0.\nExamples: `--gpu_id=0` or `--gpu_id='0 1 3'`.\n- `-R \u003csz\u003e`: Batch size of rendered rays per object. Default is 50000 (eval) and 128 (train); make it smaller if you run out of memory.  On large-memory GPUs, you can set it to 100000 for eval.\n- `-c \u003cconf/*.conf\u003e`: config file. *Automatically inferred* for the provided experiments from the expname. Thus the flag is only required when working with your own expnames.\n                    You can associate a config file with any additional expnames in the `config` section of `\u003cproject root\u003e/expconf.conf`.\n\nPlease refer the the following table for a list of provided experiments with associated config and data files:\n\n| Name                       | expname -n    | config -c (automatic from expconf.conf)   | Data file                               | data dir -D       |\n|----------------------------|---------------|-------------------------------------------|-----------------------------------------|-------------------|\n| ShapeNet category-agnostic | sn64          | conf/exp/sn64.conf                 | NMR_Dataset.zip (from AWS)              | path/NMR_Dataset  |\n| ShapeNet unseen category   | sn64_unseen   | conf/exp/sn64_unseen.conf          | NMR_Dataset.zip (from AWS) + genlist.py | path/NMR_Dataset  |\n| SRN chairs                 | srn_chair     | conf/exp/srn.conf                  | srn_chairs.zip                          | path/chairs       |\n| SRN cars                   | srn_car       | conf/exp/srn.conf                  | srn_cars.zip                            | path/cars         |\n| DTU                        | dtu           | conf/exp/dtu.conf                  | dtu_dataset.zip                         | path/rs_dtu_4     |\n| Two chairs                 | mult_obj      | conf/exp/mult_obj.conf             | multi_chair_{train/val/test}.zip        | path              |\n\n\n# Quantitative evaluation instructions\n\nAll evaluation code is in `eval/` directory.\nThe full, parallelized evaluation code is in `eval/eval.py`.\n\n## Approximate Evaluation\nThe full evaluation can be extremely slow (taking many days), especially for the SRN dataset.\nTherefore we also provide `eval_approx.py` for *approximate* evaluation.\n\n- Example `python eval/eval_approx.py -D \u003csrn_data\u003e/cars -n srn_car`\n\nAdd `--seed \u003cnumber\u003e` to try a different random seed.\n\n## Full Evaluation\n\nHere we provide commands for full evaluation with `eval/eval.py`.\nAfter running this you should also use `eval/calc_metrics.py`, described in the section below,\nto obtain final metrics.\n\nAppend `--gpu_id=\u003cGPUs\u003e` to specify GPUs, for example `--gpu_id=0` or `--gpu_id='0 1 3'`.\n**It is highly recommended to use multiple GPUs if possible to finish in reasonable time.**\nWe use 4-10 for evaluations as available.\nResume-capability is built-in, and you can simply run the command again to resume if the process is terminated.\n\nIn all cases, a source-view specification is required. This can be either `-P` or `-L`. `-P 'view1 view2..'` specifies\na set of fixed input views. In contrast, `-L` should point to a viewlist file (viewlist/src_*.txt) which specifies views to use for each object.\n\nRenderings and progress will be saved to the output directory, specified by `-O \u003cdirname\u003e`.\n\n### ShapeNet Multiple Categories (NMR)\n\n- Category-agnostic eval `python eval/eval.py -D \u003cpath\u003e/NMR_Dataset -n sn64 -L viewlist/src_dvr.txt --multicat -O eval_out/sn64`\n- Unseen category eval `python eval/eval.py -D \u003cpath\u003e/NMR_Dataset -n sn64_unseen -L viewlist/src_gen.txt --multicat -O eval_out/sn64_unseen`\n\n### ShapeNet Single-Category (SRN)\n\n- SRN car 1-view eval `python eval/eval.py -D \u003csrn_data\u003e/cars -n srn_car -P '64' -O eval_out/srn_car_1v`\n- SRN car 2-view eval `python eval/eval.py -D \u003csrn_data\u003e/cars -n srn_car -P '64 104' -O eval_out/srn_car_2v`\n\nThe command for chair is analogous (replace car with chair). The input views 64, 104 are taken from SRN.\nOur method is by no means restricted to using such views.\n\n### DTU\n- 1-view `python eval/eval.py -D \u003cdata\u003e/rs_dtu_4 --split val -n dtu -P '25' -O eval_out/dtu_1v`\n- 3-view `python eval/eval.py -D \u003cdata\u003e/rs_dtu_4 --split val -n dtu -P '22 25 28' -O eval_out/dtu_3v`\n- 6-view `python eval/eval.py -D \u003cdata\u003e/rs_dtu_4 --split val -n dtu -P '22 25 28 40 44 48' -O eval_out/dtu_6v`\n- 9-view `python eval/eval.py -D \u003cdata\u003e/rs_dtu_4 --split val -n dtu -P '22 25 28 40 44 48 0 8 13' -O eval_out/dtu_9v`\n\nIn training, we always provide 3-views, so the improvement with more views is limited.\n\n## Final Metric Computation\n\nThe above computes PSNR and SSIM without quantization. The final metrics we report in the paper\nuse the rendered images saved to disk, and also includes LPIPS + category breakdown.\nTo do so run the `eval/calc_metrics.py`, as in the following examples\n\n- NMR ShapeNet experiment: `python eval/calc_metrics.py -D \u003cdata dir\u003e/NMR_Dataset -O eval_out/sn64 -F dvr --list_name 'softras_test' --multicat --gpu_id=\u003cGPU\u003e`\n- SRN car 2-view: `python eval/calc_metrics.py -D \u003csrn data dir\u003e/cars -O eval_out/srn_car_2v -F srn --gpu_id=\u003cGPU\u003e` (warning: untested after changes)\n- DTU: `python eval/calc_metrics.py -D \u003cdata dir\u003e/rs_dtu_4/DTU -O eval_out/dtu_3v -F dvr --list_name 'new_val' --exclude_dtu_bad --dtu_sort`\n\nAdjust -O according to the -O flag of the eval.py command.\n(Note: Currently this script has an ugly standalone argument parser.)\nThis should print a metric summary like the following\n```\npsnr 26.799268696042386\nssim 0.9102204550379002\nlpips 0.10784384977842876\nWROTE eval_sn64/all_metrics.txt\nairplane     psnr: 29.756697 ssim: 0.946906 lpips: 0.084329 n_inst: 809\nbench        psnr: 26.351427 ssim: 0.911226 lpips: 0.116299 n_inst: 364\ncabinet      psnr: 27.720198 ssim: 0.910426 lpips: 0.104584 n_inst: 315\ncar          psnr: 27.579590 ssim: 0.942079 lpips: 0.094841 n_inst: 1500\nchair        psnr: 23.835303 ssim: 0.857738 lpips: 0.145518 n_inst: 1356\ndisplay      psnr: 24.217023 ssim: 0.867284 lpips: 0.129138 n_inst: 219\nlamp         psnr: 28.579184 ssim: 0.912794 lpips: 0.113561 n_inst: 464\nloudspeaker  psnr: 24.435302 ssim: 0.855195 lpips: 0.140653 n_inst: 324\nrifle        psnr: 30.597488 ssim: 0.968040 lpips: 0.065629 n_inst: 475\nsofa         psnr: 26.944224 ssim: 0.907861 lpips: 0.116114 n_inst: 635\ntable        psnr: 25.591960 ssim: 0.898314 lpips: 0.098103 n_inst: 1702\ntelephone    psnr: 27.128039 ssim: 0.921897 lpips: 0.097074 n_inst: 211\nvessel       psnr: 29.180307 ssim: 0.938936 lpips: 0.110670 n_inst: 388\n---\ntotal        psnr: 26.799269 ssim: 0.910220 lpips: 0.107844\n```\n\n# Training instructions\n\nTraining code is in `train/` directory, specifically `train/train.py`.\n\n- Example for training to DTU: `python train/train.py -n dtu_exp -c conf/exp/dtu.conf -D \u003cdata dir\u003e/rs_dtu_4 -V 3 --gpu_id=\u003cGPU\u003e --resume`\n- Example for training to SRN cars, 1 view: `python train/train.py -n srn_car_exp -c conf/exp/srn.conf -D \u003csrn data dir\u003e/cars --gpu_id=\u003cGPU\u003e --resume`\n- Example for training to ShapeNet multi-object, 2 view: `python train/train.py -n multi_obj -c conf/exp/multi_obj.conf -D \u003cparent dir of splits\u003e --gpu_id=\u003cGPU\u003e --resume`\n\nAdditional flags\n- `--resume` to resume from checkpoint, if available. Usually just pass this to be safe.\n- `-V \u003cnumber\u003e` to specify number of input views to train with. Default is 1.\n    - `-V 'numbers separated by space'` to use random number of views per batch. This does not work so well in our experience but we use it for SRN experiment.\n- `-B \u003cnumber\u003e` batch size of objects, default 4\n- `--lr \u003clearning rate\u003e`, `--epochs \u003cnumber of epochs\u003e`\n- `--no_bbox_step \u003cnumber\u003e` to specify iteration after which to stop using bounding-box sampling.\nSet to 0 to disable.\n\nIf the checkpoint becomes corrupted for some reason (e.g. if process crashes when saving), a backup is saved to `checkpoints/\u003cexpname\u003e/pixel_nerf_backup`.\nTo avoid having to specify -c, -D each time, edit `\u003cproject root\u003e/expconf.conf` and add rows for your expname in the config and datadir sections.\n\n## Log files and visualizations\nView logfiles with `tensorboard --logdir \u003cproject dir\u003e/logs/\u003cexpname\u003e`.\nVisualizations are written to  `\u003cproject dir\u003e/visuals/\u003cexpname\u003e/\u003cepoch\u003e_\u003cbatch\u003e_vis.png`.\nThey are of the form\n- Top coarse, bottom fine (1 row if fine sample disabled)\n- Left-to-right: input-views, depth, output, alpha.\n\n# BibTeX\n\n```\n@inproceedings{yu2021pixelnerf,\n      title={{pixelNeRF}: Neural Radiance Fields from One or Few Images},\n      author={Alex Yu and Vickie Ye and Matthew Tancik and Angjoo Kanazawa},\n      year={2021},\n      booktitle={CVPR},\n}\n```\n\n# Acknowledgements\n\nParts of the code were based on from kwea123's NeRF implementation: https://github.com/kwea123/nerf_pl.\nSome functions are borrowed from DVR https://github.com/autonomousvision/differentiable_volumetric_rendering\nand PIFu https://github.com/shunsukesaito/PIFu\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsxyu%2Fpixel-nerf","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fsxyu%2Fpixel-nerf","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsxyu%2Fpixel-nerf/lists"}