{"id":13603288,"url":"https://github.com/facebookresearch/hyperreel","last_synced_at":"2025-05-15T23:02:36.029Z","repository":{"id":65324852,"uuid":"579200633","full_name":"facebookresearch/hyperreel","owner":"facebookresearch","description":"Code release for HyperReel: High-Fidelity 6-DoF Video with Ray-Conditioned Sampling","archived":false,"fork":false,"pushed_at":"2025-02-12T17:10:03.000Z","size":16882,"stargazers_count":476,"open_issues_count":10,"forks_count":34,"subscribers_count":21,"default_branch":"main","last_synced_at":"2025-04-08T17:16:33.538Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/facebookresearch.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":"CODE_OF_CONDUCT.md","threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null}},"created_at":"2022-12-16T23:11:56.000Z","updated_at":"2025-04-03T07:39:12.000Z","dependencies_parsed_at":"2024-01-15T18:57:32.404Z","dependency_job_id":null,"html_url":"https://github.com/facebookresearch/hyperreel","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/facebookresearch%2Fhyperreel","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/facebookresearch%2Fhyperreel/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/facebookresearch%2Fhyperreel/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/facebookresearch%2Fhyperreel/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/facebookresearch","download_url":"https://codeload.github.com/facebookresearch/hyperreel/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":254436932,"owners_count":22070944,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-08-01T18:10:50.205Z","updated_at":"2025-05-15T23:02:35.894Z","avatar_url":"https://github.com/facebookresearch.png","language":"Python","funding_links":[],"categories":["2023","Python"],"sub_categories":["Real"],"readme":"# HyperReel\n\n## [Website](https://hyperreel.github.io/) |  [Paper](https://arxiv.org/abs/2301.02238) | [Video](https://hyperreel.github.io/figures/hyperreel.mp4)\n\n![pipeline](images/pipeline.png)\n\nThe code of HyperReel is available under the MIT license, as it draws from the following projects, which are also licensed under the MIT license: [nerf_pl](https://github.com/kwea123/nerf_pl), [TensoRF](https://github.com/apchenstu/TensoRF), and [torch-ngp](https://github.com/ashawkey/torch-ngp). Licenses for all of these projects can be found in the `licenses/` folder.\n\n\n## Table of contents\n-----\n  * [Installation](#Installation)\n  * [Dynamic Datasets](#dynamic-datasets)\n  * [Static Datasets](#static-datasets)\n  * [Running the Code on Dynamic Scenes](#running-the-code-on-dynamic-scenes)\n  * [Running the Code on Static Scenes](#running-the-code-on-static-scenes)\n  * [Running the Code with Custom Parameters](#running-the-code-with-custom-parameters)\n  * [Real-Time Viewer](#real-time-viewer)\n  * [Citation](#Citation)\n------\n\n\n# Installation\n\nTo install all required python dependences run\n\n```bash\nconda env create -f environment.yml\n```\n\nNote that we tested the HyperReel codebase on a machine running Ubuntu 20.04, with an NVIDIA 3090 RTX GPU, CUDA version 11.8, and 128 GB of RAM.\n\n# Dynamic Datasets\n\nBy default, we assume that:\n\n1. All datasets are located in the `~/data` folder (specified by the `experiment.params.data_dir` argument)\n2. With the subdirectory for each individual dataset specified by the `experiment.dataset.data_subdir` argument (e.g., see `conf/experiment/params/local.yaml` and `conf/experiment/dataset/technicolor.yaml`).\n\n## Technicolor\n\nPlease reach out to the authors of [Dataset and Pipeline for Multi-View Light-Field Video](https://ieeexplore.ieee.org/document/8014955) for access to the Technicolor dataset. We use the \"undistorted\" variants of the following sequences:\n\n* Birthday (frames 150-200)\n* Fabien (frames 50-100)\n* Painter (frames 100-150)\n* Theater (frames 50-100)\n* Trains (frames 150-200)\n\n## Google Immersive\n\nDownload the Google Immersive sequences from their [release page](https://github.com/augmentedperception/deepview_video_dataset). As an example, in order to download the flames sequence, run:\n\n```\nwget https://storage.googleapis.com/deepview_video_raw_data/02_Flames.zip\n```\n\n## Neural 3D Video\n\nDownload the Neural 3D video sequences from their [release page](https://github.com/facebookresearch/Neural_3D_Video/releases/tag/v1.0). As an example, in order to download the Flame Steak sequence, run:\n\n```\nwget https://github.com/facebookresearch/Neural_3D_Video/releases/download/v1.0/flame_steak.zip\n```\n\n# Static Datasets\n\n## DoNeRF\n\nThe DoNeRF dataset can be found [here](https://repository.tugraz.at/records/jjs3x-4f133).\n\n## LLFF\n\nThe LLFF dataset can be found [here](https://drive.google.com/drive/folders/128yBriW1IG_3NJ5Rp7APSTZsJqdJdfc1).\n\n## Shiny\n\nThe Shiny dataset can be found [here](https://vistec-my.sharepoint.com/:f:/g/personal/pakkapon_p_s19_vistec_ac_th/EnIUhsRVJOdNsZ_4smdhye0B8z0VlxqOR35IR3bp0uGupQ?e=TsaQgM).\n\n## Spaces\n\nThe Spaces dataset can be found [here](https://github.com/augmentedperception/spaces_dataset).\n\n## Stanford\n\nThe Stanford dataset can be found [here](http://lightfield.stanford.edu/lfs.html).\n\n# Running the Code on Dynamic Scenes\n\nBy default...\n\n1. Checkpoints are written to the `~/checkpoints` folder (specified by the `experiment.params.ckpt_dir` argument)\n2. Logs are written to the `~/logs` folder (specified bt the `experiment.params.log_dir` argument).\n\nNote that it can take a few minutes to load all of the training data into memory for dynamic scenes.\n\n## Technicolor\n\nIn order to train HyperReel on a 50 frame subset of a scene from the Technicolor dataset, run:\n\n```\nbash scripts/run_one_technicolor.sh \u003cgpu_to_use\u003e \u003cscene\u003e \u003cstart_frame\u003e\n```\n\nBy default, the above command will hold-out the central camera. To train a model using all available cameras, run\n\n```\nbash scripts/run_one_technicolor_no_holdout.sh \u003cgpu_to_use\u003e \u003cscene\u003e \u003cstart_frame\u003e\n```\n\nThis will also automatically create validation images and (spiral) validation videos in the log folder for the experiment. From a trained model, you can also render a video sequence with:\n\n```\nbash scripts/render_one_technicolor.sh \u003cgpu_to_use\u003e \u003cscene\u003e \u003cstart_frame\u003e\n```\n\n## Google Immersive\n\nIn order to train HyperReel on a 50 frame subset of a scene from the Google Immersive dataset, run:\n\n```\nbash scripts/run_one_immersive.sh \u003cgpu_to_use\u003e \u003cscene\u003e \u003cstart_frame\u003e\n```\n\nTo reproduce the quantitative results that we report in our paper, you should train a model for *every* 50 frame subset in a video sequence. So, for a 10 second, 300 frame video, you should train models for `start_frame = 0, 50, ..., 250`. Further, each model should be trained for about 1.5 hours (9 hours total for a 10 second, 300 frame video). \n\nBy default, the above command will hold-out the central camera. To train a model using all available cameras, run\n\n```\nbash scripts/run_one_immersive_no_holdout.sh \u003cgpu_to_use\u003e \u003cscene\u003e \u003cstart_frame\u003e\n```\n\n## Neural 3D\n\nIn order to train HyperReel on a 50 frame subset of a scene from the Neural 3D Video dataset, run:\n\n```\nbash scripts/run_one_n3d.sh \u003cgpu_to_use\u003e \u003cscene\u003e \u003cstart_frame\u003e\n```\n\nTo reproduce the quantitative results that we report in our paper, you should train a model for *every* 50 frame subset in a video sequence. So, for a 10 second, 300 frame video, you should train models for `start_frame = 0, 50, ..., 250`. Further, each model should be trained for about 1.5 hours (9 hours total for a 10 second, 300 frame video). \n\nBy default, the above command will hold-out the central camera. To train a model using all available cameras, run\n\n```\nbash scripts/run_one_n3d_no_holdout.sh \u003cgpu_to_use\u003e \u003cscene\u003e \u003cstart_frame\u003e\n```\n\n# Running the Code on Static Scenes\n\n## DoNeRF\n\nIn order to train HyperReel on a scene from the DoNeRF dataset, run:\n\n```\nbash scripts/run_one_donerf_sphere.sh \u003cgpu_to_use\u003e \u003cscene\u003e\n```\n\n## LLFF\n\nIn order to train HyperReel on a scene from the LLFF dataset, run:\n\n```\nbash scripts/run_one_llff.sh \u003cgpu_to_use\u003e \u003cscene\u003e\n```\n\n## Shiny\n\nIn order to train HyperReel on the *CD* and *Lab* sequences from the Shiny dataset, run:\n\n```\nbash scripts/run_one_shiny_dense.sh \u003cgpu_to_use\u003e \u003cscene\u003e\n```\n\n# Running the Code with Custom Parameters\n\nThe general syntax for training a model is:\n\n```\npython main.py experiment/dataset=\u003cdataset_config\u003e \\\n    experiment/training=\u003ctraining_config\u003e \\\n    experiment/model=\u003cmodel_config\u003e \\\n    experiment.dataset.collection=\u003cscene_name\u003e \\\n    +experiment/regularizers/tensorf=tv_4000\n```\n\nWhere\n\n1. `\u003cdataset_config\u003e` specifies the dataset config file, located in `conf/experiment/dataset`\n2. `\u003ctraining_config\u003e` specifies the training config file, located in `conf/experiment/training`\n3. `\u003cmodel_config\u003e` specifies the model config file, located in `conf/experiment/model`\n4. `\u003cscene_name\u003e` specifies the scene name within the dataset\n\n## Regularizer syntax\n\nThe line `+experiment/regularizers/tensorf=tv_4000` adds total variation and L1 regularizers on the volume tensor components, with configuration located in `conf/experiment/regularizers/tensorf/tv_4000`.\n\n# Real-Time Viewer\n\nOnce you have trained a HyperReel model, you can make use of the ```scripts/demo_*``` scripts in order to launch the real-time viewer.\n\nFor example, to run the real-time viewer on a scene from the technicolor dataset, run:\n\n```\nbash scripts/demo_technicolor.sh \u003cgpu_to_use\u003e \u003cscene\u003e \u003cstart_frame\u003e\n```\n\nFor static scenes, you can omit the start frame argument. For example:\n\n```\nbash scripts/demo_shiny_dense.sh \u003cgpu_to_use\u003e \u003cscene\u003e\n```\n\nHere are a couple of examples of the demo running on a workstation with a 3090 RTX GPU.\n\nhttps://user-images.githubusercontent.com/2993881/208320757-439928a1-4d35-4625-afe7-56e7c1eaa57c.mov\n\nhttps://user-images.githubusercontent.com/2993881/208320798-445fec87-54f1-4335-b345-804a7b2183fe.mov\n\n\n\n# Citation\n\n```bibtex\n@inproceedings{attal2023hyperreel,\n  title     = {{HyperReel}: {H}igh-Fidelity {6-DoF} Video with Ray-Conditioned Sampling},\n  author    = {Attal, Benjamin and Huang, Jia-Bin and Richardt, Christian and\n               Zollhoefer, Michael and Kopf, Johannes and O'Toole, Matthew and Kim, Changil},\n  booktitle = {CVPR},\n  year      = {2023}\n}\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ffacebookresearch%2Fhyperreel","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Ffacebookresearch%2Fhyperreel","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ffacebookresearch%2Fhyperreel/lists"}