{"id":13563849,"url":"https://github.com/apple/ml-gsn","last_synced_at":"2025-04-09T20:13:24.833Z","repository":{"id":39895317,"uuid":"354895953","full_name":"apple/ml-gsn","owner":"apple","description":null,"archived":false,"fork":false,"pushed_at":"2021-09-24T08:13:30.000Z","size":79900,"stargazers_count":301,"open_issues_count":9,"forks_count":42,"subscribers_count":21,"default_branch":"main","last_synced_at":"2025-04-09T20:13:06.788Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"other","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/apple.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":"CODE_OF_CONDUCT.md","threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2021-04-05T16:19:44.000Z","updated_at":"2025-03-24T11:13:02.000Z","dependencies_parsed_at":"2022-09-05T03:00:34.489Z","dependency_job_id":null,"html_url":"https://github.com/apple/ml-gsn","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/apple%2Fml-gsn","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/apple%2Fml-gsn/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/apple%2Fml-gsn/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/apple%2Fml-gsn/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/apple","download_url":"https://codeload.github.com/apple/ml-gsn/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248103872,"owners_count":21048245,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-08-01T13:01:23.915Z","updated_at":"2025-04-09T20:13:24.796Z","avatar_url":"https://github.com/apple.png","language":"Python","funding_links":[],"categories":["Python","Papers"],"sub_categories":["NeRF Related Tasks"],"readme":"## Generative Scene Networks (GSN) - Official PyTorch Implementation\n**Unconstrained Scene Generation with Locally Conditioned Radiance Fields, ICCV 2021**\u003cbr\u003e\nTerrance DeVries, Miguel Angel Bautista, Nitish Srivastava, Graham W. Taylor, Joshua M. Susskind\u003cbr\u003e\n\n### [Project Page](https://apple.github.io/ml-gsn/) | [Paper](https://arxiv.org/abs/2104.00670) | [Data](#datasets)\n\n## Requirements\nThis code was tested with Python 3.6 and CUDA 11.1.1, and uses Pytorch Lightning. A suitable conda environment named `gsn` can be created and activated with:\n```\nconda env create -f environment.yaml python=3.6\nconda activate gsn\n```\nIf you do not already have CUDA installed, you can do so with:\n```\nwget https://developer.download.nvidia.com/compute/cuda/11.1.1/local_installers/cuda_11.1.1_455.32.00_linux.run\nsh cuda_11.1.1_455.32.00_linux.run --toolkit --silent --override\nrm cuda_11.1.1_455.32.00_linux.run\n```\nCustom CUDA kernels may not work with older versions of CUDA. This code will revert to a native PyTorch implementation if the CUDA version is incompatible, although runtime may be ~25% slower.\n\n## Datasets\nWe provide camera trajectories for two datasets that we used to trained our model: Vizdoom and Replica. These datasets are composed of different sequences with corresponding rgb+depth frames and camera parameters (extrinsiscs and intrinsics).\n\nDataset | Size | Download Link\n--- | :---: | :---:\nVizdoom | 2.4 GB | [download](\u003chttps://docs-assets.developer.apple.com/ml-research/datasets/gsn/vizdoom.zip\u003e)\nReplica | 11.0 GB | [download](\u003chttps://docs-assets.developer.apple.com/ml-research/datasets/gsn/replica.zip\u003e)\n\nDatasets can be downloaded by running the following scripts:  \n**VizDoom**\u003cbr\u003e\n```\npython scripts/download_vizdoom.py\n```\n**Replica**\u003cbr\u003e\n```\npython scripts/download_replica.py\n```\n\n## Interactive exploration demo\nWe provide a [Jupyter notebook](notebooks/walkthrough_demo.ipynb) that allows for interactive exploration of scenes generated from a pre-trained model. Use the WASD keys to freely navigate through the scene! Once you are done, the notebook interpolates the camera path to render a continuous trajectory. Note: You need to download the Replica dataset before via this [script](scripts/download_replica.py) before running the notebook.\n\nExplore scene with WASD to set keypoints | Rendered trajectory\n:---: | :---:\n\u003cimg src=\"./assets/keyframes.gif\" width=256px\u003e | \u003cimg src=\"./assets/camera_trajectory.gif\" width=256px\u003e\n\n## Training models\nDownload the training dataset (if you have not done so already) and begin training with the following commands:  \n**VizDoom**\u003cbr\u003e\n```\nbash scripts/launch_gsn_vizdoom_64x64.sh\n```\n\n**Replica**\u003cbr\u003e\n```\nbash scripts/launch_gsn_replica_64x64.sh\n```\n\nTraining takes about 3 days to reach 500k iterations with a batch size of 32 on two A100 GPUs.\n\n## Pre-trained models\nWe provide pre-trained models for GSN to replicate our experimental results. In particular, we provide models for the Vizdoom dataset trained at 64x64 resolution, and for Replica dataset trained at 64x64 and 128x128. Note that either model can be rendered at higher resolutions than native resolution used durinig training by changing the intrinsic camera parameters during inference.\n\nDataset | Train Resolution | FID (5k) | Download Link\n--- | :---: | :---: | :---: \nVizdoom | 64x64 | 35.9 | [download](\u003chttps://docs-assets.developer.apple.com/ml-research/models/gsn/vizdoom_64x64.ckpt\u003e)\nReplica | 64x64 | 41.5 | [download](\u003chttps://docs-assets.developer.apple.com/ml-research/models/gsn/replica_64x64.ckpt\u003e)\nReplica | 128x128 | 43.4 | [download](\u003chttps://docs-assets.developer.apple.com/ml-research/models/gsn/replica_128x128.ckpt\u003e)\n\n### Evaluating pre-trained models\nThe evaluation script requires the [training set](#datasets) to run. Download it first if you have not yet done so.\nDownload and run evaluation for pre-trained models with the following commands:  \n**VizDoom**\u003cbr\u003e\n```\nbash scripts/eval_vizdoom_64x_64_pretrained.sh\n```\n**Replica**\u003cbr\u003e\n```\nbash scripts/eval_replica_64x_64_pretrained.sh\n```\nRunning evaluation will compute the FID score and save sample sheets in the log directory.\n\n## Citation\n```\n@article{devries2021unconstrained,\n    title={Unconstrained Scene Generation with Locally Conditioned Radiance Fields},\n    author={Terrance DeVries and Miguel Angel Bautista and \n            Nitish Srivastava and Graham W. Taylor and \n            Joshua M. Susskind},\n    journal={arXiv},\n    year={2021}\n}\n```\n## License\nThis sample code is released under the [LICENSE](LICENSE) terms.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fapple%2Fml-gsn","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fapple%2Fml-gsn","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fapple%2Fml-gsn/lists"}