{"id":18614345,"url":"https://github.com/Crishawy/NeXT","last_synced_at":"2025-04-11T00:30:33.678Z","repository":{"id":46878266,"uuid":"490730728","full_name":"Crishawy/NeXT","owner":"Crishawy","description":null,"archived":false,"fork":false,"pushed_at":"2022-07-20T06:27:40.000Z","size":703,"stargazers_count":18,"open_issues_count":1,"forks_count":0,"subscribers_count":2,"default_branch":"main","last_synced_at":"2024-11-07T03:31:06.780Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/Crishawy.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2022-05-10T14:23:16.000Z","updated_at":"2024-01-23T17:16:28.000Z","dependencies_parsed_at":"2022-09-18T17:13:00.481Z","dependency_job_id":null,"html_url":"https://github.com/Crishawy/NeXT","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Crishawy%2FNeXT","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Crishawy%2FNeXT/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Crishawy%2FNeXT/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Crishawy%2FNeXT/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/Crishawy","download_url":"https://codeload.github.com/Crishawy/NeXT/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248322208,"owners_count":21084333,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-07T03:25:54.961Z","updated_at":"2025-04-11T00:30:33.093Z","avatar_url":"https://github.com/Crishawy.png","language":"Python","funding_links":[],"categories":["Papers"],"sub_categories":["NeRF"],"readme":"## NeXT\nThis repository contains the code release for NeXT: Towards High Quality Neural Radiance Fields via Multi-Skip Transformer.\nThis implementation is based on [JaxNeRF](https://github.com/google-research/google-research/tree/master/jaxnerf) and \n[Mip-NeRF](https://github.com/google/mipnerf). NeXT can be easily incorporated into NeRF and Mip-NeRF methods to serve as \nthe query network, achieving higher quality rendering results.\n![Framework](figs/framework.jpg \"model framework\")\n\n## Abstract\nNeural Radiance Fields (NeRF) methods show impressive performance for novel view synthesis by representing a scene via a neural network. However, most existing NeRF based methods, including its variants, treat each sample point individually as input, while ignoring\nthe inherent relationships between adjacent sample points from the corresponding rays, thus hindering the reconstruction\nperformance. To address this issue, we explore a brand new scheme, namely NeXT, introducing a multi-skip transformer to \ncapture the rich relationships between various sample points in a ray-level query. Specifically, ray tokenization is \nproposed to represent each ray as a sequence of point embeddings which is taken as input of our proposed NeXT. \nIn this way, relationships between sample points are captured via the built-in self-attention mechanism \nto promote the reconstruction. Besides, our proposed NeXT can be easily combined with other NeRF based methods to \nimprove their rendering quality. Extensive experiments conducted on three datasets demonstrate that NeXT \nsignificantly outperforms all previous state-of-the-art work by a large margin. In particular, the proposed NeXT \nsurpasses the strong NeRF baseline by 2.74 PSNR on Blender dataset.\n\n## Installation\nFollowing [JaxNeRF](https://github.com/google-research/google-research/tree/master/jaxnerf).\n\nWe recommend using [Anaconda](https://www.anaconda.com/products/individual) to set\nup the environment. Run the following commands:\n\n```\n# Clone the repo\nsvn export https://github.com/google-research/google-research/trunk/jaxnerf\n# Create a conda environment, note you can use python 3.6-3.8 as\n# one of the dependencies (TensorFlow) hasn't supported python 3.9 yet.\nconda create --name jaxnerf python=3.6.12; conda activate jaxnerf\n# Prepare pip\nconda install pip; pip install --upgrade pip\n# Install requirements\npip install -r jaxnerf/requirements.txt\n# [Optional] Install GPU and TPU support for Jax\n# Remember to change cuda101 to your CUDA version, e.g. cuda110 for CUDA 11.0.\npip install --upgrade jax jaxlib==0.1.57+cuda101 -f https://storage.googleapis.com/jax-releases/jax_releases.html\n```\n\nThen, you'll need to download the datasets\nfrom the [NeRF official Google Drive](https://drive.google.com/drive/folders/128yBriW1IG_3NJ5Rp7APSTZsJqdJdfc1).\nPlease download the `nerf_synthetic.zip` and unzip to `jaxnerf/data` and `mipnerf/data`.\n\n### Generate multiscale dataset\nYou can generate the multiscale dataset by running the following command,\n```\ncd mipnerf\npython scripts/convert_blender_data.py --blenderdir ./data/nerf_synthetic --outdir ./data/nerf_synthetic_multiscale\n```\n\n## Running\n\nExample scripts for training NeXT on Blender and Multi-scale Bender dataset can be found in `jaxnerf/scripts` and `mipnerf/scrips`. \nYou'll need to change the paths to point to wherever the datasets are located.\n\nFor training NeXT-S/NeXT-B/NeXT-L of one scene on Blender dataset, run the following scrips:\n```\ncd jaxnerf\n# training\nsh scripts/train_next_blender.sh chair next_s   # next_s/next_b/next_l\n# evaluation\nsh scripts/eval_next_blender.sh chair next_s   # next_s/next_b/next_l\n```\n\nFor training NeXT-S/NeXT-B/NeXT-L of one scene on Multi-scale dataset, run the following scrips:\n```\ncd mipnerf\n# training\nsh scripts/train_next_multiblender.sh chair next_s   # next_s/next_b/next_l\n# evaluation\nsh scripts/eval_next_multiblender.sh chair next_s   # next_s/next_b/next_l\n```\n\n## Results\n### Blender\n|       | #Params |   Chair   |   Drums   |   Ficus   |   Hotdog  | Lego | Materials | Mic | Ship | Avg. |\n|---------   |:-------:|:---------:|:---------:|:---------:|:---------:|:---------:|:---------:|:---------:|:---------:|:---------:|\n| NeRF (JAX) | 1,191K|   34.06   |   25.13   |   30.48   |   36.87   | 33.33 | 29.94 | 34.66 | 28.77 | 31.66 |   \n| NeXT-S | 1,232K | 33.75 | 25.34 | 32.62 | 37.42 | 34.52 | 32.09 | 33.74 | 29.25 | 32.34 |\n| NeXT-B | 2,152K | 34.70 | 25.79 | 33.77 | 38.10 | 35.67 | 32.48 | 34.46 | 30.07 | 33.13 |\n| NeXT-L | 4,062K | **36.05** | **26.49** | **35.67** | **38.46** | **37.39** | **34.16** | **35.96** | **30.73**| **34.40** |\n\n### Multi-scale Blender\n|       | #Params |   Full Res.   |   1/2 Res.   |   1/4 Res.   |   1/8 Res.  |\n|---------   |:-------:|:---------:|:---------:|:---------:|:---------:|\n| NeRF (JAX) | 1,191K|   31.20   |   30.65   |   26.25   |   22.53   |   \n| Mip-NeRF   |612K| 32.63 | 34.34 | 35.47 | 35.60 | \n| NeXT-S | 616K | 32.18 | 34.32 | 36.43 | 37.57 |\n| NeXT-B | 1,076K | 32.92 | 35.06 | 36.99 | 38.05 |\n| NeXT-L | 2,031K | **34.38** | **36.47** | **38.19** | **39.29** |\n\n## Acknowledgements\nThanks for the open-source [JaxNeRF](https://github.com/google-research/google-research/tree/master/jaxnerf)\nand [Mip-NeRF](https://github.com/google/mipnerf).","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FCrishawy%2FNeXT","html_url":"https://awesome.ecosyste.ms/projects/github.com%2FCrishawy%2FNeXT","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FCrishawy%2FNeXT/lists"}