{"id":15967098,"url":"https://github.com/rishit-dagli/nerf-us","last_synced_at":"2025-06-21T03:33:44.119Z","repository":{"id":251936135,"uuid":"797051473","full_name":"Rishit-dagli/NeRF-US","owner":"Rishit-dagli","description":"Official code for NeRF-US: Removing Ultrasound Imaging Artifacts from Neural Radiance Fields in the Wild","archived":false,"fork":false,"pushed_at":"2024-12-04T19:58:38.000Z","size":1395,"stargazers_count":19,"open_issues_count":2,"forks_count":3,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-06-14T02:46:24.420Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"https://rishitdagli.com/nerf-us","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/Rishit-dagli.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":"CITATION.cff","codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2024-05-07T05:27:19.000Z","updated_at":"2025-06-12T22:09:58.000Z","dependencies_parsed_at":null,"dependency_job_id":"5fc9278a-373f-435e-8a00-d884c8402cbf","html_url":"https://github.com/Rishit-dagli/NeRF-US","commit_stats":null,"previous_names":["rishit-dagli/nerf-us"],"tags_count":1,"template":false,"template_full_name":null,"purl":"pkg:github/Rishit-dagli/NeRF-US","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Rishit-dagli%2FNeRF-US","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Rishit-dagli%2FNeRF-US/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Rishit-dagli%2FNeRF-US/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Rishit-dagli%2FNeRF-US/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/Rishit-dagli","download_url":"https://codeload.github.com/Rishit-dagli/NeRF-US/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Rishit-dagli%2FNeRF-US/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":261056661,"owners_count":23103487,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["3d-reconstruction","computer-vision","nerf","nerf-us","novel-view-synthesis"],"created_at":"2024-10-07T18:20:39.993Z","updated_at":"2025-06-21T03:33:39.102Z","avatar_url":"https://github.com/Rishit-dagli.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"\u003cdiv align=\"center\"\u003e\n\u003ch2\u003eNeRF-US👥: Removing Ultrasound Imaging Artifacts from Neural Radiance Fields in the Wild\u003c/h2\u003e\n\n\u003ca href=\"https://www.cs.toronto.edu/~rishit\"\u003eRishit Dagli\u003c/a\u003e\u003csup\u003e1,2\u003c/sup\u003e, \u003ca href=\"https://hibiat.github.io/cv1/\"\u003eAtsuhiro Hibi\u003c/a\u003e\u003csup\u003e2,3,4\u003c/sup\u003e, \u003ca href=\"https://www.cs.toronto.edu/~rahulgk/\"\u003eRahul G. Krishnan\u003c/a\u003e\u003csup\u003e1,5\u003c/sup\u003e, \u003ca href=\"https://scholar.google.com/citations?user=c-KHCzEAAAAJ\u0026hl=en\u0026oi=ao\"\u003ePascal Tyrrell\u003c/a\u003e\u003csup\u003e2,4,6\u003c/sup\u003e\n\nDepartments of \u003csup\u003e1\u003c/sup\u003e Computer Science; \u003csup\u003e2\u003c/sup\u003e Medical Imaging, University of Toronto, Canada\u003cbr\u003e\n    \u003csup\u003e3\u003c/sup\u003e Division of Neurosurgery, St Michael's Hospital, Unity Health Toronto, Canada\u003cbr\u003e\n    \u003csup\u003e4\u003c/sup\u003e Institute of Medical Science; Departments of \u003csup\u003e5\u003c/sup\u003e Laboratory Medicine and Pathobiology; \u003csup\u003e6\u003c/sup\u003e Statistical Sciences, University of Toronto, Canada\n\n\u003ca href=\"https://twitter.com/intent/tweet?text=Wow:\u0026url=https%3A%2F%2Fgithub.com%2FRishit-dagli%2Fnerf-us\"\u003e\n  \u003cimg src=\"https://img.shields.io/twitter/url?style=social\u0026url=https%3A%2F%2Fgithub.com%2FRishit-dagli%2Fnerf-us\" alt=\"Twitter\"\u003e\n\u003c/a\u003e\n\u003ca href=\"https://arxiv.org/abs/2408.10258\"\u003e\u003cimg src='https://img.shields.io/badge/arXiv-NeRF--US-red' alt='Paper PDF'\u003e\u003c/a\u003e\n\u003ca href='https://rishitdagli.com/nerf-us'\u003e\u003cimg src='https://img.shields.io/badge/Project_Page-NeRF--US-green' alt='Project Page'\u003e\u003c/a\u003e\n\u003ca href='https://huggingface.co/papers/2408.10258'\u003e\u003cimg src='https://img.shields.io/badge/%F0%9F%A4%97%20Hugging%20Face-Paper-yellow'\u003e\u003c/a\u003e\n\u003ca href='https://huggingface.co/datasets/rishitdagli/us-in-the-wild'\u003e\u003cimg src='https://img.shields.io/badge/%F0%9F%A4%97%20Hugging%20Face-Dataset-yellow'\u003e\u003c/a\u003e\n\u003c/div\u003e\n\nThis work presents **NeRF-US**, a method to train NeRFs in-the-wild for sound fields like ultrasound imaging data. Check out our [website](https://rishitdagli.com/nerf-us/) to view some results of this work.\n\nThis codebase is forked from the awesome [Ultra-NeRF](https://github.com/magdalena-wysocki/ultra-nerf) and [Nerfbusters](https://github.com/ethanweber/nerfbusters) repository.\n\n![](assets/methods.png)\n\n## Installation\n\n1. First, install the pip package by running:\n\n```bash\npip install nerfus\n```\n\nor you could also install the package from source:\n\n```bash\ngit clone https://github.com/Rishit-Dagli/nerf-us\ncd nerf-us\npip install -e .\n```\n\n2. Now install the dependencies, if you use the `virtualenv` you could run:\n\n```bash\npip install -r requirements.txt\n```\n\nIf you use `conda` you could run:\n\n```bash\nconda env create -f environment.yml\nconda activate nerfus\n```\n\n3. Install Nerfstudio and dependencies. Installation guide can be found [install nerfstudio](https://docs.nerf.studio/quickstart/installation.html)\n\n\u003e We also use the branch [nerfbusters-changes](https://github.com/nerfstudio-project/nerfstudio/tree/nerfbusters-changes). You may have to run the viewer locally if you want full functionality.\n\n```bash\ncd path/to/nerfstudio\npip install -e .\npip install torch==1.13.1 torchvision functorch --extra-index-url https://download.pytorch.org/whl/cu117\npip install git+https://github.com/NVlabs/tiny-cuda-nn/#subdirectory=bindings/torch\n```\n\n4. Install `binvox` to voxelize cubes\n\n```bash\nmkdir bins\ncd bins\nwget -O binvox https://www.patrickmin.com/binvox/linux64/binvox?rnd=16811490753710\ncd ../\nchmod +x bins/binvox\n```\n\nCheck out the [Tips](#tips) section for tips on installing the requirements.\n\n## Overview of Codebase\n\nFor data preparation, the `cubes` directory contains modules for processing 3D data, including dataset handling (`datasets3D.py`), rendering (`render.py`), and visualization (`visualize3D.py`). The `data_modules` directory further supports data management with modules for 3D cubes and a general datamodule for the diffusion model.\n\nThe diffusion model is primarily implemented in the `models` directory, which includes the core model definition (`model.py`), U-Net architecture (`unet.py`), and related utilities. The `lightning` directory contains the training logic for the diffusion model, including loss functions (`dsds_loss.py`) and the trainer module (`nerfus_trainer.py`). The NeRF component is housed in the `nerf` directory, which includes experiment configurations, utility functions, and the main pipeline for NeRF-US (`nerfus_pipeline.py`).\n\n![](assets/diffusion.png)\n\n```\n.\n├── config (configuration files for the datasets and models)\n│   ├── shapenet.yaml (configuration file for the shapenet dataset)\n│   └── synthetic-knee.yaml (configuration file for the diffusion model)\n├── environment.yml (conda environment file)\n├── nerfus (main codebase)\n│   ├── bins\n│   │   └── binvox (binvox executable)\n│   ├── cubes (making cubes from 3D data)\n│   │   ├── __init__.py\n│   │   ├── binvox_rw.py\n│   │   ├── datasets3D.py\n│   │   ├── render.py\n│   │   ├── utils.py\n│   │   └── visualize3D.py\n│   ├── data\n│   ├── data_modules (data modules for cubes)\n│   │   ├── __init__.py\n│   │   ├── cubes3d.py\n│   │   └── datamodule.py (data module for diffusion model)\n│   ├── download_nerfus_dataset.py (script to download the diffusion model dataset)\n│   ├── lightning (training lightning modules for diffusion model)\n│   │   ├── __init__.py\n│   │   ├── dsds_loss.py (loss for diffusion model)\n│   │   └── nerfus_trainer.py (training code for diffusion model)\n│   ├── models (model definition for diffusion model)\n│   │   ├── __init__.py\n│   │   ├── fp16_util.py\n│   │   ├── model.py\n│   │   ├── nn.py\n│   │   └── unet.py\n│   ├── nerf (main codebase for the NeRF)\n│   │   ├── experiment_configs (configurations for the Nerfacto experiments)\n│   │   │   ├── __init__.py\n│   │   │   ├── nerfacto_experiments.py\n│   │   │   └── utils.py\n│   │   ├── nerfbusters_utils.py (utils for nerfbusters)\n│   │   ├── nerfus_config.py (nerfstudio method configurations for the NeRF-US)\n│   │   └── nerfus_pipeline.py (pipeline for NeRF-US)\n│   ├── run.py (training script for diffusion model)\n│   └── utils (utility functions for the NeRF training)\n│       ├── __init__.py\n│       ├── metrics.py\n│       ├── utils.py\n│       └── visualizations.py\n└── requirements.txt (requirements file we use)\n```\n\n## Usage\n\n### Training the Diffusion Model\n\nFirst, download either the synthetic knee cubes or the synthetic phantom cubes dataset:\n\n```\n.\n├── config\n│   ├── shapenet.yaml\n│   └── synthetic-knee.yaml\n├── nerfus\n│   ├── bins\n│   │   └── binvox\n│   ├── data\n│   |   ├── syn-knee\n|   |   └── syn-spi\n```\n\nWe can now train the 3D diffusion model using the following command:\n\n```bash\npython nerfus/run.py --config config/synthetic-knee.yaml --name synthetic-knee-experiment --pt\n```\n\nThis also automatically downloads Nerfbusters checkpoint on which we run adaptation.\n\n### Training the NeRF\n\nContrary to many other NeRF + Diffusion models we do not first train a NeRF and then continue training with the diffusion model as regularizee. Instead, we train we train it with the diffusion model from scratch.\n\nWe run the training using our method using Nerfstudio commands:\n\n```bash\nns-train nerfus --data path/to/data nerfstudio-data --vis wandb\n```\n\nFor our baselines, and experiments we directly use the Nerfstudio commands to train on the 10 individual datasets. For our ablation study, we do 3 ablations:\n\n1. for training without the border probability we just set the corresponding `lambda` to 0 (this could easily be made faster)\n2. for training without the scaterring density we just set the corresponding `lambda` to 0 (this could easily be made faster)\n3. for training without ultrasound rendering, we just use standard nerstudio commands\n\nWe can use standard Nerfstudio methods to run evaluations.\n\n## Tips\n\nWe share some tips on running the code and reproducing our results.\n\n### on installing required packages\n\n- Installing Nerfstudio especially on HPC systems can be tricky. We recommend installing `open3d`, and `tiny-cuda-nn` before installing Nerfstudio separately and install it from source. We also recommend building these packages on the same GPU you plan to run it on.\n- When you install PyTorch especially on HPC systems, you will often end up with atleast two version of CUDA: one which is installed when you install PyTorch and is not a full version of CUDA and the other which is in the system. We highly recommend manually installing the same version of CUDA as in the system that PyTorch automatically installs.\n- There are some parts of the code that do not run properly with PyTorch 2.x.\n- We use virtualenv and use the `requirements.txt` file to install the required packages. While, we provide a conda `environment.yml` (especially due to some Nerfstudio problems people might face), we have not tested this but expect it to work.\n\n### on compute\n\n- We have currently optimized the code for and run all of the experiments on a A100 - 80 GB GPU. However, we have also tested the code on a A100 - 40 GB GPU where the inference and evaluation seem to work pretty well.\n- In general, we would recommend a GPU with at least above 40 GB vRAM.\n- We would recommend having at least 32 GB CPU RAM for the code to work well.\n- While training the diffusion model, we recommend doing full-precision adaptation and not use FP-16.\n\n## Credits\n\nThis code base is built on top of, and thanks to them for maintaining the repositories:\n\n- [Ultra-NeRF](https://github.com/magdalena-wysocki/ultra-nerf)\n- [Nerfbusters](https://github.com/ethanweber/nerfbusters)\n- [Nerfstudio](https://docs.nerf.studio/)\n\n## Citation\n\nIf you find NeRF-US helpful, please consider citing:\n\n```bibtex\n@misc{dagli2024nerfusremovingultrasoundimaging,\n      title={NeRF-US: Removing Ultrasound Imaging Artifacts from Neural Radiance Fields in the Wild}, \n      author={Rishit Dagli and Atsuhiro Hibi and Rahul G. Krishnan and Pascal N. Tyrrell},\n      year={2024},\n      eprint={2408.10258},\n      archivePrefix={arXiv},\n      primaryClass={cs.CV},\n      url={https://arxiv.org/abs/2408.10258}, \n}\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Frishit-dagli%2Fnerf-us","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Frishit-dagli%2Fnerf-us","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Frishit-dagli%2Fnerf-us/lists"}