{"id":18842994,"url":"https://github.com/nv-tlabs/nksr","last_synced_at":"2025-04-05T00:09:57.991Z","repository":{"id":173884360,"uuid":"651169331","full_name":"nv-tlabs/NKSR","owner":"nv-tlabs","description":"[CVPR 2023 Highlight] Neural Kernel Surface Reconstruction","archived":false,"fork":false,"pushed_at":"2023-09-04T16:20:48.000Z","size":4951,"stargazers_count":601,"open_issues_count":15,"forks_count":34,"subscribers_count":22,"default_branch":"public","last_synced_at":"2023-12-16T14:38:19.979Z","etag":null,"topics":["3d-reconstruction","computer-vision","graphics","neural-kernel","point-cloud"],"latest_commit_sha":null,"homepage":"https://research.nvidia.com/labs/toronto-ai/NKSR","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"other","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/nv-tlabs.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE.txt","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null}},"created_at":"2023-06-08T16:44:59.000Z","updated_at":"2023-12-20T16:12:22.406Z","dependencies_parsed_at":null,"dependency_job_id":"97cfa89e-a5ee-4c95-89a4-c42e231a2371","html_url":"https://github.com/nv-tlabs/NKSR","commit_stats":{"total_commits":48,"total_committers":2,"mean_commits":24.0,"dds":"0.27083333333333337","last_synced_commit":"0d4e369b1ee641204d6e6d2b53c692fed6273ca5"},"previous_names":["nv-tlabs/nksr"],"tags_count":0,"template":null,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/nv-tlabs%2FNKSR","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/nv-tlabs%2FNKSR/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/nv-tlabs%2FNKSR/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/nv-tlabs%2FNKSR/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/nv-tlabs","download_url":"https://codeload.github.com/nv-tlabs/NKSR/tar.gz/refs/heads/public","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":247266565,"owners_count":20910836,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["3d-reconstruction","computer-vision","graphics","neural-kernel","point-cloud"],"created_at":"2024-11-08T02:56:18.072Z","updated_at":"2025-04-05T00:09:57.962Z","avatar_url":"https://github.com/nv-tlabs.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Neural Kernel Surface Reconstruction\n\n![NKSR](assets/teaser.png)\n\n**Neural Kernel Surface Reconstruction**\u003cbr\u003e\n[Jiahui Huang](https://huangjh-pub.github.io/),\n[Zan Gojcic](https://zgojcic.github.io/),\n[Matan Atzmon](https://matanatz.github.io/),\n[Or Litany](https://orlitany.github.io/), \n[Sanja Fidler](https://www.cs.toronto.edu/~fidler/),\n[Francis Williams](https://www.fwilliams.info/) \u003cbr\u003e\n**[Paper](https://arxiv.org/abs/2305.19590), [Project Page](https://research.nvidia.com/labs/toronto-ai/NKSR/)**\n\nAbstract: *We present a novel method for reconstructing a 3D implicit surface from a large-scale, sparse, and noisy point cloud. \nOur approach builds upon the recently introduced [Neural Kernel Fields (NKF)](https://nv-tlabs.github.io/nkf/) representation. \nIt enjoys similar generalization capabilities to NKF, while simultaneously addressing its main limitations: \n(a) We can scale to large scenes through compactly supported kernel functions, which enable the use of memory-efficient sparse linear solvers. \n(b) We are robust to noise, through a gradient fitting solve. \n(c) We minimize training requirements, enabling us to learn from any dataset of dense oriented points, and even mix training data consisting of objects and scenes at different scales. \nOur method is capable of reconstructing millions of points in a few seconds, and handling very large scenes in an out-of-core fashion. \nWe achieve state-of-the-art results on reconstruction benchmarks consisting of single objects, indoor scenes, and outdoor scenes.*\n\nFor business inquiries, please visit our website and submit the form: [NVIDIA Research Licensing](https://www.nvidia.com/en-us/research/inquiries/)\n\n## News\n\n- 2023-06-24 🔥: Want to build a mesh from an iPhone scan? Our friend [Phong Nguyen Ha](https://github.com/phongnhhn92) kindly shared his tutorial for doing this! Check it out [here](https://drive.google.com/drive/folders/1RESJ6w6DND4mh8V-YY0a3rvSfSR0dQdD?usp=sharing).\n- 2023-06-08: Code released!\n\n## Environment setup\n\nWe recommend using the latest Python and PyTorch to run our algorithm. To install all dependencies using [conda](https://www.anaconda.com/):\n\n```bash\n# Clone the repository\ngit clone git@github.com:nv-tlabs/nksr.git\ncd nksr\n\n# Create conda environment\nconda env create\n\n# Activate it\nconda activate nksr\n\n# Install NKSR\npip install nksr -f https://nksr.huangjh.tech/whl/torch-2.0.0+cu118.html\n```\n\n\u003e For docker users, we suggest using a base image from [nvidia/cuda](https://hub.docker.com/r/nvidia/cuda) with tag `11.8.0-cudnn8-devel-ubuntu22.04`, and applying the above conda setup over it.\n\n## Testing NKSR on your own data\n\nWe have tested our algorithm on multiple different spatial scales. It can reconstruct scenes spanning kilometers with millions of points+ on an RTX 3090 GPU.\nTo use our `kitchen-sink` model (released under CC-BY-SA 4.0 license), the following code snippet suffices:\n\n```python\nimport torch\nimport nksr\n\nbunny_geom = load_bunny_example()\n\ninput_xyz = torch.from_numpy(np.asarray(bunny_geom.points)).float().to(device)\ninput_normal = torch.from_numpy(np.asarray(bunny_geom.normals)).float().to(device)\n\nreconstructor = nksr.Reconstructor(device)\nfield = reconstructor.reconstruct(input_xyz, input_normal, detail_level=1.0)\nmesh = field.extract_dual_mesh(mise_iter=1)\n```\n\nWe have prepared detailed instructions about data preparation and different example usages at [NKSR Documentation Page](NKSR-USAGE.md).\n\n## Reproducing results from the paper\n\nOur training and inference system is based on the [Zeus Deep Learning](ZEUS_DL.md) infrastructure, supporting both tensorboard and wandb (recommended) as loggers. To config Zeus, copy the default yaml file and modify the related paths:\n\n```bash\ncp configs/default/zeus.yaml zeus_config.yaml\n```\n\nModify the contents of `zeus_config.yaml` as needed to include your `wandb` account name and checkpoint/test results save directory.\n\n### Training\n\nData download links:\n- ShapeNet: Data is available [here](https://s3.eu-central-1.amazonaws.com/avg-projects/occupancy_networks/data/dataset_small_v1.1.zip) (taken from [Occupancy Networks](https://github.com/autonomousvision/occupancy_networks)). Put the extracted `onet` folder under `../data/shapenet`.\n- Points2Surf: Data is available [here](https://drive.google.com/file/d/1kWYbQpUx7MHvkdhQfO3D8qyv9Y_hVXrr/view?usp=sharing). We've used blensor to re-generate all the data following the original script in order to obtain input normals.\n- CARLA: Data is available [here](https://drive.google.com/file/d/1BFwExw7SRJaqHJ98pqqnR-k6g8XYMAqq/view?usp=sharing).\n\nThe main training script is `train.py`. We provide different config files for the different datasets we've benchmarked in our paper:\n```bash\n# ShapeNet small noise 1K input\npython train.py configs/shapenet/train_1k_perfect.yaml\n# ShapeNet medium noise 3K input\npython train.py configs/shapenet/train_3k_noise.yaml\n# ShapeNet big noise 3K input\npython train.py configs/shapenet/train_3k_noiser.yaml\n# Points2Surf dataset noisy input\npython train.py configs/points2surf/train.yaml\n# CARLA dataset\npython train.py configs/carla/train.yaml\n```\n\nIn addition, you can manually specify different training settings to obtain models that suit your needs. Common flags include:\n- `--wname`: Additional experiment name to specify for wandb logger.\n- `--voxel_size`: Size of the finest level of voxels.\n- `--feature`: Additional feature to the encoder, `none` for nothing, `normal` (default) for normal input, `sensor` for sensor direction input.\n- `--tree_depth`: Depth of the sparse feature hierarchy. Default is 4.\n\n### Inference\n\nYou can either infer using your own trained models or our pre-trained checkpoints.\n\n```bash\n# From pre-trained checkpoints\npython test.py configs/shapenet/train_3k_noise.yaml --url https://nksr.huangjh.tech/snet-n3k-wnormal.pth --exec udf.enabled=False\npython test.py configs/points2surf/train.yaml --url https://nksr.huangjh.tech/p2s.pth --include configs/points2surf/data_abc_test.yaml\npython test.py configs/carla/train.yaml --url https://nksr.huangjh.tech/carla.pth  --include configs/carla/data_no_patch.yaml\n\n# From your own trained models\npython test.py none --ckpt wdb:\u003cWANDB_USER_NAME\u003e/\u003cWANDB_PROJECT\u003e/\u003cWANDB_RUN_ID\u003e\n```\n\nUseful flags for `test.py` include:\n- `-v` or `--visualize`: visualize the results during the test process.\n- `--test_print_metrics`: print the metrics during testing.\n- `--test_n_upsample`: level of geometric upsample. Recommended value is 4.\n\n## License\n\nCopyright \u0026copy; 2023, NVIDIA Corporation \u0026 affiliates. All rights reserved.\nThis work is made available under the [Nvidia Source Code License](LICENSE.txt).\n\n## Related Works\n\nNKSR is highly based on the following existing works:\n\n- Williams et al. 2021. [Neural Fields as Learnable Kernels for 3D Reconstruction](https://nv-tlabs.github.io/nkf/).\n- Huang et al. 2022. [A Neural Galerkin solver for Accurate Surface Reconstruction](https://github.com/huangjh-pub/neural-galerkin).\n\n## Citation\n\n```bibtex\n@inproceedings{huang2023nksr,\n  title={Neural Kernel Surface Reconstruction},\n  author={Huang, Jiahui and Gojcic, Zan and Atzmon, Matan and Litany, Or and Fidler, Sanja and Williams, Francis},\n  booktitle={Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition},\n  pages={4369--4379},\n  year={2023}\n}\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fnv-tlabs%2Fnksr","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fnv-tlabs%2Fnksr","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fnv-tlabs%2Fnksr/lists"}