{"id":20444824,"url":"https://github.com/xxlong0/neuraludf","last_synced_at":"2025-04-07T13:08:23.013Z","repository":{"id":65515831,"uuid":"568677300","full_name":"xxlong0/NeuralUDF","owner":"xxlong0","description":"Volume rendering based surface reconstruction using Unsigned Distance Fields","archived":false,"fork":false,"pushed_at":"2024-02-03T14:57:42.000Z","size":26325,"stargazers_count":286,"open_issues_count":9,"forks_count":10,"subscribers_count":29,"default_branch":"main","last_synced_at":"2025-03-31T11:06:37.927Z","etag":null,"topics":["nerf","surface-reconstruction","udf","volume-rendering"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/xxlong0.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2022-11-21T07:09:05.000Z","updated_at":"2025-03-17T03:59:23.000Z","dependencies_parsed_at":"2024-02-03T15:49:39.653Z","dependency_job_id":null,"html_url":"https://github.com/xxlong0/NeuralUDF","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/xxlong0%2FNeuralUDF","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/xxlong0%2FNeuralUDF/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/xxlong0%2FNeuralUDF/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/xxlong0%2FNeuralUDF/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/xxlong0","download_url":"https://codeload.github.com/xxlong0/NeuralUDF/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":247657281,"owners_count":20974345,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["nerf","surface-reconstruction","udf","volume-rendering"],"created_at":"2024-11-15T10:09:15.852Z","updated_at":"2025-04-07T13:08:22.992Z","avatar_url":"https://github.com/xxlong0.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# NeuralUDF: Learning Unsigned Distance Fields for Multi-view Reconstruction of Surfaces with Arbitrary Topologies (CVPR2023)\n\n## [Project Page](https://www.xxlong.site/NeuralUDF/) | [Paper](https://arxiv.org/abs/2211.14173) \n\n\n![](./docs/images/teaser.png)\n\n## Introduction\nWe present a novel method, called NeuralUDF, for reconstructing surfaces with arbitrary topologies from 2D images via volume rendering.\nHowever, these methods are limited to objects with closed surfaces since they adopt Signed Distance Function (SDF)\nas surface representation which requires the target shape to be divided into inside and outside.\nIn this paper, we propose to represent surfaces as the Unsigned Distance Function (UDF) and\ndevelop a new volume rendering scheme to learn the neural UDF representation.\nSpecifically, a new density function that correlates the property of UDF with the volume rendering scheme is introduced for robust optimization of the UDF fields.\nExperiments on the DTU and DeepFashion3D datasets show that our method not only enables high-quality reconstruction of non-closed shapes with complex typologies, but also achieves comparable performance to the SDF based methods on the reconstruction of closed surfaces.\n        \n        \n## Usage\n            \n### Setup environment\nSet up a conda environment with the right packages using:\n```\nconda env create -f conda_env.yml\nconda activate neuraludf\n```\n\nWe leverage [MeshUDF](https://github.com/cvlab-epfl/MeshUDF) to extract mesh from the learned UDF field. \nThank them for the great work.\nTo compile the custom version for your system, please run:\n```\ncd custom_mc\npython setup.py build_ext --inplace\ncd ..\n```\n\n#### Data Convention\nDownload the preprocessed \n[Deepfashion3D dataset](https://connecthkuhk-my.sharepoint.com/:u:/g/personal/xxlong_connect_hku_hk/EdAROUHE2UVHjrLruv9oVhsBN_t8SrndtkbBpNFhJjEZFQ?e=sZr0LE)\n we use and [GT point clouds](https://connecthkuhk-my.sharepoint.com/:u:/g/personal/xxlong_connect_hku_hk/EeBYbTaJHXZBss0hts1xNCwBlZ-Ju-8kbFPl_P4_SQHeQw?e=lSmjI1):\nThe DTU data and Deepfashion3d data are organized as follows:\n\n```\n\u003ccase_name\u003e\n|-- cameras_xxx.npz    # camera parameters\n|-- image\n    |-- 000.png        # target image for each view\n    |-- 001.png\n    ...\n|-- mask\n    |-- 000.png        # target mask each view (For unmasked setting, set all pixels as 255)\n    |-- 001.png\n    ...\n```\n\nHere the `cameras_xxx.npz` follows the data format in [IDR](https://github.com/lioryariv/idr/blob/main/DATA_CONVENTION.md), \nwhere `world_mat_xx` denotes the world to image projection matrix, and `scale_mat_xx` denotes the normalization matrix.\n\n### Running\n\n- **On objects with closed surfaces (DTU)**\n\nThe training has two stages. \nWe apply blending-based patch loss (used in SparseNeuS) to further improve the reconstruction quality.\n\n```shell\nbash bashs/bash_dtu_blending.sh --gpu 0 --case scan118\nbash bashs/bash_dtu_blending_ft.sh --gpu 0 --case scan118\n```\n\n- **On objects with open surfaces (Deepfashion3D)**\nIf the initial sparse_weight is inappropriate, adjust it in the fine-tuning stage\n```shell\nbash bashs/bash_garment_blending.sh --gpu 0 --case scan320 -s 0.001\nbash bashs/bash_garment_blending_ft.sh --gpu 0 --case scan320 -s 0.01\n```\n\n- **Extract surface from trained model** \n\n```shell\npython exp_runner_blending.py --mode validate_udf_mesh --conf \u003cconfig_file\u003e --case \u003ccase_name\u003e --is_continue # use latest checkpoint\n```\n\nThe corresponding mesh can be found in `exp/\u003ccase_name\u003e/\u003cexp_name\u003e/meshes/\u003citer_steps\u003e.ply`.\n\n\n### Train NeuralUDF with your custom data\n\nMore information can be found in [preprocess_custom_data](https://github.com/Totoro97/NeuS/tree/main/preprocess_custom_data) of NeuS.\n\n### The reconstruction results of ours and baselines\nYou can download the results of the methods mentioned in the paper here:\n- [DeepFashion3D](https://connecthkuhk-my.sharepoint.com/:f:/g/personal/xxlong_connect_hku_hk/Et1G0_59EWJNvebXoVhv7PUBU2WQXU12UhEDsID2t-mZ7g?e=fXEKhn) \n- [DTU](https://connecthkuhk-my.sharepoint.com/:f:/g/personal/xxlong_connect_hku_hk/EuEd4C4Fj89JoDY4--ZKJUsBdbscnaBjwkwvdxTejH8IPQ?e=W916hg)\n\n### Discussions and future work\nAs we stated in the paper, it's more difficult to train a UDF field than a SDF field, \nsince UDF doesn't enforce any topological assumption (like the surfaces are closed) and UDF is not differentiable at zero-level sets.\nAlthough we propose a series of strategies to alleviate the problem,\nthere are still some limitations, and hope that they can be addressed in the future.\n-  The weight of the geometric regularization sometimes is sensitive to some cases, and need to be tuned for better results.\nMaybe a more robust regularization stragtegy can handle this.\n- How to initialize the UDF field for open surfaces ? In the work, we still adopt sphere initialization.\n- How to extract mesh from the optimized UDF in a more robust way ? MeshUDF provides an inspiring and effective solution, \nbut it's sensitive to the gradients near zero-level sets, and cannot handle non-manifold surfaces.\n\n## Citation\n\nCite as below if you find this repository is helpful to your project:\n\n```\n@article{long2022neuraludf,\n  title={NeuralUDF: Learning Unsigned Distance Fields for Multi-view Reconstruction of Surfaces with Arbitrary Topologies},\n  author={Long, Xiaoxiao and Lin, Cheng and Liu, Lingjie and Liu, Yuan and Wang, Peng and Theobalt, Christian and Komura, Taku and Wang, Wenping},\n  journal={arXiv preprint arXiv:2211.14173},\n  year={2022}\n}\n```\n\n## Acknowledgement\n\nSome code snippets are borrowed from [NeuS](https://github.com/Totoro97/NeuS), \n[MeshUDF](https://github.com/cvlab-epfl/MeshUDF) and [SparseNeuS](https://github.com/xxlong0/SparseNeuS). \nThanks for these great projects.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fxxlong0%2Fneuraludf","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fxxlong0%2Fneuraludf","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fxxlong0%2Fneuraludf/lists"}