{"id":19728654,"url":"https://github.com/ashawkey/nerf2mesh","last_synced_at":"2025-05-15T02:10:41.541Z","repository":{"id":108475417,"uuid":"609092685","full_name":"ashawkey/nerf2mesh","owner":"ashawkey","description":"[ICCV2023] Delicate Textured Mesh Recovery from NeRF via Adaptive Surface Refinement","archived":false,"fork":false,"pushed_at":"2024-11-11T10:41:31.000Z","size":1004,"stargazers_count":941,"open_issues_count":79,"forks_count":90,"subscribers_count":13,"default_branch":"main","last_synced_at":"2025-04-14T00:58:35.084Z","etag":null,"topics":["mesh","nerf","real-time"],"latest_commit_sha":null,"homepage":"https://me.kiui.moe/nerf2mesh/","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/ashawkey.png","metadata":{"files":{"readme":"readme.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2023-03-03T11:08:04.000Z","updated_at":"2025-04-03T01:20:03.000Z","dependencies_parsed_at":null,"dependency_job_id":"bf9ecec2-89ec-483c-9839-3c9e4d74a010","html_url":"https://github.com/ashawkey/nerf2mesh","commit_stats":null,"previous_names":[],"tags_count":2,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ashawkey%2Fnerf2mesh","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ashawkey%2Fnerf2mesh/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ashawkey%2Fnerf2mesh/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ashawkey%2Fnerf2mesh/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/ashawkey","download_url":"https://codeload.github.com/ashawkey/nerf2mesh/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":254259387,"owners_count":22040821,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["mesh","nerf","real-time"],"created_at":"2024-11-12T00:06:50.352Z","updated_at":"2025-05-15T02:10:36.532Z","avatar_url":"https://github.com/ashawkey.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# nerf2mesh\n\n\nThis repository contains a PyTorch re-implementation of the paper: [Delicate Textured Mesh Recovery from NeRF via Adaptive Surface Refinement](https://arxiv.org/abs/2303.02091).\n\n### [Project Page](https://ashawkey.github.io/nerf2mesh/) | [Arxiv](https://arxiv.org/abs/2303.02091) | [Paper](https://huggingface.co/ashawkey/nerf2mesh/resolve/main/paper.pdf) | [Models](https://huggingface.co/ashawkey/nerf2mesh/tree/main/scenes) \n\n**News (2023.5.3)**: support [background removal](https://github.com/OPHoperHPO/image-background-remove-tool) and [SDF](https://github.com/Totoro97/NeuS) mode for stage 0, which produces more robust and smooth mesh for single-object reconstruction:\n\n![](assets/teaser2.jpg)\n\n![](assets/teaser.jpg)\n\n# Install\n\n```bash\ngit clone https://github.com/ashawkey/nerf2mesh.git\ncd nerf2mesh\n```\n\n### Install with pip\n```bash\npip install -r requirements.txt\n\n# tiny-cuda-nn\npip install git+https://github.com/NVlabs/tiny-cuda-nn/#subdirectory=bindings/torch\n\n# nvdiffrast\npip install git+https://github.com/NVlabs/nvdiffrast/\n\n# pytorch3d\npip install git+https://github.com/facebookresearch/pytorch3d.git\n```\n\n### Build extension (optional)\nBy default, we use [`load`](https://pytorch.org/docs/stable/cpp_extension.html#torch.utils.cpp_extension.load) to build the extension at runtime.\nHowever, this may be inconvenient sometimes.\nTherefore, we also provide the `setup.py` to build each extension:\n```bash\n# install all extension modules\nbash scripts/install_ext.sh\n\n# if you want to install manually, here is an example:\ncd raymarching\npython setup.py build_ext --inplace # build ext only, do not install (only can be used in the parent directory)\npip install . # install to python path (you still need the raymarching/ folder, since this only install the built extension.)\n```\n\n### Tested environments\n* Ubuntu 22 with torch 1.12 \u0026 CUDA 11.6 on a V100.\n\n# Usage\n\nWe support the original NeRF data format like [nerf-synthetic](https://drive.google.com/drive/folders/128yBriW1IG_3NJ5Rp7APSTZsJqdJdfc1), and COLMAP dataset like [Mip-NeRF 360](http://storage.googleapis.com/gresearch/refraw360/360_v2.zip).\nPlease download and put them under `./data`.\n\nFirst time running will take some time to compile the CUDA extensions.\n\n### Basics\n```bash\n### Stage0 (NeRF, continuous, volumetric rendering), this stage exports a coarse mesh under \u003cworkspace\u003e/mesh_stage0/\n\n# nerf\npython main.py data/nerf_synthetic/lego/ --workspace trial_syn_lego/ -O --bound 1 --scale 0.8 --dt_gamma 0 --stage 0 --lambda_tv 1e-8\n\n# colmap\npython main.py data/garden/ --workspace trial_360_garden -O --data_format colmap --bound 16 --enable_cam_center --enable_cam_near_far --scale 0.3 --downscale 4 --stage 0 --lambda_entropy 1e-3 --clean_min_f 16 --clean_min_d 10 --lambda_tv 2e-8 --visibility_mask_dilation 50\n\n### Stage1 (Mesh, binarized, rasterization), this stage exports a fine mesh with textures under \u003cworkspace\u003e/mesh_stage1/\n\n# nerf\npython main.py data/nerf_synthetic/lego/ --workspace trial_syn_lego/ -O --bound 1 --scale 0.8 --dt_gamma 0 --stage 1\n\n# colmap\npython main.py data/garden/ --workspace trial_360_garden   -O --data_format colmap --bound 16 --enable_cam_center --enable_cam_near_far --scale 0.3 --downscale 4 --stage 1 --iters 10000\n\n### Web Renderer\n# you can simply open \u003cworkspace\u003e/mesh_stage1/mesh.obj with a 3D viewer to visualize the diffuse texture.\n# to render full diffuse + specular, you'll need to host this folder (e.g., by vscode live server), and open renderer.html for further instructions.\n```\n\n### Custom Dataset\n\n**Tips:**\n* To get best mesh quality, you may need to adjust `--scale` to let the most interested object fall inside the unit box `[-1, 1]^3`, which can be visualized by appending `--vis_pose`.\n* To better model background (especially for outdoor scenes), you may need to adjust `--bound` to let most sparse points fall into the full box `[-bound, bound]^3`, which can also be visualized by appending `--vis_pose`.\n* For single object centered captures focusing on mesh assets quality:\n  * remove the background by `scripts/remove_bg.py` and only reconstruct the targeted object.\n  * use `--sdf` to enable sdf based stage 0 model.\n  * use `--diffuse_only` if you only want to get the diffuse texture.\n  * adjust `--decimate_target 1e5` to control stage 0 number of mesh faces, and adjust `--refine_remesh_size 0.01` to control stage 1 number of mesh faces (average edge length).\n  * adjust `--lambda_normal 1e-2` for more smooth surface.\n* For forward-facing captures:\n  * remove `--enable_cam_center` so the scene center is determined by sparse points instead of camera positions.\n\n```bash\n# prepare your video or images under /data/custom, and run colmap (assumed installed):\npython scripts/colmap2nerf.py --video ./data/custom/video.mp4 --run_colmap # if use video\npython scripts/colmap2nerf.py --images ./data/custom/images/ --run_colmap # if use images\n\n# generate downscaled images if resolution is very high and OOM (asve to`data/\u003cname\u003e/images_{downscale}`) \npython scripts/downscale.py data/\u003cname\u003e --downscale 4\n# NOTE: remember to append `--downscale 4` as well when running main.py\n\n# perform background removal for single object 360 captures (save to 'data/\u003cname\u003e/mask')\npython scripts/remove_bg.py data/\u003cname\u003e/images\n# NOTE: the mask quality depends on background complexity, do check the mask!\n\n# recommended options for single object 360 captures\npython main.py data/custom/ --workspace trial_custom -O --data_format colmap --bound 1 --dt_gamma 0 --stage 0 --clean_min_f 16 --clean_min_d 10 --visibility_mask_dilation 50 --iters 10000 --decimate_target 1e5 --sdf\n# NOTE: for finer faces, try --decimate_target 3e5\n\npython main.py data/custom/ --workspace trial_custom -O --data_format colmap --bound 1 --dt_gamma 0 --stage 1 --iters 5000 --lambda_normal 1e-2 --refine_remesh_size 0.01 --sdf\n# NOTE: for finer faces, try --lambda_normal 1e-1 --refine_remesh_size 0.005\n\n# recommended options for outdoor 360-inwarding captures\npython main.py data/custom/ --workspace trial_custom -O --data_format colmap --bound 16 --enable_cam_center --enable_cam_near_far --stage 0 --lambda_entropy 1e-3 --clean_min_f 16 --clean_min_d 10 --lambda_tv 2e-8 --visibility_mask_dilation 50\n\npython main.py data/custom/ --workspace trial_custom -O --data_format colmap --bound 16 --enable_cam_center --enable_cam_near_far --stage 1 --iters 10000 --lambda_normal 1e-3\n\n# recommended options for forward-facing captures\npython main.py data/custom/ --workspace trial_custom -O --data_format colmap --bound 2 --scale 0.1 --stage 0 --clean_min_f 16 --clean_min_d 10 --lambda_tv 2e-8 --visibility_mask_dilation 50\n\npython main.py data/custom/ --workspace trial_custom -O --data_format colmap --bound 2 --scale 0.1 --stage 1 --iters 10000 --lambda_normal 1e-3\n```\n\n### Advanced Usage\n```bash\n### -O: the recommended setting, equals\n--fp16 --preload --mark_untrained --random_image_batch --adaptive_num_rays --refine --mesh_visibility_culling\n\n### load checkpoint\n--ckpt latest # by default we load the latest checkpoint in the workspace\n--ckpt scratch # train from scratch. For stage 1, this will still load the stage 0 model as an initialization.\n--ckpt trial/checkpoints/xxx.pth # specify it by path\n\n### testing\n--test # test, save video and mesh\n--test_no_video # do not save video\n--test_no_mesh # do not save mesh\n\n### dataset related\n--data_format [colmap|nerf|dtu] # dataset format\n--enable_cam_center # use camera center instead of sparse point center as scene center (colmap dataset only)\n--enable_cam_near_far # estimate camera near \u0026 far from sparse points (colmap dataset only)\n\n--bound 16 # scene bound set to [-16, 16]^3, note that only meshes inside the center [-1, 1]^3 will be adaptively refined!\n--scale 0.3 # camera scale, if not specified, automatically estimate one based on camera positions. Important targets should be scaled into the center [-1, 1]^3.\n\n### visualization \n--vis_pose # viusalize camera poses and sparse points (sparse points are colmap dataset only)\n--gui # open gui (only for testing, training in gui is not well supported!)\n\n### balance between surface quality / rendering quality\n\n# increase these weights to get better surface quality but worse rendering quality\n--lambda_tv 1e-7 # total variation loss (stage 0)\n--lambda_entropy 1e-3 # entropy on rendering weights (transparency, alpha), encourage them to be either 0 or 1 (stage 0)\n--lambda_lap 0.001 # laplacian smoothness loss (stage 1)\n--lambda_normal 0.001 # normal consistency loss (stage 1)\n--lambda_offsets 0.1 # vertex offsets L2 loss (stage 1)\n--lambda_edgelen 0.1 # edge length L2 loss (stage 1)\n\n# set all smoothness regularizations to 0, usually get the best rendering quality\n--wo_smooth\n\n# only use diffuse shading\n--diffuse_only\n\n### coarse mesh extraction \u0026 post-processing\n--mcubes_reso 512 # marching cubes resolution\n--decimate_target 300000 # decimate raw mesh to this face number\n--clean_min_d 5 # isolated floaters with smaller diameter will be removed\n--clean_min_f 8 # isolated floaters with fewer faces will be removed\n--visibility_mask_dilation 5 # dilate iterations after performing visibility face culling\n\n### fine mesh exportation\n--texture_size 4096 # max texture image resolution\n--ssaa 2 # super-sampling anti-alias ratio\n--refine_size 0.01 # finest edge len at subdivision\n--refine_decimate_ratio 0.1 # decimate ratio at each refine step\n--refine_remesh_size 0.02 # remesh edge len after decimation\n\n### Depth supervision (colmap dataset only)\n\n# download depth checkpoints (omnidata v2)\ncd depth_tools\nbash download_models.sh\ncd ..\n\n# generate dense depth (save to `data/\u003cname\u003e/depths`)\npython depth_tools/extract_depth.py data/\u003cname\u003e/images_4\n\n# enable dense depth training\npython main.py data/\u003cname\u003e -O --bound 16 --data_format colmap --enable_dense_depth\n```\n\nPlease check the `scripts` directory for more examples on common datasets, and check `main.py` for all options.\n\n# Acknowledgement\n\n* The NeRF framework is based on [torch-ngp](https://github.com/ashawkey/torch-ngp).\n* The GUI is developed with [DearPyGui](https://github.com/hoffstadt/DearPyGui).\n\n# Citation\n\n```\n@article{tang2022nerf2mesh,\n  title={Delicate Textured Mesh Recovery from NeRF via Adaptive Surface Refinement},\n  author={Tang, Jiaxiang and Zhou, Hang and Chen, Xiaokang and Hu, Tianshu and Ding, Errui and Wang, Jingdong and Zeng, Gang},\n  journal={arXiv preprint arXiv:2303.02091},\n  year={2022}\n}\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fashawkey%2Fnerf2mesh","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fashawkey%2Fnerf2mesh","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fashawkey%2Fnerf2mesh/lists"}