{"id":18614409,"url":"https://github.com/IGLICT/NeRF-Editing","last_synced_at":"2025-04-11T00:30:44.784Z","repository":{"id":52214755,"uuid":"520833501","full_name":"IGLICT/NeRF-Editing","owner":"IGLICT","description":"[CVPR 2022] Code for \"NeRF-Editing: Geometry Editing of Neural Radiance Fields\"","archived":false,"fork":false,"pushed_at":"2024-05-04T12:10:36.000Z","size":15516,"stargazers_count":188,"open_issues_count":2,"forks_count":12,"subscribers_count":5,"default_branch":"main","last_synced_at":"2024-11-07T03:31:23.623Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"C++","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"gpl-3.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/IGLICT.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2022-08-03T10:22:25.000Z","updated_at":"2024-11-05T06:45:51.000Z","dependencies_parsed_at":"2022-08-31T02:50:54.364Z","dependency_job_id":null,"html_url":"https://github.com/IGLICT/NeRF-Editing","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/IGLICT%2FNeRF-Editing","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/IGLICT%2FNeRF-Editing/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/IGLICT%2FNeRF-Editing/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/IGLICT%2FNeRF-Editing/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/IGLICT","download_url":"https://codeload.github.com/IGLICT/NeRF-Editing/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248322216,"owners_count":21084333,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-07T03:25:56.172Z","updated_at":"2025-04-11T00:30:39.768Z","avatar_url":"https://github.com/IGLICT.png","language":"C++","funding_links":[],"categories":["Papers"],"sub_categories":["NeRF Related Tasks"],"readme":"# NeRF-Editing: Geometry Editing of Neural Radiance Fields\n\n![Teaser image](./img/teaser.gif)\n\n## Abstract\n\nImplicit neural rendering, especially Neural Radiance Field (NeRF), has shown great potential in novel view synthesis of a scene. However, current NeRF-based methods cannot enable users to perform user-controlled shape deformation in the scene. While existing works have proposed some approaches to modify the radiance field according to the user's constraints, the modification is limited to color editing or object translation and rotation. In this paper, we propose a method that allows users to perform controllable shape deformation on the implicit representation of the scene, and synthesizes the novel view images of the edited scene without re-training the network. Specifically, we establish a correspondence between the extracted explicit mesh representation and the implicit neural representation of the target scene. Users can first utilize well-developed mesh-based deformation methods to deform the mesh representation of the scene. Our method then utilizes user edits from the mesh representation to bend the camera rays by introducing a tetrahedra mesh as a proxy, obtaining the rendering results of the edited scene. Extensive experiments demonstrate that our framework can achieve ideal editing results not only on synthetic data, but also on real scenes captured by users.\n\n## Environment\n* Install [jittor](https://github.com/Jittor/jittor)\n    \u003cdetails\u003e\n    \u003csummary\u003e Other dependencies (click to expand) \u003c/summary\u003e\n\n    - opencv_python==4.5.2.52\n    - imageio==2.17.0\n    - trimesh==3.9.8 \n    - numpy==1.19.2\n    - pyhocon==0.3.57\n    - icecream==2.1.0\n    - tqdm==4.50.2\n    - scipy==1.7.0\n    - PyMCubes==0.1.2\n    - natsort==8.1.0\n    - tensorboardX-2.5\n\n    \u003c/details\u003e\n\nWe also use the pyrender to get the depth map.\n```\npip install pyrender\n```\n\n* Download [OpenVolumeMesh](https://www.graphics.rwth-aachen.de/software/openvolumemesh/download/) to the `OpenVolumeMesh` folder\n* Download [Eigen](https://eigen.tuxfamily.org/index.php?title=Main_Page) to the `volumeARAP_batch/Eigen` folder\n\n## Data preparation\n\nSuppose the image data is in the `$data_dir/images` folder, we first estimate the camera poses with [colmap](https://github.com/colmap/colmap). Then we process the camera poses with the command \n```\npython process_colmap.py $data_dir $data_dir\n```\nFinally the data folder looks like\n```\n$data_dir\n├── colmap_output.txt (colmap output)\n├── database.db (colmap output)\n├── images ($data_dir/images)\n├── intrinsics.txt\n├── pose\n├── rgb\n└── sparse (colmap output)\n```\n\nWe now provide a provide a ready-to-use dataset `hbychair` collected by ourselves in [google drive](https://drive.google.com/drive/folders/1OdHHNxMk9t9cDTZHlMHLSb11tf9LYHtG?usp=sharing), along with the pre-trained model and deformation results. You can put the data into the `data` folder.\n\nOr you can use [nerf-synthetic dataset](https://drive.google.com/drive/folders/128yBriW1IG_3NJ5Rp7APSTZsJqdJdfc1) directly, see `./confs/wmask_lego.conf` as an example.\n\n## Technological process\n\n##### Training\nwe adopt the training strategy of [NeuS](https://github.com/Totoro97/NeuS).\n\n```\npython exp_runner.py --mode train --conf ./confs/womask_hbychair.conf --case hbychair_neus\n```\n\n##### Extract mesh\n ```\npython exp_runner.py --mode validate_mesh --conf ./confs/womask_hbychair.conf --case hbychair_neus --is_continue # use latest checkpoint\n ```\n *We have provided a simplified mesh `mesh_nofloor_simp.obj`*\n\n##### Render image before editing\n```\npython exp_runner.py --mode circle --conf ./confs/womask_hbychair_render.conf --case hbychair_neus --is_continue  --obj_path ./logs/hbychair_wo_mask/mesh_nofloor_simp.obj\n```\n\n*Note: `obj_path` is optional, which provides better rendering results.*\n\n##### Construct cage mesh\n ```\npython exp_runner.py --mode validate_mesh --conf ./confs/womask_hbychair.conf --case hbychair_neus --is_continue --do_dilation\n ```\n *We have provided a cage mesh `mesh_cage_nofloor.obj`*\n\n##### Construct tetrahedral mesh using [TetWild](https://github.com/Yixin-Hu/TetWild). \n```\n./TetWild ../../src/logs/hbychair_wo_mask/mesh_cage_nofloor.obj\n```\n*Note that we modify the tetrahedra storage format of Tetwild output. Therefore, please compile the `tetwild` in this repository following the instructions [here](https://github.com/Yixin-Hu/TetWild).*\n\n##### Change the output to `ovm` format.\n```\n./simple_mesh ../../src/logs/hbychair_wo_mask/mesh_cage_nofloor_.txt ../../src/logs/hbychair_wo_mask/mesh_cage_nofloor_.ovm\n```\n*`simple_mesh` can be obtained using the `CMakeLists.txt` in the `OpenVolumeMesh` folder.*\n\n##### Editing\n Deform the extracted mesh *with any mesh editing tool*, and put the (sequence) mesh in `$deformed_dir` folder.\n \n *We have provided a deformed mesh `deformed_mesh.obj` and a folder named as `mesh_seq`*\n\n##### Propagate editing\nGenerate the controlpoint.txt to guide the deformation.\n```\npython barycentric_control_pts_jittor.py\n```\nNote that specify the `mesh_path` (extracted mesh), `tet_path` (tetrahedra mesh) and `deformed_dir`(deformed mesh sequence) first.\n\nAnd the format of controlpoint.txt is listed below.\n\n```\n10 (Number of sequence)\nN (Num of control points)\nx1 y1 z1\nx2 y2 z2\n...\nN (Num of control points)\nx1 y1 z1\nx2 y2 z2\n...\n.\n.\n.\nN (Num of barycentric coordinate)\nid1 id2 id3 id4 (vert index of this tet)\nu1 v1 w1 z1\nid1' id2' id3' id4'\nu2 v2 w2 z2\n...\n```\nCompile the `volumeARAP_batch` project to obtain `volumeARAP`, and deform the tetrehedra mesh.\n```\n./volumeARAP ../../src/logs/hbychair_wo_mask/mesh_cage_nofloor_.ovm ../../src/logs/hbychair_wo_mask/mesh_seq/2_barycentric_control.txt ../../src/logs/hbychair_wo_mask/mesh_seq_ovm 0\n```\n##### Rendering after editing\n```\npython exp_runner.py --mode circle --conf ./confs/womask_hbychair_render.conf --case hbychair_neus --is_continue --use_deform --reconstructed_mesh_file ./logs/hbychair_wo_mask/mesh_cage_nofloor_.txt --deformed_mesh_file ./logs/hbychair_wo_mask/mesh_seq_ovm/arap_result_0000_.ovm --obj_path ./logs/hbychair_wo_mask/deformed_mesh.obj\n```\n\n* fix camera (generate sequential editing results in a fixed camera)\n```\npython exp_runner.py --mode circle --conf ./confs/womask_hbychair_render.conf --case hbychair_neus --is_continue --use_deform --reconstructed_mesh_file ./logs/hbychair_wo_mask/mesh_cage_nofloor_.txt --deformed_mesh_file ./logs/hbychair_wo_mask/mesh_seq_ovm/arap_result_0000_.ovm --obj_path ./logs/hbychair_wo_mask/deformed_mesh.obj --fix_camera\n```\n\n## Acknowledgement\nThis code borrows heavily from [NeuS](https://github.com/Totoro97/NeuS).\n\n## Citation\n\nIf you found this code useful please cite our work as:\n\n```\n@inproceedings{yuan2022nerf,\n  title={NeRF-editing: geometry editing of neural radiance fields},\n  author={Yuan, Yu-Jie and Sun, Yang-Tian and Lai, Yu-Kun and Ma, Yuewen and Jia, Rongfei and Gao, Lin},\n  booktitle={Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition},\n  pages={18353--18364},\n  year={2022}\n}\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FIGLICT%2FNeRF-Editing","html_url":"https://awesome.ecosyste.ms/projects/github.com%2FIGLICT%2FNeRF-Editing","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FIGLICT%2FNeRF-Editing/lists"}