{"id":18621437,"url":"https://github.com/ethz-asl/neuralblox","last_synced_at":"2025-04-11T03:30:40.610Z","repository":{"id":43750066,"uuid":"414923140","full_name":"ethz-asl/neuralblox","owner":"ethz-asl","description":"Real-time Neural Representation Fusion for Robust Volumetric Mapping","archived":false,"fork":false,"pushed_at":"2024-04-09T12:54:33.000Z","size":56323,"stargazers_count":130,"open_issues_count":2,"forks_count":8,"subscribers_count":15,"default_branch":"main","last_synced_at":"2025-03-25T08:23:53.218Z","etag":null,"topics":["fusion","mapping","neural","neuralblox","online","representation","volumetric"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/ethz-asl.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2021-10-08T09:23:37.000Z","updated_at":"2025-02-27T11:18:28.000Z","dependencies_parsed_at":"2024-11-07T04:21:48.957Z","dependency_job_id":null,"html_url":"https://github.com/ethz-asl/neuralblox","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ethz-asl%2Fneuralblox","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ethz-asl%2Fneuralblox/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ethz-asl%2Fneuralblox/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ethz-asl%2Fneuralblox/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/ethz-asl","download_url":"https://codeload.github.com/ethz-asl/neuralblox/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248335334,"owners_count":21086569,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["fusion","mapping","neural","neuralblox","online","representation","volumetric"],"created_at":"2024-11-07T04:11:41.024Z","updated_at":"2025-04-11T03:30:35.553Z","avatar_url":"https://github.com/ethz-asl.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# NeuralBlox: Real-Time Neural Representation Fusion for Robust Volumetric Mapping\nThis repository contains **NeuralBlox**, our framework for volumetric mapping in latent neural representation space. \n\n![teaser](media/pipeline.png \"teaser\")\n\n# Table of Contents\n* [Paper](#Paper)\n* [Installation](#Installation)\n* [Demo](#Demo)\n\n## Paper\nIf you find our code or paper useful, please consider citing us:\n\n* Stefan Lionar\\*, Lukas Schmid\\*, Cesar Cadena, Roland Siegwart, and Andrei Cramariuc. \"**NeuralBlox: Real-Time Neural Representation Fusion for Robust Volumetric Mapping**\", International Conference on 3D Vision (3DV), pp. 1279-1289, 2021. (\\* equal contribution)\n  \\[ [IEEE](https://ieeexplore.ieee.org/abstract/document/9665902) | [ArXiv](https://arxiv.org/abs/2110.09415) | [Supplementary](https://drive.google.com/file/d/1aoK41SgWYw4toiAoUfx6VCTiZftLw-7_/view?usp=share_link) \\]\n  ```bibtex\n  @inproceedings{lionar2021neuralblox,\n   title = {NeuralBlox: Real-Time Neural Representation Fusion for Robust Volumetric Mapping},\n   author={Stefan Lionar, Lukas Schmid, Cesar Cadena, Roland Siegwart, Andrei Cramariuc},\n   booktitle={2021 International Conference on 3D Vision (3DV)}, \n   year={2021},\n   pages={1279-1289},\n   doi={10.1109/3DV53792.2021.00135}}\n  }\n  ```\n\n## Installation\n\n```\nconda env create -f environment.yaml\nconda activate neuralblox\npip install torch-scatter==2.0.4 -f https://pytorch-geometric.com/whl/torch-1.4.0+cu101.html\n```\nNote: Make sure torch-scatter and PyTorch have the same cuda toolkit version. If PyTorch has a different cuda toolkit version, run:\n```\nconda install pytorch==1.4.0 cudatoolkit=10.1 -c pytorch\n```\nNext, compile the extension modules.\nYou can do this via\n```\npython setup.py build_ext --inplace\n```\n\n**Optional:** For a noticeably faster inference on CPU-only settings, upgrade PyTorch and PyTorch Scatter to a newer version:\n\n```\npip install torch==1.7.1+cu101 torchvision==0.8.2+cu101 -f https://download.pytorch.org/whl/torch_stable.html\npip install --upgrade --no-deps --force-reinstall torch-scatter==2.0.5 -f https://pytorch-geometric.com/whl/torch-1.7.1+cu101.html\n```\n\n## Demo\n\nTo generate meshes using our pretrained models and evaluation dataset, you can select several configurations below and run it.\n\n```\npython generate_sequential.py configs/fusion/pretrained/redwood_0.5voxel_demo.yaml\npython generate_sequential.py configs/fusion/pretrained/redwood_1voxel_demo.yaml\npython generate_sequential.py configs/fusion/pretrained/redwood_1voxel_demo_cpu.yaml --no_cuda\n```\n- The mesh will be generated to `out_mesh/mesh` folder.\n- To add noise, change the values under `test.scene.noise` in the config files.\n\n## Training backbone encoder and decoder\n\nThe backbone encoder and decoder mainly follow Convolutional Occupancy Networks (https://github.com/autonomousvision/convolutional_occupancy_networks) with some modifications adapted for our use case. Our pretrained model is provided in this repository.\n\n### Dataset\n\n#### ShapeNet\nThe proprocessed ShapeNet dataset is from Occupancy Networks (https://github.com/autonomousvision/occupancy_networks). You can download it (73.4 GB) by running:\n\n```\nbash scripts/download_shapenet_pc.sh\n```\n\nAfter that, you should have the dataset in `data/ShapeNet` folder.\n\n### Training\nTo train the backbone network from scratch, run\n```\npython train_backbone.py configs/pointcloud/shapenet_grid24_pe.yaml\n```\n## Latent code fusion\n\nThe pretrained fusion network is also provided in this repository.\n\n### Training dataset\n\nTo train from scratch, you can download our preprocessed Redwood Indoor RGBD Scan dataset by running:\n```\nbash scripts/download_redwood_preprocessed.sh\n```\n\nWe align the gravity direction to be the same as ShapeNet ([0,1,0]) and convert the RGBD scans following ShapeNet format. \n\nMore information about the dataset is provided here: http://redwood-data.org/indoor_lidar_rgbd/.\n\n### Training\n    \nTo train the fusion network from scratch, run\n```\npython train_fusion.py configs/fusion/train_fusion_redwood.yaml\n```\nAdjust the path to the encoder-decoder model in `training.backbone_file` of the .yaml file if necessary.\n\n### Generation\n\n```\npython generate_sequential.py CONFIG.yaml\n```\n\nIf you are interested in generating the meshes from other dataset, e.g., [ScanNet](https://github.com/ScanNet/ScanNet):\n- Structure the dataset following the format in `demo/redwood_apartment_13k`. \n- Adjust `path`, `data_preprocessed_interval` and `intrinsics` in the config file.\n- If necessary, align the dataset to have the same gravity direction as ShapeNet by adjusting `align` in the config file.\n\nFor example,\n```\n# ScanNet scene ID 0\npython generate_sequential.py configs/fusion/pretrained/scannet_000.yaml\n\n# ScanNet scene ID 24\npython generate_sequential.py configs/fusion/pretrained/scannet_024.yaml\n```\nTo use your own models, replace `test.model_file` (encoder-decoder) and `test.merging_model_file` (fusion network) in the config file to the path of your models.\n\n### Evaluation\n\nYou can evaluate the predicted meshes with respect to a ground truth mesh by following the steps below:\n1. Install CloudCompare\n```\nsudo apt install cloudcompare\n```\n2. Copy a ground truth mesh (no RGB information expected) to `evaluation/mesh_gt`\n3. Copy prediction meshes to `evaluation/mesh_pred`\n4. If the prediction mesh does not contain RGB information, such as the output from our method, run:\n```\npython evaluate.py\n```\nElse, if it contains RGB information, such as the output from Voxblox, run:\n```\npython evaluate.py --color_mesh\n```\n\nWe provide the trimmed mesh used for the ground truth of our quantitative evaluation. It can be downloaded [here](https://drive.google.com/drive/folders/11MsRXgsNMlV1oH5sfuhKwkWSuT2odzNw?usp=share_link).\n\n\nLastly, to evaluate prediction meshes with respect to the trimmed mesh as ground truth, run:\n```\npython evaluate.py --demo\n```\n\nOr for colored mesh (e.g. from Voxblox):\n```\npython evaluate.py --demo --color_mesh\n```\n\nevaluation.csv will be generated to `evaluation` directory.\n\n## Acknowledgement\n\nSome parts of the code are inherited from the official repository of Convolutional Occupancy Networks (https://github.com/autonomousvision/convolutional_occupancy_networks).\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fethz-asl%2Fneuralblox","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fethz-asl%2Fneuralblox","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fethz-asl%2Fneuralblox/lists"}