{"id":20189835,"url":"https://github.com/anttwo/macarons","last_synced_at":"2025-04-10T08:28:30.538Z","repository":{"id":168306690,"uuid":"641756731","full_name":"Anttwo/MACARONS","owner":"Anttwo","description":"(CVPR 2023) Official code of MACARONS: Mapping And Coverage Anticipation with RGB ONline Self-supervision. Also contains an updated and improved implementation of our previous work SCONE (NeurIPS 2022), on which this work is built.","archived":false,"fork":false,"pushed_at":"2023-12-23T13:28:14.000Z","size":17188,"stargazers_count":77,"open_issues_count":3,"forks_count":7,"subscribers_count":4,"default_branch":"main","last_synced_at":"2025-03-24T08:07:46.584Z","etag":null,"topics":["3d-reconstruction","computer-vision","cvpr-2023","cvpr2023","deep-learning","neurips-2022","neurips2022","next-best-view","pytorch"],"latest_commit_sha":null,"homepage":"","language":"Jupyter Notebook","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/Anttwo.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2023-05-17T05:31:33.000Z","updated_at":"2025-02-18T18:19:54.000Z","dependencies_parsed_at":"2023-12-23T15:49:21.051Z","dependency_job_id":"e0f089a2-9e4e-4caf-8c05-1775d1ea6238","html_url":"https://github.com/Anttwo/MACARONS","commit_stats":null,"previous_names":["anttwo/macarons"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Anttwo%2FMACARONS","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Anttwo%2FMACARONS/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Anttwo%2FMACARONS/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Anttwo%2FMACARONS/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/Anttwo","download_url":"https://codeload.github.com/Anttwo/MACARONS/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248184644,"owners_count":21061373,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["3d-reconstruction","computer-vision","cvpr-2023","cvpr2023","deep-learning","neurips-2022","neurips2022","next-best-view","pytorch"],"created_at":"2024-11-14T03:39:07.197Z","updated_at":"2025-04-10T08:28:30.513Z","avatar_url":"https://github.com/Anttwo.png","language":"Jupyter Notebook","funding_links":[],"categories":[],"sub_categories":[],"readme":"\u003cdiv align=\"center\"\u003e\r\n\r\n# MACARONS: Mapping And Coverage Anticipation with RGB ONline Self-supervision\r\n\r\n\u003cfont size=\"4\"\u003e\r\n\u003ca href=\"https://imagine.enpc.fr/~guedona/\"\u003eAntoine Guédon\u003c/a\u003e\u0026emsp;\r\n\u003ca href=\"https://www.tmonnier.com/\"\u003eTom Monnier\u003c/a\u003e\u0026emsp;\r\n\u003ca href=\"https://imagine.enpc.fr/~monasse/\"\u003ePascal Monasse\u003c/a\u003e\u0026emsp;\r\n\u003ca href=\"https://vincentlepetit.github.io/\"\u003eVincent Lepetit\u003c/a\u003e\u0026emsp;\r\n\u003c/font\u003e\r\n\u003cbr\u003e\r\n\r\n\u003cimg src=\"./media/trajectories/liberty2_macarons.png\" alt=\"liberty_traj.png\" width=\"400\"/\u003e\r\n\u003cimg src=\"./media/reconstructions/liberty_1_color.png\" alt=\"liberty_reco.png\" width=\"400\"/\u003e \u003cbr\u003e\r\n\u003cimg src=\"./media/trajectories/pantheon_2_macarons.png\" alt=\"pantheon_traj.png\" width=\"400\"/\u003e\r\n\u003cimg src=\"./media/reconstructions/pantheon_2_color.png\" alt=\"pantheon_reco.png\" width=\"400\"/\u003e \u003cbr\u003e\r\n\r\n\u003c/div\u003e\r\n\r\n## Description\r\n\r\nOfficial PyTorch implementation of \r\n[**MACARONS: Mapping And Coverage Anticipation with RGB ONline Self-supervision**](https://imagine.enpc.fr/~guedona/MACARONS/) \r\n(CVPR 2023).\u003cbr\u003e\r\nAlso includes an updated and improved implementation of our previous work \r\n[**SCONE: Surface Coverage Optimization in Unknown Environments by Volumetric Integration**](https://arxiv.org/abs/2208.10449) \r\n(NeurIPS 2022, Spotlight), on which this work is built.\r\n\r\nWe introduce a method that **simultaneously learns** to **explore new large environments** and to **reconstruct them in 3D** from **color images** \r\nin a **self-supervised fashion**. This is closely related to the **Next Best View problem (NBV)**, where one has to identify where to move \r\nthe camera next to improve the coverage of an unknown scene.\r\n\r\n\u003cdiv align=\"center\"\u003e\r\n\r\n\u003ca href=\"https://www.youtube.com/watch?v=NlUNFJYuBGs\"\u003e\u003cimg src=\"./media/thumbnail.PNG\" alt=\"Macarons illustration\"\u003e\u003c/a\u003e\r\n \r\n\u003c/div\u003e\r\n\r\nThis repository contains:\r\n\r\n- Scripts to generate ground truth coverage data from 3D meshes\r\n- Scripts to initialize and train both SCONE and MACARONS models\r\n- Evaluation pipelines and notebooks to reproduce and visualize results for both MACARONS and SCONE\r\n- Interactive demos to experiment with the models, built with [Gradio](https://gradio.app/)\r\n- Links to download [training data](https://drive.google.com/drive/folders/17kv7Ud882G06oRgYC84IcOb4B9X_NsN6?usp=sharing) from our Google Drive\r\n- Links to download [pretrained weights](https://drive.google.com/drive/folders/1wyc9_QFmcxOz4oerE8kCQ3I8LO5zioZL?usp=sharing) from our Google Drive\r\n\r\n\u003cdetails\u003e\r\n\u003csummary\u003eIf you find this code useful, don't forget to \u003cb\u003estar the repo :star:\u003c/b\u003e and \u003cb\u003ecite the papers :point_down:\u003c/b\u003e\u003c/summary\u003e\r\n\r\n```\r\n@inproceedings{guedon2023macarons,\r\n  title={MACARONS: Mapping And Coverage Anticipation with RGB Online Self-Supervision},\r\n  author={Gu{\\'e}don, Antoine and Monnier, Tom and Monasse, Pascal and Lepetit, Vincent},\r\n  booktitle={Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition},\r\n  pages={940--951},\r\n  year={2023}\r\n}\r\n```\r\n\r\n```\r\n@article{guedon2022scone,\r\n  title={SCONE: Surface Coverage Optimization in Unknown Environments by Volumetric Integration},\r\n  author={Gu{\\'e}don, Antoine and Monasse, Pascal and Lepetit, Vincent},\r\n  journal={Advances in Neural Information Processing Systems},\r\n  volume={35},\r\n  pages={20731--20743},\r\n  year={2022}\r\n}\r\n```\r\n\r\n\u003c/details\u003e\r\n\r\n\u003cdetails\u003e\r\n\u003csummary\u003e\u003cb\u003eMajor code updates\u003c/b\u003e\u003c/summary\u003e\r\n\r\n- 07/23: Updated a script to automatically generate settings file for custom scenes\r\n- 07/23: Added a tutorial notebook to reproduce qualitative results with better quality\r\n- 05/23: first code release\r\n\r\n\u003c/details\u003e\r\n\r\n## Installation\r\n\r\n### 1. Create a conda environment\r\n\r\nRun the following commands to create an appropriate conda environment.\r\n\r\n```\r\nconda env create -f environment.yml\r\nconda activate macarons\r\n```\r\n\r\nDepending on the config of your machine and cuda drivers, you may have problems creating a working environment. \u003cbr\u003e\r\nIf so, you can install manually the following packages with conda and versions matching your config:\r\n\r\n1. Install numpy\r\n2. Install matplotlib\r\n3. Install pytorch\r\n4. Install pytorch3d\r\n5. Install gradio\r\n\r\nTo use jupyter-lab and our rendering functions in notebooks:\r\n\r\n1. Install jupyter-lab\r\n2. Install plotly\r\n3. Install nodejs\r\n4. Install ipywidgets\r\n\r\n### 2. Download Datasets and preprocess data\r\n\r\n#### a) ShapeNetCore.v1\r\n\r\nTo facilitate the training of SCONE's architecture, we generate training data using 3D meshes from\r\n[ShapeNetCore v1](https://shapenet.org/). \r\nIn particular, we generate ground truth data on occupancy probability and surface coverage gain from multiple\r\ncamera viewpoints. \u003cbr\u003e\r\n\r\nAdditionally, because MACARONS incorporates neural modules inspired from SCONE, we suggest using this data\r\nas a pretraining tool for MACARONS, before training it in large-scale, \r\nunknown scenes with self-supervision from RGB images.\r\nThis pretraining step improves performance while reducing overall training time.\r\n\r\nTo generate this data, please start by **downloading the ShapeNetCore v1 dataset from the \r\n[source website](https://shapenet.org/)**. \u003cbr\u003e\r\nThen, select the ShapeNet object categories on which you want to train and test your model.\r\n\r\nIn our experiments, we selected the following categories from the downloaded `ShapeNetCore.v1` dataset folder:\r\n\r\n\u003cdiv align=\"center\"\u003e\r\n \r\n| Label      | Corresponding Directory | Used for...                |\r\n| :--------: | :---------------------: | :------------------------: |\r\n| Airplane   | 02691156                | Training, Validation, Test |\r\n| Cabinet    | 02933112                | Training, Validation, Test |\r\n| Car        | 02958343                | Training, Validation, Test |\r\n| Chair      | 03001627                | Training, Validation, Test |\r\n| Lamp       | 03636649                | Training, Validation, Test |\r\n| Sofa       | 04256520                | Training, Validation, Test |\r\n| Table      | 04379243                | Training, Validation, Test |\r\n| Watercraft | 04530566                | Training, Validation, Test |\r\n| Bus        | 02924116                | Test only                  |\r\n| Bed        | 02818832                | Test only                  |\r\n| Bookshelf  | 02871439                | Test only                  |\r\n| Bench      | 02828884                | Test only                  |\r\n| Guitar     | 03467517                | Test only                  |\r\n| Motorbike  | 03790512                | Test only                  |\r\n| Skateboard | 04225987                | Test only                  |\r\n| Pistol     | 03948459                | Test only                  |\r\n\r\n\u003c/div\u003e\r\n\r\nYou just have to move the directories `02691156`, `02933112`, `02958343`, `03001627`, `03636649`, \r\n`04256520`, `04379243`, `04530566` to the path `./data/ShapeNetCore.v1/train_categories/`. \u003cbr\u003e\r\n\r\nSimilarly, move the directories `02924116`, `02818832`, `02871439`, `02828884`, `03467517`, \r\n`03790512`, `04225987`, `03948459` to the path `./data/ShapeNetCore.v1/test_categories/`. \u003cbr\u003e\r\n\r\nFinally, go to `./data/ShapeNetCore.v1/` and run the following script using python:\r\n\r\n```\r\npython generate_shapenet_data.py\r\n```\r\n\r\nGenerating the training data for all meshes takes time, up to around 10 hours. \r\nHowever, it considerably reduces the training time for all future trainings of SCONE.\r\n\r\n#### b) Dataset of large-scale 3D scenes\r\n\r\nWe conducted self-supervised training and experiments in large environments using 3D meshes downloaded \r\non the website Sketchfab under the CC license (3D models courtesy of [Brian Trepanier](https://sketchfab.com/CMBC) \r\nand [Andrea Spognetta](https://sketchfab.com/spogna), we thank them for their awesome work). \u003cbr\u003e\r\n\r\nAll download links for the original `.blend` files can be found on our \r\n[project webpage](https://imagine.enpc.fr/~guedona/MACARONS/). \u003cbr\u003e\r\nHowever, we slightly modified the meshes using rotations and scaling operations, and we extracted \r\nfor each scene both a mesh file (with extension `.obj`), a material file (with extension `.mtl`) \r\nas well as various metadata in order to facilitate 3D data processing on GPU with \r\n[PyTorch3D](https://pytorch3d.org/).\r\n\r\nYou can directly download our preprocessed meshes and textures on our \r\n[Google Drive](https://drive.google.com/drive/folders/17kv7Ud882G06oRgYC84IcOb4B9X_NsN6?usp=sharing).\r\nFor any scene in the dataset, just download all files in the Google Drive subdirectory\r\nand move them to the corresponding subdirectory in `./data/scenes/`.\r\n\r\n#### c) Building your own custom dataset of 3D scenes\r\n\r\nOnce you have a `.blend` file of your own custom 3D scene (or any `.blend` file downloaded from Sketchfab, for example),\r\nyou should extract from it the following files in order to facilitate 3D data processing on GPU with \r\n[PyTorch3D](https://pytorch3d.org/):\r\n- A mesh file, with extension `.obj`\r\n- A material file, with extension `.mtl`\r\n\r\nTo this end, you can use [Blender](https://www.blender.org/) and follow the steps below. \u003cbr\u003e\r\nLet's say we want to use MACARONS to explore and reconstruct the Statue of Liberty.\r\n1. First, download the `.zip` containing the mesh of the \r\n[Statue of Liberty](https://sketchfab.com/3d-models/statue-of-liberty-new-york-ny-usa-8401e1c10480476e8d7e3085d1aec923) \r\ncreated by [Brian Trepanier](https://sketchfab.com/CMBC) on Sketchfab. It should include a `.blend` file in `source` and \r\na texture image `a-StatueOfLiberty.jpg` in `textures`.\r\n2. Open the `.blend` file with Blender, and go to `File` \u003e `Export` \u003e `Wavefront (.obj)`, \r\nas shown in the following image.\u003cbr\u003e\r\n\u003cdiv align=\"center\"\u003e\r\n\u003cimg src=\"./media/blender/export_0.png\" alt=\"blender_export_0.png\" width=\"600\"/\u003e \r\n\u003c/div\u003e\r\n3. Make sure to check `OBJ Objects` and `Material Groups`, and select `Strip Path` as the Path Mode, as shown \r\nin the following image. \r\nThen, click on `Export OBJ` to output an `.obj` file and a `.mtl` file.\u003cbr\u003e\r\n\u003cdiv align=\"center\"\u003e\r\n\u003cimg src=\"./media/blender/export_1.png\" alt=\"blender_export_1.png\" width=\"600\"/\u003e\r\n\u003c/div\u003e\r\n4. Finally, move the `.obj` file, the `.mtl` file, as well as the `.jpg` texture file contained\r\nin the original archive downloaded from Sketchfab into the corresponding directory in `./data/scenes/`.\u003cbr\u003e\r\nIn this example, we just have to move all files `liberty.obj`, `liberty.mtl` and `a-StatueOfLiberty.jpg` \r\nto the directory `./data/scenes/liberty`. \u003cbr\u003e\r\nPlease note that the `.obj` and `.mtl` files should have the same name apart from the extension.\u003cbr\u003e\r\n\r\nYou will notice that, for each scene, the corresponding subdirectory in `./data/scenes/` also contains \r\na file named `settings.json`. \u003cbr\u003e\r\nIf you want to apply MACARONS on your own custom 3D scenes,\r\nyou can either run a script to automatically generate this file (as described below), \r\nor write it by yourself.\u003cbr\u003e\r\nThese files contain input hyperparameters for the 3D scenes to explore.\r\nIn particular, to write your own `settings.json` you have to decide on:\r\n- A **bounding box** delimiting the exploration area. For better results, you should scale the 3D mesh \r\nof your scene so that the dimensions of the bounding box are similar to those the `settings.json` files we provide.\r\n- The **dimensions of a coarse 3D grid** delimiting cells in which we register newly reconstructed surface points, \r\nin order to compute accurate surface coverage gains. The dimensions of the grid should depend on your bounding box, \r\nso that cells are approximately perfect cubes. \u003cbr\u003e\r\nIn our experiments, we use between 30 and 100 cells for each scene.\r\n- A **5D camera grid** indicating which candidate camera poses to explore.\r\n\r\nFeel free to experiment with these hyperparameters, as many configurations give good coverage results \r\nbut different trajectories.\u003cbr\u003e\r\nPlease take a look at our own `settings.json` files for an illustration of a good balance between the \r\ndifferent input hyperparameters.\u003cbr\u003e\r\n\r\nFinally, you will also notice additional files named `occupied_pose.pt`.\u003cbr\u003e\r\nWe use a really simple approach to approximately identify occupied camera poses, and store the results in this file.\r\nThis is useful to ensure that we do select an empty camera pose when sampling a random camera pose at the start of \r\na new trajectory.\u003cbr\u003e\r\nAfter writing (or not) your `settings.json` file, just add the name of your custom scene directories to the config file\r\n`./data/scenes/generate_scene_data_config.json`.\u003cbr\u003e\r\nThen, you can generate automatically `occupied_pose.pt` (as well as `settings.json`, if no settings file is detected in the directory) by running the following command.\r\n\r\n```\r\npython generate_scene_data.py\r\n```\r\n\r\n## Initialization\r\n\r\nThe full MACARONS architecture simultaneously reconstructs the scene and selects the next best camera pose\r\nby running three neural modules:\r\n1. **Depth** module\r\n2. **Occupancy probability** module, inspired by our previous work SCONE\r\n3. **Surface coverage gain** module, inspired by our previous work SCONE\r\n\r\n\u003cdiv align=\"center\"\u003e\r\n\u003cimg src=\"./media/modules.png\" alt=\"modules.png\" width=\"600\"/\u003e\r\n\u003c/div\u003e\r\n \r\nBefore training, we propose the following strategy to initialize the different neural modules and better\r\nstabilize training:\r\n\r\n1. Following [Watson *et al.*](https://arxiv.org/abs/2104.14540), we initialize \r\nthe depth module with weights from a [ResNet18](https://arxiv.org/abs/1512.03385) architecture trained on ImageNet.\r\n2. We initialize the occupancy probability module with either \r\n   1. The simple initialization process\r\n      we described in the main paper, or \r\n   2. A full pretraining on ShapeNetCore v1 for even better performance.\r\n3. Using our previsouly initialized occupancy probability module, we now initialize the surface coverage gain module \r\nwith either\r\n   1. The simple initialization process\r\n         we described in the main paper, or \r\n   2. A full pretraining on ShapeNetCore v1 for even better performance.\r\n\r\nWe detail each step below. You can skip any step by downloading the corresponding weights\r\nfrom our [Google Drive](https://drive.google.com/drive/folders/1wyc9_QFmcxOz4oerE8kCQ3I8LO5zioZL?usp=sharing).\r\n\r\n### 1. Depth module\r\n\r\nTo generate a depth module with weights initialized from a trained ResNet18 architecture, go to `./weights/` and\r\nrun the following command:\r\n\r\n```\r\npython generate_depth_model.py\r\n```\r\n\r\nYou can skip this step by downloading the corresponding weights file `depth_with_resnet_imagenet_weights.pth` \r\nfrom our [Google Drive](https://drive.google.com/drive/folders/1C4l9QtzcnaLSdh4FXkoDEsUhSDZ_u8ex?usp=sharing), \r\nand move it to `./weights/resnet/`.\r\n\r\n### 2. Occupancy probability module\r\n\r\nYou can initialize an occupancy probability neural module by running the following command in the main directory:\r\n\r\n```\r\npython initialize_occ.py\r\n```\r\n\r\nThis command will use the default config file `occupancy_pretraining_config.json` located at `./configs/scone/occupancy/` \r\nand will perform a full pretraining on ShapeNetCore v1.\u003cbr\u003e\r\nIndeed, to reduce the overall training time of MACARONS and improve performance, we suggest pretraining on ShapeNetCore v1 both the occupancy probability module and the surface coverage gain module, inspired by SCONE.\u003cbr\u003e\r\n\r\nHowever, you can avoid performing a full pretraining and instead rely on our short, simple initialization process by running\r\nthe following command:\r\n```\r\npython initialize_occ.py -c occupancy_initialization_config.json\r\n```\r\n\r\nAdditionally, you can edit or create your own custom config file and pretrain an occupancy probability module with it using the following command:\r\n```\r\npython initialize_occ.py -c your_custom_config.json\r\n```\r\n\r\nYou can skip this step by directly downloading the weights files `pretrained_scone_occ.pth` or `initialized_scone_occ.pth`\r\nfrom our [Google Drive](https://drive.google.com/drive/folders/1E9-I7c4Nyx4ljq4z7cc8UQpGQDFM08KI?usp=sharing), \r\nand move it to `./weights/scone/occupancy/`.\r\n\r\n\r\n### 3. Surface coverage gain module\r\n\r\nAfter pretraining an occupancy probability module, \r\nyou can initialize a surface coverage gain neural module by running the following command in the main directory:\r\n\r\n```\r\npython initialize_cov.py\r\n```\r\n\r\nThis command will use the default config file `coverage_gain_pretraining_config.json` located at `./configs/scone/coverage_gain/` \r\nand will perform a full pretraining on ShapeNetCore v1.\u003cbr\u003e\r\nIndeed, to reduce the overall training time of MACARONS and improve performance, we suggest pretraining on ShapeNetCore v1 both the occupancy probability module and the surface coverage gain module, inspired by SCONE.\u003cbr\u003e\r\nPlease note that you should set the field `scone_occ_model_name` in the config file to the name of your pretrained occupancy probability module.\u003cbr\u003e\r\n\r\nTo avoid performing a full pretraining and instead rely on our short, simple initialization process, run\r\nthe following command:\r\n```\r\npython initialize_cov.py -c coverage_gain_initialization_config.json\r\n```\r\n\r\nAdditionally, you can edit or create your own custom config file and pretrain a surface coverage gain module with it using the following command:\r\n```\r\npython initialize_cov.py -c your_custom_config.json\r\n```\r\n\r\nYou can skip this step by directly downloading the weights files `pretrained_scone_vis.pth` or `initialized_scone_vis.pth`\r\nfrom our [Google Drive](https://drive.google.com/drive/folders/1xnWdAvTYnqaWHVc2ghvEEHStsxj3bFjP?usp=sharing), \r\nand move it to `./weights/scone/coverage_gain`.\r\n\r\n### 4. Building full MACARONS model\r\n\r\nAfter initializing all neural modules, you can finally build your ready-to-be-trained full MACARONS model.\u003cbr\u003e\r\n\r\nIf you performed a full pretraining on ShapeNetCore v1 for the occupancy probability module \r\nand surface coverage gain module, run the following command:\r\n\r\n```\r\npython generate_macarons_model.py -p -o your_occupancy_model.pth -c your_coverage_gain_model.pth\r\n```\r\nIt will generate a model file `pretrained_macarons.pth` containing all pretrained weights.\u003cbr\u003e\r\nYou can skip this step by downloading the weights file `pretrained_macarons.pth` \r\ndirectly on our [Google Drive](https://drive.google.com/drive/folders/1V2iJlFLUq5eNmUdPzT9s-kA08_imiMxb?usp=sharing).\r\n\r\nIf you used the much faster, simpler initialization process, run the following command:\r\n\r\n```\r\npython generate_macarons_model.py -o your_occupancy_model.pth -c your_coverage_gain_model.pth\r\n```\r\nIt will generate a model file `initialized_macarons.pth` containing all pretrained weights.\u003cbr\u003e\r\nYou can skip this step by downloading the weights file `initialized_macarons.pth` \r\ndirectly on our [Google Drive](https://drive.google.com/drive/folders/1V2iJlFLUq5eNmUdPzT9s-kA08_imiMxb?usp=sharing).\r\n\r\n## Training the full architecture in large 3D scenes\r\n\r\nTo train a full MACARONS model, select one of the config files with provide in `./configs/macarons/`\r\nor write your own config file and move it to `./configs/macarons`. \r\nDescriptions of all parameters in the config files are provided in `./configs/macarons/`.\u003cbr\u003e\r\nIn particular, the field `pretrained_model_path` of the config file should contain the name\r\nof your initialized MACARONS model.\r\n\r\nThen, run the following command:\r\n\r\n```\r\npython train.py -c your_config_file.py\r\n```\r\n\r\nWe provide two different config files in `./configs/macarons/`.\u003cbr\u003e\r\n\r\n### 1. With pretraining\r\nIf you decided to perform a full pretraining on ShapeNetCore v1 as explained in the previous section, you should use\r\n`macarons_default_training_config.json`. \r\nThis config file is used by default when you run the script `train.py` \r\nwithout specifying any config file, as follows:\r\n\r\n```\r\npython train.py\r\n```\r\n\r\nWith this configuration, the model focuses on comparing successive, neighboring camera viewpoints \r\nalong its trajectory during exploration. This helps the model to build qualitative trajectories.\r\n\r\nYou can skip this step by directly downloading the weights file `trained_macarons.py` \r\nfrom our [Google Drive](https://drive.google.com/drive/folders/1V2iJlFLUq5eNmUdPzT9s-kA08_imiMxb?usp=sharing).\r\n\r\n### 2. Without pretraining\r\n\r\nIf you simply used our faster initialization process, you should run the following command:\r\n\r\n```\r\npython train.py -c macarons_default_training_no_pretraining_config_0.json\r\n```\r\n\r\nWith this configuration, the model focuses on comparing very different camera viewpoints \r\nloaded from the memory. It helps the model to get a high-level understanding of surface coverage gain \r\nwhen no pretraining has been performed.\u003cbr\u003e\r\n\r\nAfter such high-level training, you can run the following command which \r\nfurther improves the model:\r\n\r\n```\r\npython train.py -c macarons_default_training_no_pretraining_config_1.json\r\n```\r\n\r\nThis configuration is almost identical to the configuration used \r\nwhen no pretraining has been performed, and helps the model \r\nto build qualitative trajectories.\r\n\r\nYou can skip these steps by directly downloading the weights file `trained_macarons_no_pretraining.py` \r\nfrom our [Google Drive](https://drive.google.com/drive/folders/1V2iJlFLUq5eNmUdPzT9s-kA08_imiMxb?usp=sharing).\r\n\r\n## Evaluation\r\n\r\n### 1. Large-scale 3D scenes exploration and reconstruction with an RGB camera\r\n\r\nRun the following command to test a full MACARONS model in large-scale 3D scenes:\r\n\r\n```\r\npython test_scenes.py\r\n\r\n```\r\nBy default, the script will use the test config file `test_in_default_scenes_config.json` located in\r\n`./configs/test/`.\u003cbr\u003e\r\n\r\nFeel free to edit the config file or create a new one and run the command with the following option:\r\n\r\n```\r\npython test_scenes.py -c your_test_config_file.json\r\n```\r\n\r\nWe will add a dedicated notebook to visualize the results file as soon as possible.\r\n\r\n### 2. 3D object reconstruction with a depth sensor\r\n\r\nRun the following command to test the pretrained occupancy probability module \r\nand surface coverage gain module, inspired by our previous work SCONE,\r\non ShapeNetCore v1:\r\n\r\n```\r\npython test_shapenet.py\r\n```\r\nBy default, the script will use the test config file `test_on_shapenet_seen_categories_config.json` located in\r\n`./configs/test/`.\u003cbr\u003e\r\nThis script will start a test on ShapeNet objects from categories already seen during training.\u003cbr\u003e\r\n\r\nRun the following command to perform a test on categories not seen during training:\r\n\r\n```\r\npython test_shapenet.py -c test_on_shapenet_novel_categories_config.json\r\n```\r\n\r\nFinally, free to edit the config file or create a new one and run the command with the following option:\r\n\r\n```\r\npython test_scenes.py -c your_test_config_file.json\r\n```\r\n\r\nWe will add a dedicated notebook to visualize the results file as soon as possible.\r\n\r\n## Interactive demo\r\n\r\n### 1. Large-scale 3D scenes exploration and reconstruction with an RGB camera\r\n\r\nGo to `./demo` and run the following command:\r\n```\r\npython macarons_demo.py\r\n```\r\nThe script will run on the GPU device `0` by default.\u003cbr\u003e\r\nYou can use the option `--device` as follows to select the GPU device to use:\r\n\r\n```\r\npython macarons_demo.py -d 1\r\n```\r\n\r\n### 2. 3D object reconstruction with a depth sensor\r\n\r\nGo to `./demo` and run the following command:\r\n```\r\npython scone_demo.py\r\n```\r\nThe script will run on the GPU device `0` by default.\u003cbr\u003e\r\nYou can use the option `--device` as follows to select the GPU device to use:\r\n\r\n```\r\npython scone_demo.py -d 1\r\n```\r\n\r\n## Further information\r\n\r\nWe adapted the code from [Phil Wang](https://github.com/lucidrains/se3-transformer-pytorch/blob/main/se3_transformer_pytorch/spherical_harmonics.py) to generate spherical harmonic features. \u003cbr\u003e\r\nWe thank him for this very useful harmonics computation script! \u003cbr\u003e\r\n\r\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fanttwo%2Fmacarons","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fanttwo%2Fmacarons","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fanttwo%2Fmacarons/lists"}