{"id":15034249,"url":"https://github.com/anttwo/sugar","last_synced_at":"2025-05-15T07:05:12.259Z","repository":{"id":209383983,"uuid":"723915956","full_name":"Anttwo/SuGaR","owner":"Anttwo","description":"[CVPR 2024] Official PyTorch implementation of SuGaR: Surface-Aligned Gaussian Splatting for Efficient 3D Mesh Reconstruction and High-Quality Mesh Rendering","archived":false,"fork":false,"pushed_at":"2024-09-24T15:13:31.000Z","size":100378,"stargazers_count":2678,"open_issues_count":172,"forks_count":213,"subscribers_count":66,"default_branch":"main","last_synced_at":"2025-05-15T07:05:05.120Z","etag":null,"topics":["3d-gaussian-splatting","3dgs","cvpr2024","gaussian-splatting","mesh","mesh-generation","nerf","neural-rendering","surface-reconstruction"],"latest_commit_sha":null,"homepage":"https://anttwo.github.io/sugar/","language":"C++","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"other","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/Anttwo.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE.md","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2023-11-27T02:49:09.000Z","updated_at":"2025-05-15T02:58:25.000Z","dependencies_parsed_at":"2024-01-12T03:37:47.549Z","dependency_job_id":"25fdd391-5aed-4052-8206-c2cc7607c98f","html_url":"https://github.com/Anttwo/SuGaR","commit_stats":{"total_commits":31,"total_committers":2,"mean_commits":15.5,"dds":"0.12903225806451613","last_synced_commit":"7c10c4ae4a267dece512f5c7f40ed212a0a2ab44"},"previous_names":["anttwo/sugar"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Anttwo%2FSuGaR","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Anttwo%2FSuGaR/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Anttwo%2FSuGaR/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Anttwo%2FSuGaR/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/Anttwo","download_url":"https://codeload.github.com/Anttwo/SuGaR/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":254292040,"owners_count":22046426,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["3d-gaussian-splatting","3dgs","cvpr2024","gaussian-splatting","mesh","mesh-generation","nerf","neural-rendering","surface-reconstruction"],"created_at":"2024-09-24T20:24:22.849Z","updated_at":"2025-05-15T07:05:07.245Z","avatar_url":"https://github.com/Anttwo.png","language":"C++","funding_links":[],"categories":[],"sub_categories":[],"readme":"\u003cdiv align=\"center\"\u003e\n\n# SuGaR: Surface-Aligned Gaussian Splatting for Efficient 3D Mesh Reconstruction and High-Quality Mesh Rendering\n\n\u003cfont size=\"4\"\u003e\nCVPR 2024\n\u003c/font\u003e\n\u003cbr\u003e\n\n\u003cfont size=\"4\"\u003e\n\u003ca href=\"https://anttwo.github.io/\" style=\"font-size:100%;\"\u003eAntoine Guédon\u003c/a\u003e\u0026emsp;\n\u003ca href=\"https://vincentlepetit.github.io/\" style=\"font-size:100%;\"\u003eVincent Lepetit\u003c/a\u003e\u0026emsp;\n\u003c/font\u003e\n\u003cbr\u003e\n\n\u003cfont size=\"4\"\u003e\nLIGM, Ecole des Ponts, Univ Gustave Eiffel, CNRS\n\u003c/font\u003e\n\n| \u003ca href=\"https://anttwo.github.io/sugar/\"\u003eWebpage\u003c/a\u003e | \u003ca href=\"https://arxiv.org/abs/2311.12775\"\u003earXiv\u003c/a\u003e | \u003ca href=\"https://github.com/Anttwo/sugar_frosting_blender_addon/\"\u003eBlender add-on\u003c/a\u003e | \u003ca href=\"https://www.youtube.com/watch?v=MAkFyWfiBQo\"\u003ePresentation video\u003c/a\u003e | \u003ca href=\"https://www.youtube.com/watch?v=YbjE0wnw67I\"\u003eViewer video\u003c/a\u003e |\n\n\u003cimg src=\"./media/examples/walk.gif\" alt=\"walk.gif\" width=\"350\"/\u003e\u003cimg src=\"./media/examples/attack.gif\" alt=\"attack.gif\" width=\"350\"/\u003e \u003cbr\u003e\n\u003cb\u003eOur method extracts meshes from 3D Gaussian Splatting reconstructions and builds hybrid representations \u003cbr\u003ethat enable easy composition and animation in Gaussian Splatting scenes by manipulating the mesh.\u003c/b\u003e\n\u003c/div\u003e\n\n## Abstract\n\n_We propose a method to allow precise and extremely fast mesh extraction from \u003ca href=\"https://repo-sam.inria.fr/fungraph/3d-gaussian-splatting/\"\u003e3D Gaussian Splatting (SIGGRAPH 2023)\u003c/a\u003e.\nGaussian Splatting has recently become very popular as it yields realistic rendering while being significantly faster to train than NeRFs. It is however challenging to extract a mesh from the millions of tiny 3D Gaussians as these Gaussians tend to be unorganized after optimization and no method has been proposed so far.\nOur first key contribution is a regularization term that encourages the 3D Gaussians to align well with the surface of the scene.\nWe then introduce a method that exploits this alignment to sample points on the real surface of the scene and extract a mesh from the Gaussians using Poisson reconstruction, which is fast, scalable, and preserves details, in contrast to the Marching Cubes algorithm usually applied to extract meshes from Neural SDFs.\nFinally, we introduce an optional refinement strategy that binds Gaussians to the surface of the mesh, and jointly optimizes these Gaussians and the mesh through Gaussian splatting rendering. This enables easy editing, sculpting, rigging, animating, or relighting of the Gaussians using traditional softwares (Blender, Unity, Unreal Engine, etc.) by manipulating the mesh instead of the Gaussians themselves.\nRetrieving such an editable mesh for realistic rendering is done within minutes with our method, compared to hours with the state-of-the-art method on neural SDFs, while providing a better rendering quality in terms of PSNR, SSIM and LPIPS._\n\n\u003cdiv align=\"center\"\u003e\n\u003cb\u003eHybrid representation (Mesh + Gaussians on the surface)\u003c/b\u003e\u003cbr\u003e\n\u003cimg src=\"./media/overview/garden_hybrid.gif\" alt=\"garden_hybrid.gif\" width=\"250\"/\u003e\n\u003cimg src=\"./media/overview/kitchen_hybrid.gif\" alt=\"kitchen_hybrid.gif\" width=\"250\"/\u003e\n\u003cimg src=\"./media/overview/counter_hybrid.gif\" alt=\"counter_hybrid.gif\" width=\"250\"/\u003e\u003cbr\u003e\n\u003cimg src=\"./media/overview/playroom_hybrid.gif\" alt=\"playroom_hybrid.gif\" width=\"323\"/\u003e\n\u003cimg src=\"./media/overview/qant03_hybrid.gif\" alt=\"qant03_hybrid.gif\" width=\"323\"/\u003e\n\u003cimg src=\"./media/overview/dukemon_hybrid.gif\" alt=\"_hybrid.gif\" width=\"102\"/\u003e\u003cbr\u003e\n\u003cb\u003eUnderlying mesh without texture\u003c/b\u003e\u003cbr\u003e\n\u003cimg src=\"./media/overview/garden_notex.gif\" alt=\"garden_notex.gif\" width=\"250\"/\u003e\n\u003cimg src=\"./media/overview/kitchen_notex.gif\" alt=\"kitchen_notex.gif\" width=\"250\"/\u003e\n\u003cimg src=\"./media/overview/counter_notex.gif\" alt=\"counter_notex.gif\" width=\"250\"/\u003e\u003cbr\u003e\n\u003cimg src=\"./media/overview/playroom_notex.gif\" alt=\"playroom_notex.gif\" width=\"323\"/\u003e\n\u003cimg src=\"./media/overview/qant03_notex.gif\" alt=\"qant03_notex.gif\" width=\"323\"/\u003e\n\u003cimg src=\"./media/overview/dukemon_notex.gif\" alt=\"dukemon_notex.gif\" width=\"102\"/\u003e\u003cbr\u003e\n\u003c/div\u003e\n\n\n## BibTeX\n\n```\n@article{guedon2023sugar,\n  title={SuGaR: Surface-Aligned Gaussian Splatting for Efficient 3D Mesh Reconstruction and High-Quality Mesh Rendering},\n  author={Gu{\\'e}don, Antoine and Lepetit, Vincent},\n  journal={CVPR},\n  year={2024}\n}\n```\n\n## Updates and To-do list\n\n\u003cdetails\u003e\n\u003csummary\u003e\u003cspan style=\"font-weight: bold;\"\u003eUpdates\u003c/span\u003e\u003c/summary\u003e\n\u003cul\u003e\n  \u003cli\u003e\u003cb\u003e[09/18/2024]\u003c/b\u003e Improved the quality of the extracted meshes with the new `dn_consistency` regularization method, and added compatibility with the new Blender add-on for composition and animation. \u003c/li\u003e\n  \u003cli\u003e\u003cb\u003e[01/09/2024]\u003c/b\u003e Added a dedicated, real-time viewer to let users visualize and navigate in the reconstructed scenes (hybrid representation, textured mesh and wireframe mesh).\u003c/li\u003e\n  \u003cli\u003e\u003cb\u003e[12/20/2023]\u003c/b\u003e Added a short notebook showing how to render images with the hybrid representation using the Gaussian Splatting rasterizer.\u003c/li\u003e\n  \u003cli\u003e\u003cb\u003e[12/18/2023]\u003c/b\u003e Code release.\u003c/li\u003e\n\u003c/ul\u003e\n\u003c/details\u003e\u003cbr\u003e\n\n\u003cdetails\u003e\n\u003csummary\u003e\u003cspan style=\"font-weight: bold;\"\u003eTo-do list\u003c/span\u003e\u003c/summary\u003e\n\u003cul\u003e\n  \u003cli\u003e\u003cb\u003eViewer:\u003c/b\u003e Add option to load the postprocessed mesh.\u003c/li\u003e\n  \u003cli\u003e\u003cb\u003eMesh extraction:\u003c/b\u003e Add the possibility to edit the extent of the background bounding box.\u003c/li\u003e\n  \u003cli\u003e\u003cb\u003eTips\u0026Tricks:\u003c/b\u003e Add to the README.md file (and the webpage) some tips and tricks for using SuGaR on your own data and obtain better reconstructions (see the tips provided by user kitmallet).\u003c/li\u003e\n  \u003cli\u003e\u003cb\u003eImprovement:\u003c/b\u003e Add an \u003ccode\u003eif\u003c/code\u003e block to \u003ccode\u003esugar_extractors/coarse_mesh.py\u003c/code\u003e to skip foreground mesh reconstruction and avoid triggering an error if no surface point is detected inside the foreground bounding box. This can be useful for users that want to reconstruct \"\u003ci\u003ebackground scenes\u003c/i\u003e\". \u003c/li\u003e\n  \u003cli\u003e\u003cb\u003eUsing precomputed masks with SuGaR:\u003c/b\u003e Add a mask functionality to the SuGaR optimization, to allow the user to mask out some pixels in the training images (like white backgrounds in synthetic datasets).\n  \u003c/li\u003e\n  \u003cli\u003e\u003cb\u003eUsing SuGaR with Windows:\u003c/b\u003e Adapt the code to make it compatible with Windows. Due to path-writing conventions, the current code is not compatible with Windows. \n  \u003c/li\u003e\n  \u003cli\u003e\u003cb\u003eSynthetic datasets:\u003c/b\u003e Add the possibility to use the NeRF synthetic dataset (which has a different format than COLMAP scenes)\n  \u003c/li\u003e\n  \u003cli\u003e\u003cb\u003eComposition and animation:\u003c/b\u003e Finish to clean the code for composition and animation, and add it to the \u003ccode\u003esugar_scene/sugar_compositor.py\u003c/code\u003e script.\n  \u003c/li\u003e\n  \u003cli\u003e\u003cb\u003eComposition and animation:\u003c/b\u003e Make a tutorial on how to use the scripts in the \u003ccode\u003eblender\u003c/code\u003e directory and the \u003ccode\u003esugar_scene/sugar_compositor.py\u003c/code\u003e class to import composition and animation data into PyTorch and apply it to the SuGaR hybrid representation.\n  \u003c/li\u003e\n  \u003c!-- \u003cli\u003e\u003cb\u003eImprovement:\u003c/b\u003e Implement a simple method to avoid artifacts when reconstructing thin objects with poor coverage/visibility in the training images.\u003c/li\u003e\n  \u003c/li\u003e --\u003e\n\u003c/ul\u003e\n\u003c/details\u003e\n\n## Overview\n\nAs we explain in the paper, SuGaR optimization starts with first optimizing a 3D Gaussian Splatting model for 7k iterations with no additional regularization term. Consequently, the current implementation contains a version of the original \u003ca href=\"https://github.com/graphdeco-inria/gaussian-splatting\"\u003e3D Gaussian Splatting code\u003c/a\u003e, and we built our model as a wrapper of a vanilla 3D Gaussian Splatting model.\nPlease note that, even though this wrapper implementation is convenient for many reasons, it may not be the most optimal one for memory usage.\n\nThe full SuGaR pipeline consists of 4 main steps, and an optional one:\n1. **Short vanilla 3DGS optimization**: optimizing a vanilla 3D Gaussian Splatting model for 7k iterations, in order to let Gaussians position themselves in the scene.\n2. **SuGaR optimization**: optimizing Gaussians alignment with the surface of the scene.\n3. **Mesh extraction**: extracting a mesh from the optimized Gaussians.\n4. **SuGaR refinement**: refining the Gaussians and the mesh together to build a hybrid Mesh+Gaussians representation.\n5. **Textured mesh extraction (Optional)**: extracting a traditional textured mesh from the refined SuGaR model as a tool for visualization, composition and animation in Blender using our \u003ca href=\"https://github.com/Anttwo/sugar_frosting_blender_addon/\"\u003eBlender add-on\u003c/a\u003e.\n\nWe provide a dedicated script for each of these steps, as well as a script `train_full_pipeline.py` that runs the entire pipeline. We explain how to use this script in the next sections. \u003cbr\u003e\n\n\u003cdiv align=\"center\"\u003e\u003cbr\u003e\n\u003cimg src=\"./media/blender/blender_edit.png\" alt=\"blender_edit.png\" height=\"200\"/\u003e\n\u003cimg src=\"./media/examples/attack.gif\" alt=\"attack.gif\" height=\"200\"/\u003e\n\u003cbr\u003e\u003cb\u003eYou can visualize, edit, combine or animate the reconstructed textured meshes in Blender \u003ci\u003e(left)\u003c/i\u003e \u003cbr\u003eand render the result with SuGaR \u003ci\u003e(right)\u003c/i\u003e thanks to our \u003ca href=\"https://github.com/Anttwo/sugar_frosting_blender_addon/\"\u003eBlender add-on\u003c/a\u003e.\u003c/b\u003e\u003cbr\u003e\n\u003c/div\u003e\u003cbr\u003e\n\nPlease note that the final step, _Textured mesh extraction_, is optional but is enabled by default in the `train_full_pipeline.py` script. Indeed, it is very convenient to have a traditional textured mesh for visualization, composition and animation using traditional softwares such as \u003ca href=\"https://github.com/Anttwo/sugar_frosting_blender_addon/\"\u003eBlender\u003c/a\u003e. If you installed Nvdiffrast as described below, this step should only take a few seconds anyway.\n\n\u003cdiv align=\"center\"\u003e\n\u003cb\u003eHybrid representation (Mesh + Gaussians on the surface)\u003c/b\u003e\u003cbr\u003e\n\u003cimg src=\"./media/overview/garden_hybrid.png\" alt=\"garden_hybrid.gif\" height=\"135\"/\u003e\n\u003cimg src=\"./media/overview/kitchen_hybrid.png\" alt=\"kitchen_hybrid.gif\" height=\"135\"/\u003e\n\u003cimg src=\"./media/overview/qant03_hybrid.png\" alt=\"qant03_hybrid.gif\" height=\"135\"/\u003e\n\u003cimg src=\"./media/overview/dukemon_hybrid.png\" alt=\"_hybrid.gif\" height=\"135\"/\u003e\u003cbr\u003e\n\u003cb\u003eUnderlying mesh with a traditional colored UV texture\u003c/b\u003e\u003cbr\u003e\n\u003cimg src=\"./media/overview/garden_texture.png\" alt=\"garden_notex.gif\" height=\"135\"/\u003e\n\u003cimg src=\"./media/overview/kitchen_texture.png\" alt=\"kitchen_notex.gif\" height=\"135\"/\u003e\n\u003cimg src=\"./media/overview/qant03_texture.png\" alt=\"qant03_notex.gif\" height=\"135\"/\u003e\n\u003cimg src=\"./media/overview/dukemon_texture.png\" alt=\"dukemon_notex.gif\" height=\"135\"/\u003e\u003cbr\u003e\n\u003c/div\u003e\u003cbr\u003e\n\nBelow is another example of a scene showing a robot with a black and specular material. The following images display the hybrid representation (Mesh + Gaussians on the surface), the mesh with a traditional colored UV texture, and a depth map of the mesh:\n\u003cdiv align=\"center\"\u003e\n\u003cb\u003eHybrid representation - Textured mesh - Depth map of the mesh\u003c/b\u003e\u003cbr\u003e\n\u003cimg src=\"./media/examples/alpha_hybrid.png\" alt=\"alpha_hybrid.png\" height=\"400\"/\u003e\n\u003cimg src=\"./media/examples/alpha_texture.png\" alt=\"alpha_texture.gif\" height=\"400\"/\u003e\n\u003cimg src=\"./media/examples/alpha_depth.png\" alt=\"alpha_depth.gif\" height=\"400\"/\u003e\n\u003c/div\u003e\n\n## Installation\n\n\u003cdetails\u003e\n\u003csummary\u003e\u003cspan style=\"font-weight: bold;\"\u003eClick here to see content.\u003c/span\u003e\u003c/summary\u003e\n\n### 0. Requirements\n\nThe software requirements are the following:\n- Conda (recommended for easy setup)\n- C++ Compiler for PyTorch extensions\n- CUDA toolkit 11.8 for PyTorch extensions\n- C++ Compiler and CUDA SDK must be compatible\n\nPlease refer to the original \u003ca href=\"https://github.com/graphdeco-inria/gaussian-splatting\"\u003e3D Gaussian Splatting repository\u003c/a\u003e for more details about requirements.\n\n### 1. Clone the repository\n\nStart by cloning this repository:\n\n```shell\n# HTTPS\ngit clone https://github.com/Anttwo/SuGaR.git --recursive\n```\n\nor\n\n```shell\n# SSH\ngit clone git@github.com:Anttwo/SuGaR.git --recursive\n```\n\n### 2. Creating the Conda environment\n\nTo create and activate the Conda environment with all the required packages, go inside the `SuGaR/` directory and run the following command:\n\n```shell\npython install.py\nconda activate sugar\n```\n\nThis script will automatically create a Conda environment named `sugar` and install all the required packages. It will also automatically install the \u003ca href=\"https://github.com/graphdeco-inria/gaussian-splatting\"\u003e3D Gaussian Splatting\u003c/a\u003e rasterizer as well as the \u003ca href=\"https://nvlabs.github.io/nvdiffrast/\"\u003eNvdiffrast\u003c/a\u003e library for faster mesh rasterization.\n\nIf you encounter any issues with the installation, you can try to follow the detailed instructions below to install the required packages manually.\n\n\u003cdetails\u003e\n\u003csummary\u003e\u003cspan style=\"font-weight: bold;\"\u003e\nDetailed instructions for manual installation\n\u003c/span\u003e\u003c/summary\u003e\n\n#### a) Install the required Python packages\nTo install the required Python packages and activate the environment, go inside the `SuGaR/` directory and run the following commands:\n\n```shell\nconda env create -f environment.yml\nconda activate sugar\n```\n\nIf this command fails to create a working environment, you can try to install the required packages manually by running the following commands:\n```shell\nconda create --name sugar -y python=3.9\nconda activate sugar\nconda install pytorch==2.0.1 torchvision==0.15.2 torchaudio==2.0.2 pytorch-cuda=11.8 -c pytorch -c nvidia\nconda install -c fvcore -c iopath -c conda-forge fvcore iopath\nconda install pytorch3d==0.7.4 -c pytorch3d\nconda install -c plotly plotly\nconda install -c conda-forge rich\nconda install -c conda-forge plyfile==0.8.1\nconda install -c conda-forge jupyterlab\nconda install -c conda-forge nodejs\nconda install -c conda-forge ipywidgets\npip install open3d\npip install --upgrade PyMCubes\n```\n\n#### b) Install the Gaussian Splatting rasterizer\n\nRun the following commands inside the `SuGaR` directory to install the additional Python submodules required for Gaussian Splatting:\n\n```shell\ncd gaussian_splatting/submodules/diff-gaussian-rasterization/\npip install -e .\ncd ../simple-knn/\npip install -e .\ncd ../../../\n```\nPlease refer to the \u003ca href=\"https://github.com/graphdeco-inria/gaussian-splatting\"\u003e3D Gaussian Splatting repository\u003c/a\u003e for more details.\n\n#### c) (Optional) Install Nvdiffrast for faster Mesh Rasterization\n\nInstalling Nvdiffrast is optional but will greatly speed up the textured mesh extraction step, from a few minutes to less than 10 seconds.\n\n```shell\ngit clone https://github.com/NVlabs/nvdiffrast\ncd nvdiffrast\npip install .\ncd ../\n```\n\n\u003c/details\u003e\n\n\u003c/details\u003e\n\n\n## Quick Start\n\n\u003cdetails\u003e\n\u003csummary\u003e\u003cspan style=\"font-weight: bold;\"\u003eClick here to see content.\u003c/span\u003e\u003c/summary\u003e\n\n### Training from scratch\n\nYou can run the following single script to optimize a full SuGaR model from scratch using a COLMAP dataset:\n\n```shell\npython train_full_pipeline.py -s \u003cpath to COLMAP dataset\u003e -r \u003c\"dn_consistency\", \"density\" or \"sdf\"\u003e --high_poly True --export_obj True\n```\n\nYou can choose the regularization method with the `-r` argument, which can be `\"dn_consistency\"`, `\"density\"` or `\"sdf\"`. We recommend using the newer `\"dn_consistency\"` regularization for best quality meshes, but the results presented in the paper were obtained with the `\"density\"` regularization for object-centered scenes and `\"sdf\"` for scenes with a challenging background, such as the Mip-NeRF 360 dataset.\n\nYou can also replace the `--high_poly True` argument with `--low_poly True` to extract a mesh with 200k vertices instead of 1M, and 6 Gaussians per triangle instead of 1.\n\nMoreover, you can add `--refinement_time \"short\"`, `\"medium\"` or `\"long\"` to set the time spent on the refinement step. The default is `\"long\"` (15k iterations), but `\"short\"` (2k iterations) can be enough to produce a good-looking hybrid representation.\n\nFinally, you can choose to export a traditional textured mesh with the `--export_obj` argument. This step is optional but is enabled by default in the `train_full_pipeline.py` script, as the mesh is required for using the \u003ca href=\"https://github.com/Anttwo/sugar_frosting_blender_addon/\"\u003eBlender add-on\u003c/a\u003e and editing, combining or animating scenes in Blender.\n\nResults are saved in the `output/` directory.\n\n\u003cdetails\u003e\n\u003csummary\u003e\u003cspan style=\"font-weight: bold;\"\u003ePlease click here to see the most important arguments for the `train_full_pipeline.py` script.\u003c/span\u003e\u003c/summary\u003e\n\n| Parameter | Type | Description |\n| :-------: | :--: | :---------: |\n| `--scene_path` / `-s`   | `str` | Path to the source directory containing a COLMAP dataset.|\n| `--gs_output_dir` | `str` | Path to the checkpoint directory of a vanilla 3D Gaussian Splatting model. If no path is provided, the script will start from scratch and first optimize a vanilla 3DGS model. |\n| `--regularization_type` / `-r` | `str` | Type of regularization to use for aligning Gaussians. Can be `\"dn_consistency\"`, `\"density\"` or `\"sdf\"`. We recommend using the newer `\"dn_consistency\"` regularization for best quality meshes. |\n| `--eval` | `bool` | If True, performs an evaluation split of the training images. Default is `True`. |\n| `--low_poly` | `bool` | If True, uses the standard config for a low poly mesh, with `200_000` vertices and `6` Gaussians per triangle. |\n| `--high_poly` | `bool` | If True, uses the standard config for a high poly mesh, with `1_000_000` vertices and `1` Gaussian per triangle. |\n| `--refinement_time` | `str` | Default configs for time to spend on refinement. Can be `\"short\"` (2k iterations), `\"medium\"` (7k iterations) or `\"long\"` (15k iterations). |\n| `--export_ply` | `bool` | If True, export a `.ply` file with the refined 3D Gaussians at the end of the training. This file can be large (+/- 500MB), but is needed for using 3DGS viewers. Default is `True`. |\n| `--export_obj` / `-t` | `bool` | If True, will optimize and export a traditional textured mesh as an `.obj` file from the refined SuGaR model, after refinement. Computing a traditional color UV texture should just take a few seconds with Nvdiffrast. Default is `True`. |\n| `--square_size` | `int` | Size of the square allocated to each pair of triangles in the UV texture. Increase for higher texture resolution. Please decrease if you encounter memory issues. Default is `8`. |\n|`--white_background` | `bool` | If True, the background of the images will be set to white. Default is `False`. |\n\n\n\u003c/details\u003e\n\u003cbr\u003e\n\nAs we explain in the paper, this script extracts a mesh in 30 minutes on average on a single GPU. After mesh extraction, the refinement time only takes a few minutes when using `--refinement_time \"short\"`, but can be much longer when using `--refinement_time \"long\"`. A short refinement time is enough to produce a good-looking hybrid representation in most cases.\n\nPlease note that the optimization time may vary depending on the complexity of the scene and the GPU used. Moreover, the current implementation splits the optimization into modular scripts that can be run separately so it reloads the data at each part, which is not optimal and takes several minutes.\n\nPlease see the `train_full_pipeline.py` for more details on all the command line arguments.\n\n### Training from a vanilla Gaussian Splatting model\n\nIf you have already trained a \u003ca href=\"https://github.com/graphdeco-inria/gaussian-splatting\"\u003evanilla Gaussian Splatting model\u003c/a\u003e for a scene (we recommend training it for 7k iterations), you can use the `--gs_output_dir` argument to provide the path to the output directory of the vanilla Gaussian Splatting model. This will skip the first part of the optimization and directly load the Gaussians from the vanilla model:\n\n```shell\npython train_full_pipeline.py -s \u003cpath to COLMAP dataset\u003e -r \u003c\"dn_consistency\", \"density\" or \"sdf\"\u003e --high_poly True --export_obj True --gs_output_dir \u003cpath to the Gaussian Splatting output directory\u003e\n```\n\n\u003c/details\u003e\n\n\n## Visualize a SuGaR model in real-time\n\n\u003cdetails\u003e\n\u003csummary\u003e\u003cspan style=\"font-weight: bold;\"\u003eClick here to see content.\u003c/span\u003e\u003c/summary\u003e\u003cbr\u003e\n\nAfter optimizing a SuGaR model, you can visualize the model in real-time using the 3D Gaussian Splatting viewer of your choice. \u003cbr\u003e\nIndeed, after optimization, we automatically export a `.ply` file in the `./output/refined_ply/` directory, containing the refined 3D Gaussians of SuGaR's hybrid representation and compatible with any 3DGS viewer.\nFor instance, you can use the viewer provided in the original implementation of \u003ca href=\"https://github.com/graphdeco-inria/gaussian-splatting\"\u003e3D Gaussian Splatting\u003c/a\u003e, or the awesome \u003ca href=\"https://github.com/playcanvas/supersplat\"\u003eSuperSplat viewer\u003c/a\u003e. \u003cbr\u003e\nAn online, \u003ca href=\"https://playcanvas.com/supersplat/editor\"\u003ein-browser version of SuperSplat\u003c/a\u003e is also available.\n\nWe also propose a dedicated real-time viewer that allows you to visualize not only the refined 3D Gaussians but also the textured mesh and the wireframe mesh. Please see the instructions below to install and use this viewer.\n\n\u003cdetails\u003e\n\u003csummary\u003e\u003cspan style=\"font-weight: bold;\"\u003ePlease click here to see the instructions for installing and using our real-time viewer.\u003c/span\u003e\u003c/summary\u003e\n\nPlease find \u003ca href=\"https://www.youtube.com/watch?v=YbjE0wnw67I\"\u003ehere\u003c/a\u003e a short video illustrating how to use the viewer.\n\n### 1. Installation\n\n*The viewer is currently built for Linux and Mac OS. It is not compatible with Windows. For Windows users, we recommend to use WSL2 (Windows Subsystem for Linux), as it is very easy to install and use. Please refer to the \u003ca href=\"https://docs.microsoft.com/en-us/windows/wsl/install-win10\"\u003eofficial documentation\u003c/a\u003e for more details.\u003cbr\u003e We thank \u003ca href=\"https://github.com/mkkellogg/GaussianSplats3D\"\u003eMark Kellogg for his awesome 3D Gaussian Splatting implementation for Three.js\u003c/a\u003e, which we used for building this viewer.*\n\nPlease start by installing the latest versions of Node.js (such as 21.x) and npm.\nA simple way to do this is to run the following commands (using aptitude):\n\n```shell\ncurl -fsSL https://deb.nodesource.com/setup_21.x | sudo -E bash -\nsudo apt-get install -y nodejs\nsudo apt-get install aptitude\nsudo aptitude install -y npm\n```\n\nThen, go inside the `./sugar_viewer/` directory and run the following commands:\n\n```shell\nnpm install\ncd ..\n```\n\n### 2. Usage\n\nFirst, make sure you have exported a `.ply` file and an `.obj` file using the `train.py` script. The `.ply` file contains the refined 3D Gaussians, and the `.obj` file contains the textured mesh. These files are exported by default when running the `train.py` script, so if you ran the code with default values for `--export_ply` and `--export_obj`, you should be good to go.\n\nThe ply file should be located in `./output/refined_ply/\u003cyour scene name\u003e/`. Then, just run the following command in the root directory to start the viewer:\n\n```shell\npython run_viewer.py -p \u003cpath to the .ply file\u003e\n```\nPlease make sure your `.ply` file is located in the right folder, and use a relative path starting with `./output/refined_ply`.\nThis command will redirect you to a local URL. Click on the link to open the viewer in your browser. Click the icons on the top right to switch between the different representations (hybrid representation, textured mesh, wireframe mesh). Use the mouse to rotate the scene, and the mouse wheel to zoom in and out. \n\n\u003c/details\u003e\n\n\u003cdiv align=\"center\" \u003e\n\u003cimg src=\"./media/examples/viewer_example.png\" alt=\"viewer_example.png\" width=\"800\"/\u003e\n\u003c/div\u003e\u003cbr\u003e\n\nWe also recommend using our \u003ca href=\"https://github.com/Anttwo/sugar_frosting_blender_addon/\"\u003eBlender add-on\u003c/a\u003e to create animations and video clips of SuGaR representations. \nMore specifically, the add-on allows you to import SuGaR meshes and visualize, edit, combine or animate them in Blender.\nFinally, you can render the result using the 3DGS rasterizer, which provides high-quality and realistic rendering of SuGaR's hybrid representation.\n\n\u003c/details\u003e\n\n\n## Rendering, composition and animation with Blender\n\n\u003cdetails\u003e\n\u003csummary\u003e\u003cspan style=\"font-weight: bold;\"\u003eClick here to see content.\u003c/span\u003e\u003c/summary\u003e\u003cbr\u003e\n\nThe `view_sugar_results.ipynb` notebook and the `metrics.py` script provides examples of how to load a refined SuGaR model for rendering a scene.\n\nWe also provide a \u003ca href=\"https://github.com/Anttwo/sugar_frosting_blender_addon/\"\u003eBlender add-on\u003c/a\u003e for editing, combining, and animating SuGaR meshes within Blender and render the result using SuGaR's hybrid representations. Meshes are located in the `./output/refined_mesh/` directory.\n\nPlease refer to the Blender add-on repository for more details on how to use the add-on and create a rendering package for SuGaR.\nAfter preparing the rendering package with Blender, which should be a `.JSON` file located in the `./output/blender/package/` directory, you can render the scene using the `render_blender_scene.py` script:\n\n```shell\npython render_blender_scene.py -p \u003cPath to the rendering package\u003e\n```\n\nThe rendered images will be saved in the `./output/blender/renders/` directory.\u003cbr\u003e\nFeel free to adjust the arguments of the script to adjust the rendering quality if you observe artifacts in the images.\nSpecifically, you can switch to `--adaptation_method simple` or reduce `deformation_threshold` to mitigate artifacts in the rendering.\nPlease refer to the script for more details on the command line arguments.\n\n\u003c/details\u003e\n\n\n## Evaluation\n\n\u003cdetails\u003e\n\u003csummary\u003e\u003cspan style=\"font-weight: bold;\"\u003eClick here to see content.\u003c/span\u003e\u003c/summary\u003e\u003cbr\u003e\n\nTo evaluate the quality of the reconstructions, we provide a script `metrics.py` that computes the PSNR, SSIM and LPIPS metrics on test images. Start by optimizing SuGaR models for the desired scenes and a regularization method (`\"dn_consistency\"`, `\"density\"` or `\"sdf\"`), then create a `.json` config file containing the paths to the scenes in the following format: `{source_images_dir_path: vanilla_gaussian_splatting_checkpoint_path}`.\n\nFinally, run the script as follows:\n\n```shell\npython metrics.py --scene_config \u003cPath to the .json file\u003e -r \u003c\"sdf\" or \"density\"\u003e \n```\n\nResults are saved in a `.json` file in the `output/metrics/` directory. \nPlease refer to the script for more details on the command line arguments.\n\n\u003c/details\u003e\n\n## Tips for using SuGaR on your own data and obtain better reconstructions\n\n\u003cdetails\u003e\n\u003csummary\u003e\u003cspan style=\"font-weight: bold;\"\u003eClick here to see content.\u003c/span\u003e\u003c/summary\u003e\u003cbr\u003e\n\n\u003cdetails\u003e\n\u003csummary\u003e\u003cspan style=\"font-weight: bold;\"\u003e1. Capture images or videos that cover the entire surface of the scene\u003c/span\u003e\u003c/summary\u003e\u003cbr\u003e\n\nUsing a smartphone or a camera, capture images or a video that cover the entire surface of the 3D scene you want to reconstruct. The easiest way to do this is to move around the scene while recording a video. Try to move slowly and smoothly in order to avoid motion blur. For consistent reconstruction and easier camera pose estimation with COLMAP, maintaining a uniform focal length and a constant exposure time is also important. We recommend to disable auto-focus on your smartphone to ensure that the focal length remains constant.\n\nFor better reconstructions, try to cover objects from several and different angles, especially for thin and detailed parts of the scene. \nIndeed, SuGaR is able to reconstruct very thin and detailed objects, but some artifacts may appear if these thin objects are not covered enough and are visible only from one side in the training images.\n\nTo convert a video to images, you can install `ffmpeg` and run the following command:\n```shell\nffmpeg -i \u003cPath to the video file\u003e -qscale:v 1 -qmin 1 -vf fps=\u003cFPS\u003e %04d.jpg\n```\nwhere `\u003cFPS\u003e` is the desired sampling rate of the video images. An FPS value of 1 corresponds to sampling one image per second. We recommend to adjust the sampling rate to the length of the video, so that the number of sampled images is between 100 and 300.\n\n\u003c/details\u003e\n\n\u003cdetails\u003e\n\u003csummary\u003e\u003cspan style=\"font-weight: bold;\"\u003e2. Estimate camera poses with COLMAP\u003c/span\u003e\u003c/summary\u003e\u003cbr\u003e\n\nPlease first install a recent version of COLMAP (ideally CUDA-powered) and make sure to put the images you want to use in a directory `\u003clocation\u003e/input`. Then, run the script `gaussian_splatting/convert.py` from the original Gaussian splatting implementation to compute the camera poses from the images using COLMAP. Please refer to the original \u003ca href=\"https://github.com/graphdeco-inria/gaussian-splatting\"\u003e3D Gaussian Splatting repository\u003c/a\u003e for more details.\n\n```shell\npython gaussian_splatting/convert.py -s \u003clocation\u003e\n```\n\nSometimes COLMAP fails to reconstruct all images into the same model and hence produces multiple sub-models. The smaller sub-models generally contain only a few images. However, by default, the script `convert.py` will apply Image Undistortion only on the first sub-model, which may contain only a few images.\n\nIf this is the case, a simple solution is to keep only the largest sub-model and discard the others. To do this, open the source directory containing your input images, then open the sub-directory `\u003cSource_directory\u003e/distorted/sparse/`. You should see several sub-directories named `0/`, `1/`, etc., each containing a sub-model. Remove all sub-directories except the one containing the largest files, and rename it to `0/`. Then, run the script `convert.py` one more time but skip the matching process:\n\n```shell\npython gaussian_splatting/convert.py -s \u003clocation\u003e --skip_matching\n```\n\n_Note: If the sub-models have common registered images, they could be merged into a single model as post-processing step using COLMAP; However, merging sub-models requires to run another global bundle adjustment after the merge, which can be time consuming._\n\u003c/details\u003e\n\n\n\u003cdetails\u003e\n\u003csummary\u003e\u003cspan style=\"font-weight: bold;\"\u003e3. DN-Consistency, Density or SDF? Choose a regularization method that fits your scene\u003c/span\u003e\u003c/summary\u003e\u003cbr\u003e\n\n**We recommend using the newer DN-Consistency regularization for best quality meshes**.\n\nHowever, the results presented in the paper were obtained with the Density regularization for object-centered scenes and the SDF regularization for scenes with a challenging background, such as the Mip-NeRF 360 dataset.\n\nAs we explain in the paper, the density regularization is the simplest one and works well with objects centered in the scene. The SDF provides a stronger regularization, especially in background regions. \nAs a consequence, the SDF regularization produces higher metrics on standard datasets. \nHowever, for reconstructing an object centered in the scene with images taken from all around the object, the simpler density regularization generally produces a better mesh than SDF.\n\nThe DN-Consistency regularization is a new regularization method that (a) enforces the Gaussians to align with the surface of the scene with the density regularization, while also (b) enforcing the consistency between the gradient of the depth and the normal maps, all rendered using the 3D Gaussians. \u003cbr\u003e\nAs described in the paper, the normal of a Gaussian is defined as the shortest axis of the covariance matrix of the Gaussian. \u003cbr\u003e \nThis regularization method provides the best quality meshes.\n\n\u003c/details\u003e\n\n\u003cdetails\u003e\n\u003csummary\u003e\u003cspan style=\"font-weight: bold;\"\u003e4. I have holes in my mesh, what can I do?\u003c/span\u003e\u003c/summary\u003e\u003cbr\u003e\n\nIf you have holes in your mesh, this means the cleaning step of the Poisson mesh is too aggressive for your scene. You can reduce the treshold `vertices_density_quantile` used for cleaning by modifying line 43 of `sugar_extractors/coarse_mesh.py`. For example, you can change this line from\n```python\n  vertices_density_quantile = 0.1\n```\nto\n```python\n  vertices_density_quantile = 0.\n```\n\n\u003c/details\u003e\n\n\u003cdetails\u003e\n\u003csummary\u003e\u003cspan style=\"font-weight: bold;\"\u003e5. I have messy ellipsoidal bumps on the surface of my mesh, what can I do?\u003c/span\u003e\u003c/summary\u003e\u003cbr\u003e\n\nDepending on your scene, the default hyperparameters used for Poisson reconstruction may be too fine compared to the size of the Gaussians. Gaussian could then become visible on the mesh, which results in messy ellipsoidal bumps on the surface of the mesh.\nThis could happen if the camera trajectory is very close to a simple foreground object, for example.\u003cbr\u003e\nTo fix this, you can reduce the depth of Poisson reconstruction `poisson_depth` by modifying line 42 of `sugar_extractors/coarse_mesh.py`. \u003cbr\u003e\nFor example, you can change line 42 from\n```python\n  poisson_depth = 10\n```\nto\n```python\n  poisson_depth = 7\n```\nYou may also try `poisson_depth = 6`, or `poisson_depth = 8` if the result is not satisfying.\n\n\u003c/details\u003e\n\n\n\u003cdetails\u003e\n\u003csummary\u003e\u003cspan style=\"font-weight: bold;\"\u003e6. (Optional) Adapt the scale and the bounding box of the scene\u003c/span\u003e\u003c/summary\u003e\u003cbr\u003e\n\nAs it is explained in the original \u003ca href=\"https://github.com/graphdeco-inria/gaussian-splatting\"\u003e3D Gaussian Splatting repository\u003c/a\u003e, the method is expected to reconstruct a scene with reasonable scale. For reconstructing much larger datasets, like a city district, the original authors recommend to lower the learning rates of the positions and scaling factors of the Gaussians. The more extensive the scene, the lower these values should be.\n\nConcerning SuGaR, such learning rates should also be lowered when reconstructing a very large scene. Moreover, as we explain in the supplementary material of the paper, for extracting a mesh from the Gaussians with an optimal repartition of vertices, we apply two Poisson reconstructions in practice: one on _foreground_ Gaussians, and one on _background_ Gaussians. The foreground Gaussians are defined as the Gaussians located inside a predefined bounding box, and the background Gaussians are defined as the Gaussians located outside this bounding box. \n\nBy default, this bounding box is computed as the bounding box of all camera centers. This general approach is coherent with how the original 3D Gaussian Splatting scales the learning rates. We used this default bounding box for all the reconstructions shown in the paper and the presentation video.\n\nHowever, this bounding box might not be optimal in very specific cases, especially when the user wants to reconstruct with high details a very specific object located somewhere in the scene, or if the scene is very large, or if the camera centers are very far from the scene.\nThe user is free to provide a custom bounding box to the `train.py` script, using the parameters `--bboxmin` and `--bboxmax`. Please note that the bounding box must be provided as strings, formatted as `\"(x,y,z)\"`, where `x`, `y` and `z` are the coordinates of the min and max points of the bounding box.\n\n\u003c/details\u003e\n\n\u003c/details\u003e","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fanttwo%2Fsugar","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fanttwo%2Fsugar","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fanttwo%2Fsugar/lists"}