{"id":29414113,"url":"https://github.com/suous/learn-gaussian-splatting","last_synced_at":"2025-10-10T21:07:08.180Z","repository":{"id":302193544,"uuid":"1003593022","full_name":"suous/learn-gaussian-splatting","owner":"suous","description":"Learn 3D Gaussian Splatting Basics.","archived":false,"fork":false,"pushed_at":"2025-07-01T03:13:32.000Z","size":2938,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2025-07-01T04:25:52.659Z","etag":null,"topics":["gaussian-splatting"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/suous.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2025-06-17T11:32:24.000Z","updated_at":"2025-07-01T03:13:35.000Z","dependencies_parsed_at":"2025-07-01T04:36:08.063Z","dependency_job_id":null,"html_url":"https://github.com/suous/learn-gaussian-splatting","commit_stats":null,"previous_names":["suous/learn-gaussian-splatting"],"tags_count":1,"template":false,"template_full_name":null,"purl":"pkg:github/suous/learn-gaussian-splatting","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/suous%2Flearn-gaussian-splatting","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/suous%2Flearn-gaussian-splatting/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/suous%2Flearn-gaussian-splatting/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/suous%2Flearn-gaussian-splatting/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/suous","download_url":"https://codeload.github.com/suous/learn-gaussian-splatting/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/suous%2Flearn-gaussian-splatting/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":264812208,"owners_count":23667890,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["gaussian-splatting"],"created_at":"2025-07-11T12:46:11.077Z","updated_at":"2025-10-10T21:07:03.121Z","avatar_url":"https://github.com/suous.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Learning Gaussian Splatting\n\n\u003e I want to fully grasp [3D Gaussian Splatting](https://repo-sam.inria.fr/fungraph/3d-gaussian-splatting/). The original code was a bit tough to follow without computer graphics or CUDA background, especially tile-based rendering. So I'm trying my best to implement it from scratch with NumPy and PyTorch, starting with rendering. Simplicity is prioritized over performance. \n\n![rasterizer](https://yindaheng98.github.io/%E5%9B%BE%E5%BD%A2%E5%AD%A6/zhimg.com/v2-7cbe3b0c3b67ce80593fad0d73a814b5_r.jpg)\n\n\n## Core Code Statistics\n\n```bash\n\n# tokei camera.py gaussian.py render.py utils.py\n├── 📄 camera.py   # Camera model and view transformation utilities\n├── 📄 gaussian.py # 3D Gaussian representation and operations\n├── 📄 render.py   # Rendering pipeline and projection functions\n└── 📄 utils.py    # Helper functions and common utilities\n\n===============================================================================\n Language            Files        Lines         Code     Comments       Blanks\n===============================================================================\n Python                  4          662          529           44           89\n===============================================================================\n```\n\n## Getting Started\n\n### Installation\n\n1. Clone the repository:\n\n```bash\ngit clone https://github.com/suous/learn-gaussian-splatting.git\ncd learn-gaussian-splatting\n```\n\n2. Install the required dependencies.\n\n```bash\nuv sync\n```\n\n3. Activate the environment.\n\n```bash\nsource .venv/bin/activate\n```\n\n### Data Download\n\nDownload the COLMAP and PLY data from [Release v1.0.0](https://github.com/suous/learn-gaussian-splatting/releases/tag/v1.0.0):\n```bash\nwget -qO-  https://github.com/suous/learn-gaussian-splatting/releases/download/v1.0.0/data.tgz  | tar xz -C .\n```\n\nThis will create a data directory with the following structure:\n\n```bash\ndata\n├── colmap\n│   ├── images\n│   │   ├── 000000.png\n│   │   ├── ..........\n│   │   └── 000024.png\n│   └── sparse\n│       └── 0\n│           ├── cameras.bin\n│           ├── images.bin\n│           ├── points3D.bin\n│           └── points3D.ply\n└── point_cloud.ply\n```\n\n```bash\n# python gaussian.py\n==================================================\n          Gaussian Splatting Statistics\n==================================================\n\n    🎯 Positions  : (37942, 3)\n    🔄 Rotations  : (37942, 4)\n    📏 Scalings   : (37942, 3)\n    💫 Opacities  : (37942, 1)\n    🎨 Features   : (37942, 3, 16)\n    📊 Covariances: (37942, 3, 3)\n    \n--------------------------------------------------\nTotal Gaussians: 37,942\nSH Degree      : 3\n==================================================\n```\n\n### Usage\n\n- Render an Image\n\n```bash\npython main.py --image_number 0\n```\n\nThis will display the original image and the rendered image using matplotlib.\n\n- Generate animation\n\n```bash\npython main.py --generate_animation\n```\n\n```bash\nffmpeg -framerate 8 -pattern_type glob -i \"*.png\" -vf \"fps=10,scale=640:-1,split[s0][s1];[s0]palettegen[p];[s1][p]paletteuse\" -loop 0 animation.gif\n```\n\n### Results\n\n![keyboard](./images/reconstruct-keyboard.png)\n\n\u003cimg alt=\"gaussian animation\" src=\"./images/gaussian-animation.gif\" width=\"100%\"\u003e\n\n## Basic Idea of 3D Gaussian Splatting\n\n3D Gaussian Splatting (3DGS) is a novel method for representing and rendering 3D scenes, introduced in the paper [\"3D Gaussian Splatting for Real-Time Radiance Field Rendering\"](https://repo-sam.inria.fr/fungraph/3d-gaussian-splatting/downloads/3d_gaussian_splatting_high.pdf) by Kerbl et al. (2023). It has gained significant attention as an alternative to Neural Radiance Fields (NeRFs) due to its impressive rendering speed and quality.\n\nAt its core, 3DGS represents a 3D scene as a collection of millions of 3D Gaussians. Each Gaussian is a point in space with specific parameters that define its appearance and shape.\n\n### Gaussian Parameters\n\n![gaussian-parameters](./images/gaussian-parameters.png)\n\u003e image from https://3dgstutorial.github.io/3dv_part2.pdf\n\nEach 3D Gaussian is defined by the following parameters:\n\n- **Position (Mean $\\mu$):** A 3D vector representing the center of the Gaussian: $[x, y, z]$.\n- **Covariance ($\\Sigma$):** A $3\\times3$ matrix that defines the shape, size, and orientation of the Gaussian. To ensure that the covariance matrix remains physically meaningful (positive semi-definite) during optimization, it is decomposed into:\n    - **Scaling ($S$):** A 3D vector representing the scale of the Gaussian along its local axes: $[S_x, S_y, S_z]$.\n    - **Rotation ($R$):** A quaternion representing the orientation of the Gaussian in space: $q=[r, x, y, z]$.\n- **Color ($c$):** The color of the Gaussian, which can be represented as RGB values or, more commonly, as coefficients of Spherical Harmonics (SH) to model view-dependent effects like reflections.\n- **Opacity ($\\alpha$):** A value between 0 and 1 that controls the transparency of the Gaussian.\n\n$$\nR = \n\\begin{bmatrix}\n1 - 2(y^{2} + z^{2}) \u0026 2(xy - rz)           \u0026 2(xz + ry)        \\\\\n2(xy + rz)           \u0026 1 - 2(x^{2} + z^{2}) \u0026 2(yz - rx)        \\\\\n2(xz - ry)           \u0026 2(yz + rx)           \u0026 1 - 2(x^{2} + y^{2})\n\\end{bmatrix}\n$$\n\n$$\\Sigma = RSS^TR^T$$\n\n### Rendering\n\nThe rendering process in 3DGS is a form of rasterization. For a given camera viewpoint, the 3D Gaussians are projected onto the 2D image plane. These 2D projections are then sorted by depth and blended together, front-to-back, to produce the final color for each pixel. \n\n## Mathematical Formulation\n\nThe influence of a 3D Gaussian at a point **$x$** in space is given by the Gaussian function:\n\n$$ G(x) = e^{-\\frac{1}{2}(x-\\mu)^T \\Sigma^{-1} (x-\\mu)} $$\n\nWhere:\n- **$\\mu$** is the mean (center) of the Gaussian.\n- **$\\Sigma$** is the covariance matrix.\n\n$$\n\\Sigma = \n\\begin{bmatrix}\n\\sigma_x^2 \u0026 \\text{Cov}(x, y) \u0026 \\text{Cov}(x, z) \\\\\n\\text{Cov}(y, x) \u0026 \\sigma_y^2 \u0026 \\text{Cov}(y, z) \\\\\n\\text{Cov}(z, x) \u0026 \\text{Cov}(z, y) \u0026 \\sigma_z^2\n\\end{bmatrix}\n$$\n\nDuring the 3D-to-2D projection process, the Jacobian matrix $J$ represents the derivative of the 2D spatial coordinates $(u, v)$ with respect to the 3D spatial coordinates $(x,y,z)$, i.e.:\n\n$$\nJ = \\frac{\\partial (u,v)}{\\partial (x,y,z)} = \n\\begin{bmatrix}\n\\frac{\\partial u}{\\partial x} \u0026 \\frac{\\partial u}{\\partial y} \u0026 \\frac{\\partial u}{\\partial z} \\\\\n\\frac{\\partial v}{\\partial x} \u0026 \\frac{\\partial v}{\\partial y} \u0026 \\frac{\\partial v}{\\partial z}\n\\end{bmatrix}\n$$\n\n3D Gaussian Splatting (3DGS) specifically uses perspective projection. According to the perspective projection formula:\n\n$$\nz \\begin{bmatrix} u \\\\ v \\\\ 1 \\end{bmatrix} = \n\\begin{bmatrix}\nf_x \u0026 0 \u0026 c_x \\\\\n0 \u0026 f_y \u0026 c_y \\\\\n0 \u0026 0 \u0026 1\n\\end{bmatrix}\n\\cdot\n\\begin{bmatrix} x \\\\ y \\\\ z \\end{bmatrix}\n= z \\begin{bmatrix}\nf_x \\frac{x}{z} + c_x \\\\\nf_y \\frac{y}{z} + c_y \\\\\n1\n\\end{bmatrix}\n$$\n\nThe relationship between $(u,v)$ and $(x,y,z)$ is:\n\n$$\n\\begin{cases}\nu = f_x \\dfrac{x}{z} + c_x \\\\\n\\\\\nv = f_y \\dfrac{y}{z} + c_y\n\\end{cases}\n$$\n\nThe Jacobian matrix $J$ is then derived as:\n\n$$\nJ = \n\\begin{bmatrix}\nf_x \\dfrac{1}{z} \u0026 0 \u0026 -f_x \\dfrac{x}{z^2} \\\\\n\\\\\n0 \u0026 f_y \\dfrac{1}{z} \u0026 -f_y \\dfrac{y}{z^2}\n\\end{bmatrix}\n$$\n\n$$\n\\Sigma^\\prime = JW\\Sigma W^TJ^T\n$$\n\nThe final color **$C$** of a pixel is computed by blending the projected Gaussians that overlap with it, ordered by depth:\n\n$$ C = \\sum_{i \\in N} c_i \\alpha_i \\prod_{j=1}^{i-1}(1 - \\alpha_j) $$\n\nWhere:\n- **$N$** is the set of Gaussians overlapping the pixel, sorted by depth.\n- **$c_i$** is the color of the i-th Gaussian.\n- **$\\alpha_i$** is the final opacity of the i-th Gaussian, which is calculated from its 2D projection.\n\n$$ \\alpha_i = \\sigma(\\alpha_i^\\prime) \\times e^{-\\frac{1}{2}x^T \\Sigma_i^{\\prime-1} x} $$\n- **$\\alpha^\\prime$** is the learned opacity.\n- **$\\Sigma^{\\prime}$** is the 2d projection of the 3d covariance.\n\n$$\n\\Sigma^\\prime = \n\\begin{bmatrix}\n\\sigma_x^2 \u0026 \\text{Cov}(x, y) \\\\\n\\text{Cov}(y, x) \u0026 \\sigma_y^2 \\\\\n\\end{bmatrix}\n$$\n\n## Training\n\nThe parameters of the Gaussians are optimized to reconstruct a scene from a set of input images with known camera poses. The training process involves:\n\n1.  **Initialization:** The process starts with a sparse point cloud generated from the input images using Structure-from-Motion (SfM), for example, with [COLMAP](https://github.com/colmap/colmap). These points are used to initialize the positions ($\\mu$) of the Gaussians.\n2.  **Optimization:** The parameters of the Gaussians (position, covariance, color, opacity) are optimized using stochastic gradient descent. The loss function is typically a combination of an L1 loss and a D-SSIM (Structural Dissimilarity Index) term, comparing the rendered images with the training images.\n3.  **Adaptive Densification:** During optimization, the set of Gaussians is adaptively modified to better represent the scene. This involves:\n    -   **Cloning:** Duplicating small Gaussians in areas that are under-reconstructed.\n    -   **Splitting:** Splitting large Gaussians in areas with high variance.\n    -   **Pruning:** Removing Gaussians that are nearly transparent (very low opacity).\n\n## TODO\n\n- [ ] Understand and implement tile based rendering.\n- [ ] Train from scratch with pytorch code.\n\n## References\n\n### Papers\n\n- [3D Gaussian Splatting for Real-Time Radiance Field Rendering](https://repo-sam.inria.fr/fungraph/3d-gaussian-splatting/downloads/3d_gaussian_splatting_high.pdf) \n- [A Survey on 3D Gaussian Splatting](https://arxiv.org/abs/2401.03890)\n- [Mip-Splatting: Alias-free 3D Gaussian Splatting](https://arxiv.org/abs/2311.16493)\n- [2D Gaussian Splatting for Geometrically Accurate Radiance Fields](https://arxiv.org/abs/2403.17888)\n- [Dynamic 3D Gaussians: Tracking by Persistent Dynamic View Synthesis](https://arxiv.org/abs/2308.09713)\n- [4D Gaussian Splatting for Real-Time Dynamic Scene Rendering](https://arxiv.org/abs/2310.08528)\n- [VastGaussian: Vast 3D Gaussians for Large Scene Reconstruction](https://arxiv.org/abs/2402.17427)\n\n### Courses\n\n- [Games101](https://sites.cs.ucsb.edu/~lingqi/teaching/games101.html)\n- [Visualizing quaternions (4d numbers) with stereographic projection](https://www.youtube.com/watch?v=d4EgbgTm0Bg)\n- [【论文讲解】用点云结合3D高斯构建辐射场，成为快速训练、实时渲染的新SOTA！](https://www.bilibili.com/video/BV1uV4y1Y7cA)\n\n### Blogs\n\n- [3D Gaussian Splatting Tutorial](https://3dgstutorial.github.io/)\n- [A Comprehensive Overview of Gaussian Splatting](https://towardsdatascience.com/a-comprehensive-overview-of-gaussian-splatting-e7d570081362/) \n- [Introduction to 3D Gaussian Splatting](https://huggingface.co/blog/gaussian-splatting) \n- [Exploring 3D Gaussian Splatting](https://medium.com/@thibaut.chauffier/exploring-3d-gaussian-splatting-e50b8c9cee93) \n- [3D Gaussian Splatting Introduction – Paper Explanation \u0026 Training on Custom Datasets with NeRF Studio Gsplats](https://learnopencv.com/3d-gaussian-splatting/)\n- [Rendering in 3D Gaussian Splatting](https://www.sctheblog.com/blog/gaussian-splatting/)\n- [3D Gaussian Splatting](https://yindaheng98.github.io/%E5%9B%BE%E5%BD%A2%E5%AD%A6/3DGaussianSplatting.html#splatting)\n- [详解3D Gaussian Splatting CUDA Kernel：前向传播](https://yindaheng98.github.io/%E5%9B%BE%E5%BD%A2%E5%AD%A6/3D%E9%AB%98%E6%96%AF%E4%BB%A3%E7%A0%81%E8%A7%A3%E6%9E%90.html#foward%E5%87%BD%E6%95%B0-rasterizegaussianscuda)\n- [2DGS: 2D Gaussian Splatting for Geometrically Accurate Radiance Fields](https://surfsplatting.github.io)\n- [Understanding 3D Gaussian Splats by writing a software renderer](https://www.thomasantony.com/posts/gaussian-splatting-renderer/)\n- [[Concept summary] 3D Gaussian and 2D projection](https://xoft.tistory.com/49)\n\n### Code\n- [3dgs-warp-scratch](https://github.com/guoriyue/3dgs-warp-scratch)\n- [torch-splatting](https://github.com/hbb1/torch-splatting)\n\n\n❤️ Thanks to [Cursor](https://www.cursor.com/) for generating well-formatted code comments!\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsuous%2Flearn-gaussian-splatting","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fsuous%2Flearn-gaussian-splatting","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsuous%2Flearn-gaussian-splatting/lists"}