{"id":13816222,"url":"https://github.com/pointrix-project/msplat","last_synced_at":"2025-05-15T15:32:17.921Z","repository":{"id":228321819,"uuid":"766527133","full_name":"pointrix-project/msplat","owner":"pointrix-project","description":"A modular differential gaussian rasterization library.","archived":false,"fork":false,"pushed_at":"2024-07-28T09:03:04.000Z","size":8205,"stargazers_count":170,"open_issues_count":2,"forks_count":10,"subscribers_count":8,"default_branch":"master","last_synced_at":"2024-11-19T13:37:41.769Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":"Cuda","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"other","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/pointrix-project.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-03-03T14:13:36.000Z","updated_at":"2024-10-26T08:32:51.000Z","dependencies_parsed_at":"2024-03-27T11:42:07.256Z","dependency_job_id":"54a3045f-9b38-4d6c-84f7-a0c2f2e4dcdc","html_url":"https://github.com/pointrix-project/msplat","commit_stats":null,"previous_names":["pointrix-project/dptr","pointrix-project/msplat"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/pointrix-project%2Fmsplat","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/pointrix-project%2Fmsplat/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/pointrix-project%2Fmsplat/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/pointrix-project%2Fmsplat/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/pointrix-project","download_url":"https://codeload.github.com/pointrix-project/msplat/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":254367630,"owners_count":22059544,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-08-04T05:00:37.539Z","updated_at":"2025-05-15T15:32:17.353Z","avatar_url":"https://github.com/pointrix-project.png","language":"Cuda","funding_links":[],"categories":["Open Source Implementations","Tools, Pipeline \u0026 Utilities"],"sub_categories":["Framework","Gaussian Splatting \u0026 NeRF"],"readme":"\n\n# MSplat\n\nMSplat is a modular differential gaussian rasterization library. We have refactored the original repository for easier understanding and extension.\n\n**What's new：**\n- The camera model has been changed from FOV to pinhole.\n- Optimization on camera model is support.\n- SH evaluation is now supported up to level 10.\n- Differentiable rendering of extensible features (RGB, depth and even more) is now supported. \n\n## How to install\n```shell\ngit clone https://github.com/pointrix-project/msplat.git --recursive\ncd msplat\npip install .\n```\n\n## Camera Model\nWe use a pinhole camera model:\n\n$$dx=K[R_{cw}|t_{cw}]X$$\n\n$$d\\begin{bmatrix}\n  u \\\\ \n  v \\\\ \n  1\n\\end{bmatrix} = \\begin{bmatrix}\n  f_x \u0026 0 \u0026 c_x \\\\\n  0 \u0026  f_y \u0026 c_y \\\\\n  0 \u0026 0 \u0026 1 \\\\\n\\end{bmatrix}\\begin{bmatrix}\n  r_{11} \u0026 r_{12} \u0026 r_{13} \u0026 t_1 \\\\\n  r_{21} \u0026 r_{22} \u0026 r_{23} \u0026 t_2 \\\\\n  r_{31} \u0026 r_{32} \u0026 r_{33} \u0026 t_3 \\\\\n\\end{bmatrix}\\begin{bmatrix}\n  X \\\\ \n  Y \\\\ \n  Z \\\\ \n  1\n\\end{bmatrix}$$\n\n* $(X, Y, Z)$ is the coordinate of a 3D point in the world coordinate system.\n* $(u,v)$ are the coordinate of the projection point in **pixels**, $d$ is the depth value. \n* $K$ is a matrix of intrinsic parameters in pixels. $(c_x, c_y)$ is the principal point, which is usually the image center $(\\frac{W}{2},\\frac{H}{2})$. $f_x$ and $f_y$ are the focal lengths.\n* $R_{cw}$ and $T_{cw}$ are extrinsic parameters (view matrix) indicated **world-to-camera** transformation in  [**OpenCV/Colmap** convention](https://kit.kiui.moe/camera/#common-camera-coordinate-systems).\n\nIn our API, you need to provide the camera parameters in the following format:\n- Intrinsic Parameters $[f_x, f_y, c_x, c_y]$\n- Extrinsic Parameters $[R_{cw}|t_{cw}]$\n\n## Optimization on Camera\nIt hasn't been validated too much. Since a camera is involved in the optimization of many 3D Gaussian points, the gradient of the camera is  accumulating, potentially leading to numerical issues.\n\n## High-level SH\nWe have upgraded the level of the Spherical Harmonics (SH) from the original 3 to 10.\n\n## Extensiable features\nWe abstract concepts such as RGB, depth, etc. to features and enable feature rendering for any channels. As a result, we easily render rgb map, depth map and orther feature maps without the need to modify and recompile the kernel.\n\n## More easy to understand\nWe break the heavy and confusing kernel to clear and easy-understant parts, which is more friendly for beginers. You can invoke the renderer with a single line of command, or you can break it down into steps.\n\n```python\nimport torch\nimport gs\n\n# the inputs\nH: int = ... # image height\nW: int = ... # image width\nintr: torch.Tensor = ... # [4,], (fx, fy, cx, cy) in pixels\nextr: torch.Tensor = ... # [4, 4], camera extrinsics in OpenCV convention.\nxyzs: torch.Tensor = ... # [N, 3], Gaussian centers\nfeats: torch.Tensor = ... # [N, C], Gaussian RGB features (or any other features with arbitrary channels!)\nopacity: torch.Tensor = ... # [N, 1], Gaussian opacity\nscales: torch.Tensor = ... # [N, 3], Gaussian scales\nrotations: torch.Tensor = ... # [N, 4], Gaussian rotations in quaternion\nbg: float = 0 # scalar, background for rendered feature maps\n\n# =================== one-line interface =================== \nrendered_image = gs.rasterization(\n    xyzs, scales, rotations, opacity, feats, intr, extr, W, H, bg\n) # [C, H, W]\n\n# =================== steps interface =================== \n# project points\n(uv, depth) = gs.project_point(xyzs, intr, extr, W, H)\nvisible = depth != 0\n\n# compute cov3d\ncov3d = gs.compute_cov3d(scales, rotations, visible)\n\n# ewa project\n(conic, radius, tiles_touched) = gs.ewa_project(\n    xyzs, cov3d, intr, extr, uv, W, H, visible\n)\n\n# sort\n(gaussian_ids_sorted, tile_range) = gs.sort_gaussian(\n    uv, depth, W, H, radius, tiles_touched\n)\n\n# alpha blending\nrendered_image = gs.alpha_blending(\n    uv, conic, opacity, rgbs, gaussian_ids_sorted, tile_range, bg, W, H,\n)\n```\n\n## Acknowledgment\nWe express our sincerest appreciation to the developers and contributors of the following projects:\n- [3D Gaussian Splatting](https://github.com/graphdeco-inria/gaussian-splatting): 3D Gaussian Splatting for Real-Time Radiance Field Rendering.\n\nIf you find that 3D gaussian splatting implemetation in our project helpful, please consider to cite:\n```\n@Article{kerbl3Dgaussians,\n      author       = {Kerbl, Bernhard and Kopanas, Georgios and Leimk{\\\"u}hler, Thomas and Drettakis, George},\n      title        = {3D Gaussian Splatting for Real-Time Radiance Field Rendering},\n      journal      = {ACM Transactions on Graphics},\n      number       = {4},\n      volume       = {42},\n      month        = {July},\n      year         = {2023},\n      url          = {https://repo-sam.inria.fr/fungraph/3d-gaussian-splatting/}\n}\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fpointrix-project%2Fmsplat","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fpointrix-project%2Fmsplat","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fpointrix-project%2Fmsplat/lists"}