{"id":28223289,"url":"https://github.com/xbillowy/diff-surfel-tracing","last_synced_at":"2026-03-07T01:02:36.640Z","repository":{"id":283930105,"uuid":"911002454","full_name":"xbillowy/diff-surfel-tracing","owner":"xbillowy","description":"A fully differentiable 2D Gaussian ray tracer built on 2DGS and OptiX, supporting multiple-bounce path tracing for complex light transport simulation. Easily integrate additional parameters and outputs for custom rendering and optimization tasks.","archived":false,"fork":false,"pushed_at":"2025-10-14T07:33:44.000Z","size":66,"stargazers_count":34,"open_issues_count":0,"forks_count":1,"subscribers_count":3,"default_branch":"main","last_synced_at":"2025-10-24T05:44:16.785Z","etag":null,"topics":["2dgs","3dgs","nerf","optix","path-tracing","ray-tracing"],"latest_commit_sha":null,"homepage":"","language":"C++","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"other","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/xbillowy.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2025-01-02T02:50:44.000Z","updated_at":"2025-10-21T18:38:06.000Z","dependencies_parsed_at":"2025-03-23T05:25:02.216Z","dependency_job_id":"77be80a8-1629-42b5-9797-6ef4bc195268","html_url":"https://github.com/xbillowy/diff-surfel-tracing","commit_stats":null,"previous_names":["xbillowy/diff-surfel-tracing"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/xbillowy/diff-surfel-tracing","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/xbillowy%2Fdiff-surfel-tracing","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/xbillowy%2Fdiff-surfel-tracing/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/xbillowy%2Fdiff-surfel-tracing/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/xbillowy%2Fdiff-surfel-tracing/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/xbillowy","download_url":"https://codeload.github.com/xbillowy/diff-surfel-tracing/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/xbillowy%2Fdiff-surfel-tracing/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":30204452,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-03-06T19:07:06.838Z","status":"ssl_error","status_checked_at":"2026-03-06T18:57:34.882Z","response_time":250,"last_error":"SSL_connect returned=1 errno=0 peeraddr=140.82.121.6:443 state=error: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["2dgs","3dgs","nerf","optix","path-tracing","ray-tracing"],"created_at":"2025-05-18T07:15:08.276Z","updated_at":"2026-03-07T01:02:36.601Z","avatar_url":"https://github.com/xbillowy.png","language":"C++","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Differential Surfel Tracing\n\nThis is a differentiable 2D Gaussian ray tracer, built on the foundation of [2DGS](https://surfsplatting.github.io/) and NVIDIA OptiX, tailored for differentiable optimization and rendering tasks. Key features of this tracer include:\n\n- **Differentiability**: The tracer is entirely differentiable, encompassing the 2D Gaussian parameters and the input ray origins and directions, should the rays be optimized.\n- **Path Tracing**: It supports path tracing with multiple bounces, which is beneficial for rendering complex materials and simulating intricate light transport phenomena.\n- **Customizable Rendering**: The tracer allows for customized rendering, enabling you to incorporate additional precomputed parameters and outputs tailored to your specific requirements.\n\nThis 2D Gaussian ray tracer is developed for our paper [**EnvGS**](https://github.com/zju3dv/EnvGS), feel free to check it out!\n\n- [EnvGS: Modeling View-Dependent Appearance with Environment Gaussian](https://github.com/zju3dv/EnvGS): the tracer is **developed for**.\n- [LiDAR-RT: Gaussian-based Ray Tracing for Dynamic LiDAR Re-simulation](https://github.com/zju3dv/LiDAR-RT): uses this tracer.\n\n\nIf you could make use of it in your research, please be so kind as to cite us as [Citation](#citation) and leave your star 🌟.\n\nhttps://github.com/xbillowy/assets/diff-surfel-tracing/assets/bb3095a6-71ed-4f55-8b77-0effbc85af37\n\n\n## 📦 Installation\n\nThe installation is similar to the installation of [diff-gaussian-rasterization](https://github.com/graphdeco-inria/diff-gaussian-rasterization/tree/main) and [diff-surfel-rasterization](https://github.com/hbb1/diff-surfel-rasterization) except that the [NVIDIA OptiX SDK](https://developer.nvidia.com/designworks/optix/download) is required.\n\nWe have included the [official OptiX SDK header files](https://github.com/NVIDIA/optix-dev) in the `third_party/optix` directory as a submodule, so by default, you don't need to download the OptiX SDK from the [NVIDIA official website](https://developer.nvidia.com/designworks/optix/download), just add the `--recursive` flag when cloning the repository.\n\nThe default installation is as follows:\n\n```bash\n# Clone the repository\ngit clone https://github.com/xbillowy/diff-surfel-tracing.git --recursive\ncd diff-surfel-tracing\n\n# Install using pip\npip install -v .  # add the `-v` flag for verbose output\n```\n\nIf you want to use the OptiX SDK from the [NVIDIA official website](https://developer.nvidia.com/designworks/optix/download) for maybe a specific version, you can download the OptiX SDK and set the environment variable `OPTIX_HOME` to the download directory of the OptiX SDK to your `.zshrc` or `.bashrc` to expose related paths for compilation.\n\n```bash\n# CUDA related configs, you may need to change the path according to your installation\nexport PATH=\"/usr/local/cuda/bin:$PATH\"\nexport LD_LIBRARY_PATH=\"/usr/local/cuda/lib64:$LD_LIBRARY_PATH\"\nexport CUDA_HOME=\"/usr/local/cuda\"\n# Set the environment variable OPTIX_HOME to the installation directory of the OptiX SDK\nexport OPTIX_HOME=/path/to/optix\n```\n\n\n## 🛠️ Usage\n\nWe provide a simple example in the [`example/render.py`](example/render.py) to demonstrate how to use the tracer. To use the tracer, you can download our pre-trained 2DGS model and a pre-defined camera path from the [Google Drive](https://drive.google.com/file/d/1drKlXptpkht0ZVp6Ywh8ZSSXsi8RddKx/view?usp=sharing). Once you have the tracer installed and the example data downloaded, unzip the files to the root directory of this repository, and run the following commands to render the example scene:\n\n```bash\n# Install the dependencies\npip install -r example/requirements.txt\n\n# Run the example to render the scene\npython example/render.py\n```\n\nThe rendered RGB images, depth maps, normal maps, and corresponding videos will be saved to the `data/result/` directory.\n\n### Core Snippets\n\nThe usage of this tracer is quite similar to the use of [diff-surfel-rasterization](https://github.com/hbb1/diff-surfel-rasterization), here is an example of how to use this tracer, note that you may need a pre-trained [2DGS](https://github.com/hbb1/2d-gaussian-splatting) model first:\n\n```python\nimport torch\nfrom diff_surfel_tracing import SurfelTracer, SurfelTracingSettings\n\n# Create a SurfelTracer\ntracer = SurfelTracer()\n\n# Convert 2D Gaussian primitives to triangle vertices and faces\nv, f = get_triangles(pcd)\n\n# Build the acceleration structure\ntracer.build_acceleration_structure(v, f, rebuild=True)\n\n# NOTE: To avoid weird behavior, it is recommended to manually invoke the\n# NOTE: `.contiguous()` on every input tensor before passing them to the tracer.\n\n# Set the surfel tracing settings\n# Check the details of the parameters in the Parameters Explanation section\ntracer_settings = SurfelTracingSettings(\n    image_height=int(viewpoint_camera.image_height),\n    image_width=int(viewpoint_camera.image_width),\n    tanfovx=math.tan(viewpoint_camera.FoVx * 0.5),\n    tanfovy=math.tan(viewpoint_camera.FoVy * 0.5),\n    bg=bg_color,\n    scale_modifier=scale_modifier,\n    viewmatrix=viewpoint_camera.world_view_transform,\n    projmatrix=viewpoint_camera.full_proj_transform,\n    sh_degree=pcd.active_sh_degree,\n    campos=viewpoint_camera.camera_center,\n    prefiltered=False,\n    debug=False,\n    max_trace_depth=max_trace_depth,\n    specular_threshold=specular_threshold,\n)\n\n# Create dummy input to receive the gradient for densification\ngrads3D = (\n    torch.zeros_like(\n        means3D, dtype=means3D.dtype, requires_grad=True, device=means3D.device\n    )\n    + 0\n)\ntry:\n    grads3D.retain_grad()\nexcept:\n    pass\n\n# Perform the ray tracing\n# Check the details of the inputs in the Parameters Explanation section\nrgb, dpt, acc, norm, dist, aux, mid, wet = tracer(\n    ray_o,  # (H, W, 3) or (B, P, 3)\n    ray_d,  # (H, W, 3) or (B, P, 3)\n    v,  # (P * 4, 3)\n    means3D=means3D,  # (P, 3)\n    grads3D=grads3D,  # (P, 3)\n    shs=shs,\n    colors_precomp=colors_precomp,\n    others_precomp=others_precomp,\n    opacities=opacities,  # (P, 1)\n    scales=scales,  # (P, 2)\n    rotations=rotations,  # (P, 4)\n    cov3D_precomp=cov3D_precomp,\n    tracer_settings=tracer_settings,\n    start_from_first=start_from_first,\n)\n```\n\n### 2DGS to Triangles\n\nThe `get_triangles` function is used to convert the 2DGS to vertices and faces in order to build the acceleration structure, here is an example of how to implement this function.\n\n\u003cdetails\u003e\n\n\u003csummary\u003eExample implementation of \u003ccode\u003eget_triangles()\u003c/code\u003e\u003c/summary\u003e\n\n```python\ndef get_triangles(pcd: GaussianModel):\n    # Build the uv tangent plane to world transformation matrix, splat2world\n    T = pcd.get_covariance()  # (P, 4, 4)\n    T = T.permute(0, 2, 1)  # (P, 4, 4)\n    T[..., 2] = 0  # (P, 4, 4)\n\n    # Deal with nasty shapes\n    P, V = T.shape[0], 4  # 1 2DGS \u003c-\u003e 2 triangles \u003c-\u003e 4 vertices\n\n    # 3-sigma range in local uv splat coordiantes\n    sigma3 = (\n        torch.as_tensor(\n            [[-1.0, 1.0], [-1.0, -1.0], [1.0, 1.0], [1.0, -1.0]], device=T.device\n        )\n        * 3\n    )  # (V, 2)\n    sigma3 = torch.cat([sigma3, torch.ones_like(sigma3)], dim=-1)  # (V, 4)\n    # Expand\n    sigma3 = sigma3[None].repeat(P, 1, 1)  # (P, V, 4)\n    T = T[:, None].expand(-1, V, -1, -1)  # (P, V, 4, 4)\n\n    # Convert the vertices to the world coordinate\n    v = T.reshape(-1, 4, 4) @ sigma3.reshape(-1, 4, 1)  # (P * V, 4, 1)\n    v = v[..., :3, 0]  # (P * V, 3)\n\n    # Generate face indices\n    indices = torch.arange(0, v.shape[0]).reshape(P, V).to(T.device)  # (P, V)\n    f = (\n        torch.stack([indices[:, :3], indices[:, 1:]], dim=1).reshape(-1, 3).int()\n    )  # (P, 2, 3) -\u003e (P * 2, 3)\n\n    # NOTE: `.contiguous()` is necessary for the following OptiX CUDA operations!\n    v, f = v.contiguous(), f.contiguous()\n\n    return v, f\n```\n\n\u003c/details\u003e\n\n### Parameters Explanation\n\nMost parameters are consistent with [diff-gaussian-rasterization](https://github.com/graphdeco-inria/diff-gaussian-rasterization) and [diff-surfel-rasterization](https://github.com/hbb1/diff-surfel-rasterization), here we provide details on the parameters that may be different or newly added.\n\n\u003cdetails\u003e\n\n\u003csummary\u003e\u003ccode\u003eSurfelTracingSettings\u003c/code\u003e Parameters\u003c/summary\u003e\n\n- `viewmatrix`: no actual use in the ray tracing, only for consistency with the rasterizer.\n- `projmatrix`: no actual use in the ray tracing, only for consistency with the rasterizer.\n- `campos`: no actual use in the ray tracing, only for consistency with the rasterizer.\n- **`max_trace_depth`**: number of path tracing bounces, default is 0, means only trace once.\n- **`specular_threshold`**: the threshold for continuing the path tracing, default is 0.0. Ignore this if you are not using the path tracing or any BRDFs rendering.\n\n\u003c/details\u003e\n\n\u003cdetails\u003e\n\n\u003csummary\u003e\u003ccode\u003eSurfelTracer\u003c/code\u003e Parameters\u003c/summary\u003e\n\n- **`ray_o`**: the origin of the rays, a 3-dimension Tensor of shape `(H, W, 3)` or `(B, P, 3)`.\n- **`ray_d`**: the direction of the rays, a 3-dimension Tensor of shape `(H, W, 3)` or `(B, P, 3)`.\n- **`v`**: the covering triangles of the 2D Gaussian splats, a 2-dimension Tensor of shape `(P, 3)`, which is used to support fully-differentiable backpropagation.\n- `grads3D`: the gradient tensor for the densification, the same as [`means2D`](https://github.com/graphdeco-inria/gaussian-splatting/blob/54c035f7834b564019656c3e3fcc3646292f727d/gaussian_renderer/__init__.py#L55) in the original 3DGS rasterizer.\n- `colors_precomp`: used for RGB only, since we use pixel ray direction rather than the Gaussian center minus camera center direction as the ray direction, which means the original precomputation of the color is not applicable.\n- **`others_precomp`**: support custom rendering, you can add more. Remember to add the corresponding parameters and offsets in the [config.h](./optix_tracer/config.h).\n- **`start_from_first`**: indicates whether the rays start from the camera or any other starting point (e.g., the bounce surface point), default is `True`.\n\n\u003c/details\u003e\n\n### Outputs Explanation\n\nIn addition to the default output of [diff-surfel-rasterization](https://github.com/hbb1/2d-gaussian-splatting/blob/df1f6c684cc4e41a34937fd45a7847260e9c6cd7/gaussian_renderer/__init__.py#L97-L156), namely `rgb`, `dpt`, `acc`, `norm` for RGB image, depth map, accumulated opacity, and normal map, respectively, we also provide the following outputs: `dist`, `aux`, `mid`, `wet`, see the details below.\n\n\u003cdetails\u003e\n\n\u003csummary\u003eAdditional Outputs\u003c/summary\u003e\n\n- `aux`: corresponding to the rendered `others_precomp` map in the input, used for custom rendering.\n- `mid`: the middle rendering results for each path tracing bounce, e.g., the accumulated color, opacity, and normal of the first trace will be stored if you set `max_trace_depth` to 1.\n- `wet`: the accumulated contribution weight for each 2D Gaussian splat.\n- `dist`: invalid distortion map, all zeros for now, maybe implement in the future.\n\n\u003c/details\u003e\n\n\n## 🚧 TODOs\n\n- [x] TODO: Release the initial version.\n- [ ] TODO: Test OptiX 8.0.0 and OptiX 8.1.0 compatibility.\n- [ ] TODO: Apply more of the CUDA optimization techniques.\n\n\n## 📚 Credits\n\nWe would like to acknowledge the following inspiring prior work:\n\n- [3D Gaussian Ray Tracing: Fast Tracing of Particle Scenes](https://gaussiantracer.github.io/)\n- [3D Gaussian Splatting for Real-Time Radiance Field Rendering](https://github.com/graphdeco-inria/gaussian-splatting)\n- [2DGS: 2D Gaussian Splatting for Geometrically Accurate Radiance Fields](https://surfsplatting.github.io/)\n\n\n## 📜 Citation\n\n```bibtex\n@article{xie2024envgs,\n  title={EnvGS: Modeling View-Dependent Appearance with Environment Gaussian},\n  author={Xie, Tao and Chen, Xi and Xu, Zhen and Xie, Yiman and Jin, Yudong and Shen, Yujun and Peng, Sida and Bao, Hujun and Zhou, Xiaowei},\n  journal={arXiv preprint arXiv:2412.15215},\n  year={2024}\n}\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fxbillowy%2Fdiff-surfel-tracing","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fxbillowy%2Fdiff-surfel-tracing","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fxbillowy%2Fdiff-surfel-tracing/lists"}