Ecosyste.ms: Awesome

An open API service indexing awesome lists of open source software.

Awesome Lists | Featured Topics | Projects

https://github.com/uc-vision/taichi-splatting


https://github.com/uc-vision/taichi-splatting

Last synced: 3 months ago
JSON representation

Awesome Lists containing this project

README 

        
# Taichi Splatting



Rasterizer for Guassian Splatting using Taichi and PyTorch - embedded in python library. Currently very usable but in active development, so likely will break with new versions! 



Trainer: [here](https://github.com/uc-vision/splat-trainer)

Viewer: [here](https://github.com/uc-vision/splat-trainer)



This work is originally derived off [Taichi 3D Gaussian Splatting](https://github.com/wanmeihuali/taichi_3d_gaussian_splatting), with significant re-organisation and changes.



Key differences are the rendering algorithm is decomposed into separate operations (projection, shading functions, tile mapping and rasterization) which can be combined in different ways in order to facilitate a more flexible use, and gradients can be enabled on "all the things" as required for the application (and not when disabled, to save performance).



Using the Taichi autodiff for a simpler implementation where possible (e.g. for projection, but not for the rasterization).



Examples:

  * Projecting features for lifting 2D to 3D

  * Colours via. spherical harmonics

  * Depth covariance without needing to build it into the renderer and remaining differentiable.

  * Fully differentiable camera parameters (and ability to swap in new camera models)



## Performance



A document describing some performance benchmarks of taichi-splatting [here](BENCHMARK.md). Through various optimizations, in particular optimizing the summation of gradients in the backward gradient kernel. Taichi-splatting achieves a very large speedup (often an order of magnitude) over the original taichi_3d_gaussian_splatting, and is faster than the reference diff_guassian_rasterization for a complete optimization pass (forward+backward), in particular much faster at higher resolutions.



## Installing



### External dependencies

Create an environment (for example conda with mambaforge) with the following dependencies:



* python >= 3.10

* pytorch - from either conda  Follow instructions [https://pytorch.org/](here).

* taichi-nightly `pip install --upgrade -i https://pypi.taichi.graphics/simple/ taichi-nightly`



### Install



One of:

* `pip install taichi-splatting`

* Clone down with `git clone` and install with `pip install ./taichi-splatting`



## Executables



### fit_image_gaussians



There exists a toy optimizer for fitting a set of randomly initialized gaussians to some 2D images `fit_image_gaussians` - useful for testing rasterization without the rest of the dependencies.



Fitting an image (fixed points): \

`fit_image_gaussians  --show  --n 20000` 



Fitting an image (split and prune to target): \

`fit_image_gaussians  --show --n 1000 --target 20000` 



See `--help` for other options.



### benchmarks



There exist benchmarks to evaluate performance on individual components in isolation under `taichi_splatting/benchmarks/`



### tests 



Tests (gradient tests and tests comparing to torch-based reference implementations) can be run with pytest, or individually under 

`taichi_splatting/tests/`



### splat-viewer



A viewer for reconstructions created with the original gaussian-splatting repository can be found [here](https://github.com/uc-vision/splat-viewer) or installed with pip. Has dependencies on open3d and Qt. 



### splat-benchmark



A benchmark for a full rendererer (in the same repository as above) with real reconstructions (rendering the original camera viewpoints).  Options exist for tweaking all the renderer parameters, benchmarking backward pass etc.



## Progress



### Done

* Benchmarks with original + taichi_3dgs rasterizer



* Simple view culling 

* Projection with autograd

* Tile mapping (optimized and improved culling) 

* Rasterizer forward pass and optimized backward pass



* Spherical harmonics with autograd

* Gradient tests for most parts (float64) - including rasterizer!

* Fit to image training example/test

* Depth and depth-covariance rendering



* Compute point visibility in backward pass (useful for model pruning)

* Example training on images with split/prune operations

* Novel heuristics for split and prune operations computed optionally in backward pass



### Todo



* Backward projection autograd takes a while to compile and is not cached properly

* 16 bit representations of parameters

* Depth rendering/regularization method (e.g. 2DGS or related method)

* Some ideas for optimized tilemapper with flat representations (no inner loop)



### Improvements



* Exposed all internal constants as parameters

* Switched to matrices as inputs instead of quaternions

* Tile mapping tighter culling for tile overlaps (~30% less rendered splats!)

* All configuration parameters exposed (e.g. tile_size, saturation threshold etc.)

* Warp reduction based backward pass for rasterizer, a decent boost in performance



## Conventions



### Transformation matrices



Transformations are notated `T_x_y`, for example `T_camera_world` can be used to transform points in the world to points in the local camera by `points_camera = T_camera_world @ points_world`