{"id":19575661,"url":"https://github.com/belkakari/drawingwithgaussians","last_synced_at":"2025-04-27T06:31:27.848Z","repository":{"id":229865137,"uuid":"775065219","full_name":"belkakari/DrawingWithGaussians","owner":"belkakari","description":"2D and 3D Gaussian Splatting + tweaks in JAX","archived":false,"fork":false,"pushed_at":"2024-07-06T18:39:41.000Z","size":16249,"stargazers_count":7,"open_issues_count":1,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2024-07-06T19:48:49.357Z","etag":null,"topics":["gaussian-splatting","image-representation"],"latest_commit_sha":null,"homepage":"","language":"Jupyter Notebook","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"gpl-3.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/belkakari.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-03-20T17:38:55.000Z","updated_at":"2024-07-06T18:39:41.000Z","dependencies_parsed_at":"2024-06-07T08:48:33.390Z","dependency_job_id":"0e392c91-9c41-4818-8572-03b4257d1160","html_url":"https://github.com/belkakari/DrawingWithGaussians","commit_stats":null,"previous_names":["belkakari/drawingwithgaussians"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/belkakari%2FDrawingWithGaussians","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/belkakari%2FDrawingWithGaussians/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/belkakari%2FDrawingWithGaussians/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/belkakari%2FDrawingWithGaussians/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/belkakari","download_url":"https://codeload.github.com/belkakari/DrawingWithGaussians/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":224062752,"owners_count":17249290,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["gaussian-splatting","image-representation"],"created_at":"2024-11-11T06:49:05.080Z","updated_at":"2024-11-11T06:49:06.466Z","avatar_url":"https://github.com/belkakari.png","language":"Jupyter Notebook","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Experiments with 2D gaussians\n\n## Set up\n\n```bash\ngit clone https://github.com/belkakari/DrawingWithGaussians.git\ncd DrawingWithGaussians\npip install poetry\npoetry install\n```\n\nFor CUDA machine I suggest using the docker image provided or install jax with CUDA support following their official [installation guide](https://jax.readthedocs.io/en/latest/installation.html)\n\n## Overview\nThis is not a \"production-ready project\" by any means but rather my attempts at a low-level tweaking of different image representations based on gaussians and how to control them. I might move to 3D at some point but for now it's more about 2D and adapting some methods I like to 2D setup.\n\n## Fit 2D gaussians to an image\n\n```bash\npython fit.py --config-name fit_to_image.yaml\n```\n\n![An example of fitting an image](./static/eye_fitting.gif)\n\nHere I initialize 50 gaussians and split them every epoch based on the gradient values. After each epoch I multiply all of the gaussians colors by `cfg.gaussians.color_demp_coeff` which is 0.1 here\n\n## Fit 2D gaussians with a diffusion prior\n\n```bash\npython fit.py --config-name diffusion_guidance.yaml\n```\n\nPrompts is `A man standing on the street`, it nicely detereorates in an abstract image probably due to the low number of gaussians (50-\u003e88-\u003e159-\u003e256-\u003e378), annealing is on, img2img every 50 steps\n\n![An example of fitting an image with a diffusion guidance](./static/diffusion_guided.gif)\n\nAnother example without annealing but with img2img every 250 steps\n\n![An example of fitting an image with a diffusion guidance](./static/diffusion_guided_2.gif)\n\nThis is not SDS but rather something similar to [StableDreamer](https://arxiv.org/abs/2312.02189), meaning that I just use img2img mode of SD as a ground truth. No gradients are being passed through the sampling process\nThere is a lot to be optimized in terms of the sampling itself (e.g. different checkpoint, different scheduler)\n\nA bit about the config:\n- `img2img_freq` sets up how often to run img2img, 1 means every step, \u003e1 means that gaussians are being optimised for a diffusion image for a img2img_freq steps\n- `height`/`width` sets up the dimensions in which SD is being inferred since SD1.4 (I know, I'll pick a fancier one later) is quite sensitive to resolution. It is downsampled to `image.height`/`image.width` after inference for gaussians optimization\n- `strength_annealing` toggles diffusion strength scaling so that in the beggining it's set to `strength`, but at `num_epochs * num_steps` it becomes zero (the idea is that with more optimization steps one does not need to diffuse that much)\n\n## ToDO\n- [x] Move boilerplate to separate functions\n- [x] Add SSIM\n- [ ] Ability to copy optimizer state from before the pruning (copy for the splitted gaussians)\n- [ ] Test \"deferred rendering\" like in [SpacetimeGaussians](https://oppo-us-research.github.io/SpacetimeGaussians-website/)\n- [x] Add SDS with SD\n- [ ] Add basic 3D version\n- [ ] Add alternative alpha-composing with occlusions (prune gaussians based on opacity, currently prunning based on color norm, probably won't do this untill I'll decide to move to 3D)\n\n## References\nBased on [3DGS](https://repo-sam.inria.fr/fungraph/3d-gaussian-splatting/), [fmb-plus](https://leonidk.com/fmb-plus/), [GaussianImage](https://arxiv.org/abs/2403.08551), works ok on macbook m1 up to ~300 gaussians\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fbelkakari%2Fdrawingwithgaussians","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fbelkakari%2Fdrawingwithgaussians","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fbelkakari%2Fdrawingwithgaussians/lists"}