{"id":13561358,"url":"https://github.com/threedle/3d-paintbrush","last_synced_at":"2025-04-03T17:30:43.935Z","repository":{"id":207440287,"uuid":"710596170","full_name":"threedle/3d-paintbrush","owner":"threedle","description":"Local text-driven editing of 3D shapes with Cascaded Score Distillation","archived":false,"fork":false,"pushed_at":"2024-06-23T20:41:36.000Z","size":50150,"stargazers_count":69,"open_issues_count":0,"forks_count":10,"subscribers_count":9,"default_branch":"main","last_synced_at":"2024-11-04T13:37:18.290Z","etag":null,"topics":["computer-graphics","computer-vision","deep-learning","differentiable-rendering","diffusion-models","generative-ai","machine-learning","meshes","neural-fields","python","pytorch","research","score-distillation","texturing"],"latest_commit_sha":null,"homepage":"https://threedle.github.io/3d-paintbrush/","language":"Jupyter Notebook","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/threedle.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2023-10-27T03:07:12.000Z","updated_at":"2024-10-28T04:39:54.000Z","dependencies_parsed_at":"2024-06-23T21:40:34.901Z","dependency_job_id":null,"html_url":"https://github.com/threedle/3d-paintbrush","commit_stats":null,"previous_names":["threedle/3d-paintbrush"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/threedle%2F3d-paintbrush","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/threedle%2F3d-paintbrush/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/threedle%2F3d-paintbrush/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/threedle%2F3d-paintbrush/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/threedle","download_url":"https://codeload.github.com/threedle/3d-paintbrush/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":247046771,"owners_count":20874719,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["computer-graphics","computer-vision","deep-learning","differentiable-rendering","diffusion-models","generative-ai","machine-learning","meshes","neural-fields","python","pytorch","research","score-distillation","texturing"],"created_at":"2024-08-01T13:00:55.261Z","updated_at":"2025-04-03T17:30:39.532Z","avatar_url":"https://github.com/threedle.png","language":"Jupyter Notebook","funding_links":[],"categories":["Jupyter Notebook"],"sub_categories":[],"readme":"# 🎨🖌️ 3D Paintbrush [CVPR 2024]\n\n*[Dale Decatur](https://ddecatur.github.io/), [Itai Lang](https://itailang.github.io/), [Kfir Aberman](https://kfiraberman.github.io/), [Rana Hanocka](https://people.cs.uchicago.edu/~ranahanocka/)*\n\n\u003ca href=\"https://threedle.github.io/3d-paintbrush/\"\u003e\u003cimg src=\"https://img.shields.io/website?down_color=lightgrey\u0026down_message=offline\u0026label=Project%20Page\u0026up_color=lightgreen\u0026up_message=online\u0026url=https%3A//threedle.github.io/3d-paintbrush\" height=22\u003e\u003c/a\u003e\n\u003ca href=\"https://arxiv.org/abs/2311.09571\"\u003e\u003cimg src=\"https://img.shields.io/badge/arXiv-3DPaintbrush-b31b1b.svg\" height=22\u003e\u003c/a\u003e\n\n\u003c!-- [[Project Page](https://threedle.github.io/3d-paintbrush/)] [[ArXiv](https://arxiv.org/abs/2311.09571)] --\u003e\n\n![](https://github.com/threedle/3d-paintbrush/raw/docs/docs/spot/gold_chain_necklace.gif)![](https://github.com/threedle/3d-paintbrush/raw/docs/docs/spot/heart-shaped_sunglasses.gif)![](https://github.com/threedle/3d-paintbrush/raw/docs/docs/spot/colorful_crochet_hat.gif)\n\n### Abstract\n*In this work we develop 3D Paintbrush, a technique for automatically texturing local semantic regions on meshes via text descriptions.\nOur method is designed to operate directly on meshes, producing texture maps which seamlessly integrate into standard graphics pipelines.\nWe opt to simultaneously produce a localization map (to specify the edit region) and a texture map which conforms to it.\nThis synergistic approach improves the quality of both the localization and the stylization.\nTo enhance the details and resolution of the textured area, we leverage multiple stages of a cascaded diffusion model to supervise our local editing technique with generative priors learned from images at different resolutions.\nOur technique, referred to as Cascaded Score Distillation (CSD), simultaneously distills scores at multiple resolutions in a cascaded fashion, enabling control over both the granularity and global understanding of the supervision.\nWe demonstrate the effectiveness of 3D Paintbrush to locally texture a variety of shapes within different semantic regions.*\n\n## CSD Demo Notebook\nTo see a demo of the Cascaded Score Distillation (CSD) loss, check out this [notebook](src/csd_demo.ipynb) applying CSD to image generation and image editing. CSD enables us to supervise our optimization with multiple cascaded stages of the diffusion model instead of just the base stage used in standard SDS. We distill scores across multiple stages of a cascaded diffusion model simultaneously in order to leverage both the global awareness of the first stage and the higher level of detail contained in later stages.\n\u003cp align=\"left\"\u003e\n  \u003cimg src=\"https://github.com/threedle/3d-paintbrush/raw/site/static/images/figures/super-res-csd.png\" width=\"500px\"/\u003e\n\u003c/p\u003e\n\n## Getting Started\n### Requirements\n- 48 GB GPU\n- CUDA 11.3\n- Python 3.10\n\nIf you have less than 48 GB of GPU memory, you can still run the code, see the section on [memory optimization](#memory-optimization) for more details.\n\n### Setup environemnt\nFirst create the conda environment:\n```\nconda create -n \"3d-paintbrush\" python=3.10\n```\nand activate it with:\n```\nconda activate 3d-paintbrush\n```\nNext install the required packages by running the install script. **Make sure to run this script with access to a GPU**.\n```\nbash ./install_environment.sh\n```\n\n### Login to Hugging Face (to use DeepFloyd IF w/ Diffusers)\nInstructions from [DeepFloyd IF](https://github.com/deep-floyd/IF):\n1) If you do not already have one, create a [Hugging Face account](https://huggingface.co/join)\n2) Accept the license on the model card of [DeepFloyd/IF-I-XL-v1.0](https://huggingface.co/DeepFloyd/IF-I-XL-v1.0)\n3) Log in to Hugging face locally. First install `huggingface_hub`\n```\npip install huggingface_hub --upgrade\n```\nrun the login function in a python shell\n```\nfrom huggingface_hub import login\n\nlogin()\n```\nand enter your [Hugging Face Hub access token](https://huggingface.co/docs/hub/security-tokens#what-are-user-access-tokens).\n\n## Reproduce paper results\n### (Optional) From Pre-trained\nTo use our pre-trained models download both the `trained_models` and `inverse_map_cache` folders from [here](https://drive.google.com/drive/folders/1kEHMtwgIGzv94Xbb8J-UaAdQ-an7drkd?usp=sharing) and add them under the data folder to create the following directory structure:\n```\n├── data\n│   ├── inverse_map_cache\n│   ├── trained_models\n│   ├── hand.obj\n...\n│   ├── spot.obj\n```\nTo run the pre-trained models, use the commands below. Results will be saved at `results/[name-of-mesh]/[name-of-edit]/renders/infernce.png`.\n\nSpot:\n```\npython src/main.py --config_path demo/spot/gold_chain_necklace.yaml --log.inference true --log.model_path ./data/trained_models/spot/gold_chain_necklace.pth\npython src/main.py --config_path demo/spot/heart-shaped_sunglasses.yaml --log.inference true --log.model_path ./data/trained_models/spot/heart-shaped_sunglasses.pth\npython src/main.py --config_path demo/spot/colorful_crochet_hat.yaml --log.inference true --log.model_path ./data/trained_models/spot/colorful_crochet_hat.pth\n```\n\u003cp align=\"left\"\u003e\n  \u003cimg src=\"https://github.com/threedle/3d-paintbrush/raw/docs/docs/spot/gold_chain_necklace_prob.png\" width=\"133px\"/\u003e\n  \u003cimg src=\"https://github.com/threedle/3d-paintbrush/raw/docs/docs/spot/gold_chain_necklace.png\" width=\"133px\"/\u003e\n  \u003cimg src=\"https://github.com/threedle/3d-paintbrush/raw/docs/docs/spot/heart-shaped_sunglasses_prob.png\" width=\"133px\"/\u003e\n  \u003cimg src=\"https://github.com/threedle/3d-paintbrush/raw/docs/docs/spot/heart-shaped_sunglasses.png\" width=\"133px\"/\u003e\n  \u003cimg src=\"https://github.com/threedle/3d-paintbrush/raw/docs/docs/spot/colorful_crochet_hat_prob.png\" width=\"133px\"/\u003e\n  \u003cimg src=\"https://github.com/threedle/3d-paintbrush/raw/docs/docs/spot/colorful_crochet_hat.png\" width=\"133px\"/\u003e\n\u003c/p\u003e\n\nPerson:\n```\npython src/main.py --config_path demo/person/tie-dye_apron.yaml --log.inference true --log.model_path ./data/trained_models/person/tie-dye_apron.pth\npython src/main.py --config_path demo/person/colorful_polo_shirt.yaml --log.inference true --log.model_path ./data/trained_models/person/colorful_polo_shirt.pth\npython src/main.py --config_path demo/person/superman_chest_emblem.yaml --log.inference true --log.model_path ./data/trained_models/person/superman_chest_emblem.pth\n```\n\u003cp align=\"left\"\u003e\n  \u003cimg src=\"https://github.com/threedle/3d-paintbrush/raw/docs/docs/person/tie-dye_apron_prob.png\" width=\"133px\"/\u003e\n  \u003cimg src=\"https://github.com/threedle/3d-paintbrush/raw/docs/docs/person/tie-dye_apron.png\" width=\"133px\"/\u003e\n  \u003cimg src=\"https://github.com/threedle/3d-paintbrush/raw/docs/docs/person/colorful_polo_shirt_prob.png\" width=\"133px\"/\u003e\n  \u003cimg src=\"https://github.com/threedle/3d-paintbrush/raw/docs/docs/person/colorful_polo_shirt.png\" width=\"133px\"/\u003e\n  \u003cimg src=\"https://github.com/threedle/3d-paintbrush/raw/docs/docs/person/superman_emblem_prob.png\" width=\"133px\"/\u003e\n  \u003cimg src=\"https://github.com/threedle/3d-paintbrush/raw/docs/docs/person/superman_emblem.png\" width=\"133px\"/\u003e\n\u003c/p\u003e\n\nLego Minfigure:\n```\npython src/main.py --config_path demo/lego_minifig/barcelona_jersey.yaml --log.inference true --log.model_path ./data/trained_models/lego_minifig/barcelona_jersey.pth\npython src/main.py --config_path demo/lego_minifig/blue_denim_overalls.yaml --log.inference true --log.model_path ./data/trained_models/lego_minifig/blue_denim_overalls.pth\npython src/main.py --config_path demo/lego_minifig/red_bow_tie.yaml --log.inference true --log.model_path ./data/trained_models/lego_minifig/red_bow_tie.pth\n```\n\u003cp align=\"left\"\u003e\n  \u003cimg src=\"https://github.com/threedle/3d-paintbrush/raw/docs/docs/lego_minifig/barcelona_jersey_prob.png\" width=\"133px\"/\u003e\n  \u003cimg src=\"https://github.com/threedle/3d-paintbrush/raw/docs/docs/lego_minifig/barcelona_jersey.png\" width=\"133px\"/\u003e\n  \u003cimg src=\"https://github.com/threedle/3d-paintbrush/raw/docs/docs/lego_minifig/blue_denim_overalls_prob.png\" width=\"133px\"/\u003e\n  \u003cimg src=\"https://github.com/threedle/3d-paintbrush/raw/docs/docs/lego_minifig/blue_denim_overalls.png\" width=\"133px\"/\u003e\n  \u003cimg src=\"https://github.com/threedle/3d-paintbrush/raw/docs/docs/lego_minifig/red_bow_tie_prob.png\" width=\"133px\"/\u003e\n  \u003cimg src=\"https://github.com/threedle/3d-paintbrush/raw/docs/docs/lego_minifig/red_bow_tie.png\" width=\"133px\"/\u003e\n\u003c/p\u003e\n\nHand:\n```\npython src/main.py --config_path demo/hand/fancy_gold_watch.yaml --log.inference true --log.model_path ./data/trained_models/hand/fancy_gold_watch.pth\n```\n\u003cp align=\"left\"\u003e\n  \u003cimg src=\"https://github.com/threedle/3d-paintbrush/raw/docs/docs/hand/fancy_gold_watch_prob.png\" width=\"133px\"/\u003e\n  \u003cimg src=\"https://github.com/threedle/3d-paintbrush/raw/docs/docs/hand/fancy_gold_watch.png\" width=\"133px\"/\u003e\n\u003c/p\u003e\n\n### From Scratch\nTo reproduce results from the paper from scratch, just pass a predefined demo config file. Results will be saved at `results/[name-of-mesh]/[name-of-edit]`.\n\nSpot:\n```\npython src/main.py --config_path demo/spot/gold_chain_necklace.yaml\npython src/main.py --config_path demo/spot/heart-shaped_sunglasses.yaml\npython src/main.py --config_path demo/spot/colorful_crochet_hat.yaml\n```\n\nPerson:\n```\npython src/main.py --config_path demo/person/tie-dye_apron.yaml\npython src/main.py --config_path demo/person/colorful_polo.yaml\npython src/main.py --config_path demo/person/superman_chest_emblem.yaml\n```\n\nLego Minifigure:\n```\npython src/main.py --config_path demo/lego_minifig/barcelona_jersey.yaml\npython src/main.py --config_path demo/lego_minifig/blue_denim_overalls.yaml\npython src/main.py --config_path demo/lego_minifig/red_bow_tie.yaml\n```\n\nHand:\n```\npython src/main.py --config_path demo/hand/fancy_gold_watch.yaml\n```\n\n## Run your own examples\nTo run your own examples you can create your own config files and pass those as done in the prior section. Additionally, you can instead pass values for any of the fields in `src/configs/train_config` as command line arguments. For example, to run the hand example without passing a config file, you may call:\n```\npython src/main.py --log.exp_dir results/hand/fancy_gold_watch --mesh.path ./data/spot.obj --guidance.object_name \"hand\" --guidance.style \"fancy gold\" --guidance.edit \"watch\"\n```\n\n## Memory Optimization\nIf you do not have access to a 48 GB GPU, you can...\n1) Enable CPU offloading by setting the flag `cpu_offload` to `True` in `src/configs/guidance_config.py`. This will significanly reduce memory usage, but comes at the cost of speed.\n2) Use a smaller batch size by changing the `batch_size` parameter either with the command line argument `--optim.batch_size` or in a custom config file. While this reduces memory usage, it can affect the accuracy.\n3) Sample a subset of surface points each iteration (set `sample_points` to `True` and adjust `mlp_batch_size`). While this reduces memory usage, it can affect the accuracy.\n4) Turn off batched score distillation calls (set `batched_sd` to `False`). This will reduce memory usage, but comes at the cost of speed.\n\n## Acknowledgements\nOur codebase is based on [Latent-NeRF/Latent-Paint](https://github.com/eladrich/latent-nerf) and our CSD guidance code is structured in the format of [ThreeStudio](https://github.com/threestudio-project/threestudio)'s guidance modules. We thank these authors for their amazing work.\n\n## Citation\nIf you find this code helpful for your research, please cite our paper [3D Paintbrush: Local Stylization of 3D Shapes with Cascaded Score Distillation](https://arxiv.org/abs/2311.09571).\n\n```\n@article{decatur2023paintbrush,\n    author = {Decatur, Dale and Lang, Itai and Aberman, Kfir and Hanocka, Rana},\n    title  = {3D Paintbrush: Local Stylization of 3D Shapes with Cascaded Score Distillation},\n    journal = {arXiv},\n    year = {2023}\n}\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fthreedle%2F3d-paintbrush","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fthreedle%2F3d-paintbrush","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fthreedle%2F3d-paintbrush/lists"}