{"id":20988614,"url":"https://github.com/cvlab-columbia/pix2gestalt","last_synced_at":"2025-05-14T18:31:25.828Z","repository":{"id":218453818,"uuid":"734750039","full_name":"cvlab-columbia/pix2gestalt","owner":"cvlab-columbia","description":"Code for the paper \"pix2gestalt: Amodal Segmentation by Synthesizing Wholes\" (CVPR 2024)","archived":false,"fork":false,"pushed_at":"2024-05-03T02:48:53.000Z","size":15000,"stargazers_count":93,"open_issues_count":0,"forks_count":7,"subscribers_count":8,"default_branch":"main","last_synced_at":"2024-05-15T09:43:38.122Z","etag":null,"topics":["3d","amodal-instance-segmentation","amodal-perception","cvpr2024","recognition","stable-diffusion","zero-shot"],"latest_commit_sha":null,"homepage":"https://gestalt.cs.columbia.edu/","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"other","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/cvlab-columbia.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2023-12-22T14:08:34.000Z","updated_at":"2024-05-15T09:43:38.122Z","dependencies_parsed_at":"2024-05-03T03:55:36.184Z","dependency_job_id":null,"html_url":"https://github.com/cvlab-columbia/pix2gestalt","commit_stats":null,"previous_names":["cvlab-columbia/gestalt","cvlab-columbia/pix2gestalt"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/cvlab-columbia%2Fpix2gestalt","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/cvlab-columbia%2Fpix2gestalt/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/cvlab-columbia%2Fpix2gestalt/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/cvlab-columbia%2Fpix2gestalt/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/cvlab-columbia","download_url":"https://codeload.github.com/cvlab-columbia/pix2gestalt/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":225304497,"owners_count":17453173,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["3d","amodal-instance-segmentation","amodal-perception","cvpr2024","recognition","stable-diffusion","zero-shot"],"created_at":"2024-11-19T06:22:17.546Z","updated_at":"2024-11-19T06:22:18.079Z","avatar_url":"https://github.com/cvlab-columbia.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# pix2gestalt: Amodal Segmentation by Synthesizing Wholes\n### CVPR 2024 (Highlight)\n### [Project Page](https://gestalt.cs.columbia.edu/)  | [Paper](https://arxiv.org/pdf/2401.14398.pdf) | [arXiv](https://arxiv.org/abs/2401.14398) | [Weights](https://huggingface.co/cvlab/pix2gestalt-weights) | [Citation](https://github.com/cvlab-columbia/pix2gestalt#citation)\n\n[pix2gestalt: Amodal Segmentation by Synthesizing Wholes](https://gestalt.cs.columbia.edu/)  \n [Ege Ozguroglu](https://egeozguroglu.github.io/)\u003csup\u003e1\u003c/sup\u003e, [Ruoshi Liu](https://ruoshiliu.github.io/)\u003csup\u003e1\u003c/sup\u003e, [Dídac Surís](https://www.didacsuris.com/)\u003csup\u003e1\u003c/sup\u003e, [Dian Chen](https://scholar.google.com/citations?user=zdAyna8AAAAJ\u0026hl=en)\u003csup\u003e2\u003c/sup\u003e, [Achal Dave](https://www.achaldave.com/)\u003csup\u003e2\u003c/sup\u003e, [Pavel Tokmakov](https://pvtokmakov.github.io/home/)\u003csup\u003e2\u003c/sup\u003e, [Carl Vondrick](https://www.cs.columbia.edu/~vondrick/)\u003csup\u003e1\u003c/sup\u003e \u003cbr\u003e\n \u003csup\u003e1\u003c/sup\u003eColumbia University, \u003csup\u003e2\u003c/sup\u003eToyota Research Institute\n\n![teaser](./assets/teaser.gif \"Teaser\")\n\n## Updates\n- We have released our [training script](https://github.com/cvlab-columbia/pix2gestalt?tab=readme-ov-file#training), [dataset](https://github.com/cvlab-columbia/pix2gestalt?tab=readme-ov-file#dataset), and [Gradio demo](https://github.com/cvlab-columbia/pix2gestalt?tab=readme-ov-file#inference-and-weights) with inference instructions.\n- Custom training \u0026 fine-tuning instructions coming soon. Beyond amodal perception, our repository can also be used to fine-tune Stable Diffusion in an image-conditioned manner with spatial prompts, such as binary masks.\n- Pretrained models are released on [Huggingface](https://huggingface.co/cvlab/pix2gestalt-weights), more details provided [here](https://github.com/cvlab-columbia/pix2gestalt#inference-and-weights).  \n- pix2gestalt was accepted to CVPR 2024, available on [arXiv](https://arxiv.org/abs/2401.14398)!\n\n##  Installation\n```\nconda create -n pix2gestalt python=3.9\nconda activate pix2gestalt\ncd pix2gestalt\npip install -r requirements.txt\ngit clone https://github.com/CompVis/taming-transformers.git\npip install -e taming-transformers/\ngit clone https://github.com/openai/CLIP.git\npip install -e CLIP/\n```\nNote: We tested the installation processes on a system with Ubuntu 20.04 with NVIDIA GPUs using Ampere architecture. \n\n## Inference and Weights\n\nFirst, download the pix2gestalt weights under `pix2gestalt/ckpt` through one of the following sources:\n\n```\nhttps://huggingface.co/cvlab/pix2gestalt-weights/tree/main\n\nwget -c -P ./ckpt https://gestalt.cs.columbia.edu/assets/epoch=000005.ckpt\n```\nNote that we have released 2 model weights: epoch=000005.ckpt and epoch=000010.ckpt. By default, we use epoch=000005.ckpt which is the checkpoint after finetuning for 5 epochs on our [dataset](https://github.com/cvlab-columbia/pix2gestalt?tab=readme-ov-file#dataset). We have also released epoch=000010.ckpt, trained for 10 epochs. This checkpoint can be desirable for synthetic occlusion settings (given our dataset approach), though it may naturally suffer in zero-shot generalization compared to our default model.\n\nDownload [SAM](https://segment-anything.com/) checkpoints:\n```\nwget -c -P ./ckpt https://gestalt.cs.columbia.edu/assets/sam_vit_{b,h,l}.pth\n```\n\nRun our Gradio demo for amodal completion and segmentation:\n\n```\npython app.py\n```\n\nNote that this app uses 22-28 GB of VRAM, so it may not be possible to run it on any GPU.\n\nFor inference without the Gradio demo, we provide standalone functionality for each component [here](./pix2gestalt/inference.py), encapsulated by the [run_pix2gestalt](./pix2gestalt/inference.py#L138) method. It supports both predicted modal masks from SAM (like our demo) or ground truth modal masks. \n\n### Training\nDownload the image-conditioned Stable Diffusion checkpoint released by Lambda Labs: \n\n```\nwget -c -P ./ckpt https://gestalt.cs.columbia.edu/assets/sd-image-conditioned-v2.ckpt\n```\n\nThen, download our fine-tuning dataset via the instructions [here](https://github.com/cvlab-columbia/pix2gestalt?tab=readme-ov-file#dataset) and update its path (see `data:params:root_dir`) in our [config](./pix2gestalt/configs/sd-finetune-pix2gestalt-c_concat-256.yaml).\n\nRun training command:  \n```\npython main.py \\\n    -t \\\n    --base configs/sd-finetune-pix2gestalt-c_concat-256.yaml \\\n    --gpus 0,1,2,3,4,5,6,7 \\\n    --scale_lr False \\\n    --num_nodes 1 \\\n    --seed 42 \\\n    --check_val_every_n_epoch 2 \\\n    --finetune_from ckpt/sd-image-conditioned-v2.ckpt\n```\nNote that this training script is set for an 8-GPU system, each with 80GB of VRAM. Empirically, the large batch size is very important for \"stably\" fine-tuning Stable Diffusion in an image conditioned manner. If you have smaller GPUs, consider using smaller batch sizes with gradient accumulation to obtain a similar effective batch size.\n\n### Dataset\nDownload and extract our dataset of occluded objects \u0026 their whole counterparts with:\n```\nwget https://gestalt.cs.columbia.edu/assets/pix2gestalt_occlusions_release.tar.gz\n\ntar -xvf pix2gestalt_occlusions_release.tar.gz\n```\nDisclaimer: note that the source images are from the [Segment Anything-1B Dataset](https://segment-anything.com/dataset/index.html), which has faces and license plates de-identified. For amodal perception targeted specifically for such domains, we recommend re-training or fine-tuning pix2gestalt via our custom trainining instructions. \n\nThe dataset is intended for research purposes only. The licenses for the source images are released under the same license that they are in SA-1B.\n\n### Amodal Recognition and 3D Reconstruction\nSince we synthesize RGB images of whole objects (amodal completion), our approach makes it straightforward to equip various computer vision methods with the ability to handle occlusions, beyond amodal segmentation.\n\nFor recognition, we use [CLIP](https://github.com/openai/CLIP) as the base open-vocabulary classifier. For novel view synthesis and  3D reconstruction, we use [SyncDreamer](https://github.com/liuyuan-pal/SyncDreamer). Refer to our [paper](https://gestalt.cs.columbia.edu/static/pix2gestalt.pdf) and [supplementary](https://gestalt.cs.columbia.edu/static/supplementary.pdf) for more details.\n\n\n## Citation\nIf you use this code, please consider citing the paper as:\n```\n@article{ozguroglu2024pix2gestalt,\n        title={pix2gestalt: Amodal Segmentation by Synthesizing Wholes},\n        author={Ege Ozguroglu and Ruoshi Liu and D\\'idac Sur\\'s and Dian Chen and Achal Dave and Pavel Tokmakov and Carl Vondrick},\n        journal={Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)},\n        year={2024}\n}\n```\n\n##  Acknowledgement\nThis research is based on work partially supported by the Toyota Research Institute, the DARPA MCS program under Federal Agreement No. N660011924032, the NSF NRI Award \\#1925157, and the NSF AI Institute for Artificial and Natural Intelligence Award \\#2229929. DS is supported by the Microsoft PhD Fellowship.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcvlab-columbia%2Fpix2gestalt","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fcvlab-columbia%2Fpix2gestalt","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcvlab-columbia%2Fpix2gestalt/lists"}