{"id":23500619,"url":"https://github.com/KaKituken/affordance-aware-any","last_synced_at":"2025-08-28T00:32:04.739Z","repository":{"id":268412398,"uuid":"904267783","full_name":"KaKituken/affordance-aware-any","owner":"KaKituken","description":"Affordance-Aware Object Insertion via Mask-Aware Dual Diffusion","archived":false,"fork":false,"pushed_at":"2025-02-21T04:45:25.000Z","size":54860,"stargazers_count":35,"open_issues_count":6,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-02-21T05:26:58.828Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"bsd-3-clause","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/KaKituken.png","metadata":{"files":{"readme":"Readme.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-12-16T15:04:57.000Z","updated_at":"2025-02-21T04:45:29.000Z","dependencies_parsed_at":"2025-02-21T05:33:37.455Z","dependency_job_id":null,"html_url":"https://github.com/KaKituken/affordance-aware-any","commit_stats":null,"previous_names":["kakituken/affordance-aware-any"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/KaKituken/affordance-aware-any","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/KaKituken%2Faffordance-aware-any","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/KaKituken%2Faffordance-aware-any/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/KaKituken%2Faffordance-aware-any/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/KaKituken%2Faffordance-aware-any/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/KaKituken","download_url":"https://codeload.github.com/KaKituken/affordance-aware-any/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/KaKituken%2Faffordance-aware-any/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":272405092,"owners_count":24929227,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-08-27T02:00:09.397Z","response_time":76,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-12-25T07:00:29.454Z","updated_at":"2025-08-28T00:32:04.717Z","avatar_url":"https://github.com/KaKituken.png","language":"Python","funding_links":[],"categories":["Datasets","Paper List","Poster"],"sub_categories":["Follow-up Papers"],"readme":"# Affordance-Aware Object Insertion via Mask-Aware Dual Diffusion\n\n[![arXiv](https://badgen.net/badge/arXiv/2412.14462/red?cache=300)](https://arxiv.org/abs/2412.14462) [![License](https://badgen.net/badge/License/BSD%203-Clause%20License?color=blue\u0026cache=300)](https://github.com/KaKituken/affordance-aware-any/blob/main/LICENSE) [![Project Page](https://badgen.net/badge/Project%20Page/Visit/green?icon=github)](https://kakituken.github.io/affordance-any.github.io/) \n\n[**Installation**](#-installation) | [**Dataset**](#-dataset) | [**Checkpoint**](#-checkpoint) |[**Demo**](#-demo) | [**Training \u0026 Evaluation**](#-training-\u0026-evaluation)\n\nThis repository maintains the official implementation of the paper **Affordance-Aware Object Insertion via Mask-Aware Dual Diffusion** by [Jixuan He](https://kakituken.github.io/home), [Wanhua Li](https://li-wanhua.github.io/),  [Ye Liu](https://yeliu.dev/), [Junsik Kim](https://sites.google.com/site/jskimcv/), [Donglai Wei](https://donglaiw.github.io/) and [Hanspeter Pfister](https://vcg.seas.harvard.edu/people/).\n\n![](./imgs/teaser_new.png)\n\n[[arXiv]](https://arxiv.org/abs/2412.14462)[[Project Page]](https://kakituken.github.io/affordance-any.github.io/)\n\n## 🔨 Installation\n\nRun the following instruction to set up the environment.\n\n```sh\ncd model\nconda env create -f environment.yaml\n```\n\n\n\n## 🏷️ Dataset\n\n### SAM-FB Dataset\n\nWe build the SAM-FB dataset on top of original [SA-1B](https://ai.meta.com/datasets/segment-anything/) dataset.\n\nDue to the storage limitation, you can download the first 10 sub-folder of SAM-FB dataset [here](https://huggingface.co/datasets/Kakituken/SAM-FB/tree/main). \n\nTo build the rest of the dataset, please download the [SA-1B](https://ai.meta.com/datasets/segment-anything/) dataset and follow the instructions in the next section.\n\n### Build Your Own Dataset\n\nOur automatic pipeline allows user to build their own dataset by providing only the images. You can either download the original [SA-1B](https://ai.meta.com/datasets/segment-anything/) dataset or provide a set of images under `./data` folder.\n\n#### 1. Download SAM\nFirst, download the SAM checkpoint from [here](https://github.com/facebookresearch/segment-anything#model-checkpoints) and put it int the `./ckpt` folder.\n\n#### 2. Get the masks\nSaying you have a folder from SA-1B, `./data/sa_000010/`\n\nPlease run the following instruction to get the mask after NMS\n```sh\ncd dataset-builder/img_data\n\npython mask_nms.py \\\n\t--image-dir ./data/sa_000010/ \\\n\t--save-dir ./data/sa_000010/ \\\n\t--model-path ./ckpt/sam_vit_h_4b8939.pth\n```\n\n#### 3. Inpainting\nAfter getting the masks, run following to inpaint the background and construct pair data. The filter will also perform here.\n\nDownload the checkpoint [here](https://huggingface.co/Kakituken/affordance-insertion-any/blob/main/epoch_2000.pth) for foreground filtering.\n\n```sh\npython inpainting_sa1b.py \\\n\t--annotation-path ./data/sa_000010/ \\\n\t--image-path ./data/sa_000010/ \\\n\t--save-path ./SAM-FB/ \\\n\t--model-path ./ckpt/quality_ctrl.pth\n```\n\n### Video Dataset (Youtube-VOS)\nWe also support video dataset. For [YTB-VOS]()-like dataset, simply run\n\n```sh\ncd dataset-builder/video_data\n\npython vdata_process.py \\\n\t-a ./data/Youtube-VOS/train/Annotations \\\n\t-i ./data/Youtube-VOS/train/JPEGImages \\\n\t-s ./SAM-FB-video\n```\n\n\n\n## 📌 Checkpoint\n\n\u003cimg src=\"./imgs/video_demo.gif\" style=\"zoom:150%;\" /\u003e\n\nWe provide two checkpoints for both 256x256 and 512x512 resolution. Download them here\n\n- 256 x 256: [Hugging Face](https://huggingface.co/Kakituken/affordance-insertion-any/resolve/main/affordance_256.ckpt?download=true)\n- 512 x 512: [Hugging Face](https://huggingface.co/Kakituken/affordance-insertion-any/tree/main/affordance_512.ckpt)\n\nPut them at `./ckpt/affordance_256.ckpt` or  `./ckpt/affordance_512.ckpt`.\n\n\n\n## 💻 Demo\n\nYou can run the demo on your local machine using `gradio` by running\n\n```sh\ncd model/script\npython infer_gui.py\n```\n\nYou can also do inference on single image by \n\n```sh\npython inference_dual.py \\\n    -b path/to/background.png \\\n    -f path/to/foreground.png \\\n    -m path/to/mask.png \\\n    -s path/to/save \\\n    -c ../config/affordance_512.yaml \\\n    -k ../../ckpt/affordance_512.ckpt \\\n    --num-predict 4 \\\n    --num-repeats 4\n```\n\n\n\n## 🚀 Training \u0026 Evaluation\n\nIf you hope to train the model by yourself, please follow the instructions.\n\n### 1. Download SD-inpainting checkpoint\n\nInstead of training from the scratch, we load the `sd1.5-inpainting` checkpoint as initailization. You can download the checkpoint [here](https://huggingface.co/runwayml/stable-diffusion-inpainting/resolve/main/sd-v1-5-inpainting.ckpt), and put it into the `ckpt` folder.\n\n### 2. Modify the config\n\nIf you want to train on 512 x 512 resolution, please modify the `./config/affordance_512.yaml`. Change all the item with `TODO` to fit your own dataset/checkpoint path.\n\n### 3. Training \u0026 Eval\n\nThen, run \n\n```sh\nexport CUDA_VISIBLE_DEVICES=0,1,\npython main.py \\\n    --base config/affordance_512.yaml \\\n    -t \\\n    --gpus 2 \\\n    --name affordance_512 \\\n    --logdir ./logs/affordance_512\n```\n\nto train the model. You can then view the evaluation curve using `tensorboard` by\n\n```sh\ntensorboard --logdir=./logs/affordance_512\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FKaKituken%2Faffordance-aware-any","html_url":"https://awesome.ecosyste.ms/projects/github.com%2FKaKituken%2Faffordance-aware-any","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FKaKituken%2Faffordance-aware-any/lists"}