{"id":15787237,"url":"https://github.com/ysig/diff-mining","last_synced_at":"2025-08-20T12:32:26.564Z","repository":{"id":247485250,"uuid":"792861000","full_name":"ysig/diff-mining","owner":"ysig","description":"Diffusion base mining","archived":false,"fork":false,"pushed_at":"2024-10-08T12:54:16.000Z","size":21071,"stargazers_count":48,"open_issues_count":1,"forks_count":4,"subscribers_count":2,"default_branch":"main","last_synced_at":"2024-10-30T10:32:08.744Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"other","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/ysig.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-04-27T18:55:07.000Z","updated_at":"2024-10-24T22:46:03.000Z","dependencies_parsed_at":"2024-07-09T02:46:17.644Z","dependency_job_id":"f33d4f21-7eaa-41fe-b900-716778c29b81","html_url":"https://github.com/ysig/diff-mining","commit_stats":{"total_commits":26,"total_committers":3,"mean_commits":8.666666666666666,"dds":"0.15384615384615385","last_synced_commit":"1194b8c264087153872a28fc10a0c4c795198225"},"previous_names":["ysig/diff-mining"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ysig%2Fdiff-mining","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ysig%2Fdiff-mining/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ysig%2Fdiff-mining/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ysig%2Fdiff-mining/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/ysig","download_url":"https://codeload.github.com/ysig/diff-mining/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":230423564,"owners_count":18223435,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-10-04T21:06:44.367Z","updated_at":"2024-12-19T11:13:21.469Z","avatar_url":"https://github.com/ysig.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"\u003cdiv align=\"center\"\u003e\n\u003ch2\u003eDiffusion Models as Data Mining Tools\n\u003cp\u003e\u003c/p\u003e\n\u003ca href=\"https://imagine.enpc.fr/~siglidii/\"\u003eIoannis Siglidis\u003c/a\u003e, \u003ca href=\"https://holynski.org/\"\u003eAleksander Hołyński\u003c/a\u003e,\n\u003ca href=\"https://people.eecs.berkeley.edu/~efros/\"\u003eAlexei A. Efros\u003c/a\u003e, \u003cbr/\u003e\n\u003ca href=\"https://imagine.enpc.fr/~aubrym/\"\u003eMathieu Aubry\u003c/a\u003e,\n\u003ca href=\"https://people.eecs.berkeley.edu/~shiry/\"\u003eShiry Ginosar\u003c/a\u003e\n\n\u003cp\u003e\u003c/p\u003e\n\u003ca href=\"https://diff-mining.github.io/\"\u003e\u003cimg \nsrc=\"https://img.shields.io/badge/-Webpage-blue.svg?colorA=333\u0026logo=html5\" height=35em\u003e\u003c/a\u003e\n\u003ca href=\"https://arxiv.org/abs/2307.05473\"\u003e\u003cimg \nsrc=\"https://img.shields.io/badge/-Paper-blue.svg?colorA=333\u0026logo=arxiv\" height=35em\u003e\u003c/a\u003e\n\u003ca href=\"https://diff-mining.github.io/ref.bib\"\u003e\u003cimg \nsrc=\"https://img.shields.io/badge/-BibTeX-blue.svg?colorA=333\u0026logo=latex\" height=35em\u003e\u003c/a\u003e\n\u003cp\u003e\u003c/p\u003e\n\n![teaser.png](./.media/Teaser.png)\n\n\u003c/h2\u003e\n\u003c/div\u003e\n\nOfficial PyTorch implementation of [Diffusion Models as Data Mining Tools](https://imagine.enpc.fr/~ioannis.siglidis/diff-mining), which has been accepted in ECCV'24.\n\n## Introduction \nOur approach allows you to take a large labelled input dataset, and mine the patches that are important for each label.\nIt involves three steps: \n1. First you finetune Stable-Diffusion v1.5 with its standard loss $L_t(x, \\epsilon, c)$ with prompts of the form $\\text{\"An image of Y\"}$ (where Y is your label) in your custom dataset.  \n2. For a sample of your input data you want to analyze, you then compute typicality $\\mathbf{T}(x|c) =  \\mathbb{E}_{\\epsilon,t}[L_t(x, \\epsilon, \\varnothing) - L_t(x, \\epsilon, c)]$ for all images.\n3. You extract the top-1000 patches according to $\\mathbf{T}(x | c)$ and then you cluster them using DIFT-161 features (ranking clusters according to median typicality of their elements).\n\n### Installation 🌱\nOur codebase is mainly developed on [diffusers](https://github.com/huggingface/diffusers/) implementation of [LDMs](https://arxiv.org/abs/2112.10752).\n```bash\nconda env create -f environment.yaml\nconda activate diff-mining\n```\n\n### Data 💽\nWe apply our method in 5 different types of datasets: cars ([CarDB](https://pages.cs.wisc.edu/~yongjaelee/projects/lee_efros_hebert_bookchapter.pdf)), faces ([FTT](https://facesthroughtime.github.io/)), street-view images ([G^3](https://arxiv.org/abs/2211.15521)), scenes ([Places](http://places2.csail.mit.edu/download.html), high-res) and X-rays ([ChestX-ray](https://arxiv.org/abs/1705.02315)):\n\n- A properly extracted version of CarDB can be found [here](https://huggingface.co/datasets/diff-mining/cardb) and can be downloaded with:\n```bash\npython scripts/download-cardb.py\n```\n- FTT: you can request access for downloading the dataset in the [original project page](https://facesthroughtime.github.io/).: \n- G^3: unfortunately proprietary but information about PanoramaIDs can be found on the [original repo](https://github.com/g-luo/geolocation_via_guidebook_grounding).\n- Places: you can request access (trivial to get) from [original project page](https://forms.gle/w4VoNMED3hgb1m3AA).\n\n### Models 🔬\nWe share our models on huggingface which you can access through the handles:\n- [`diff-mining/cardb`](https://huggingface.co/diff-mining/cardb)\n- [`diff-mining/ftt`](https://huggingface.co/diff-mining/ftt)\n- [`diff-mining/g3`](https://huggingface.co/diff-mining/g3)\n- [`diff-mining/xray`](https://huggingface.co/diff-mining/xray)\n- [`diff-mining/places`](https://huggingface.co/diff-mining/places)\n\nor download them locally using:\n```bash\npython scripts/download-models.py\n```\n\n## Approach \n\nA full walkthrough of the pipeline can be seen in scripts: `scripts/training.sh` and `scripts/typicality.sh`.\n- Code for finetuning models can be found under: `diffmining/finetuning/`.\n- Code for computing typicality can be found at: `diffmining/typicality/compute.py`.\n- Code for averaging typicality across patches, computing DIFT features and clustering can be found at: `diffmining/typicality/cluster.py`\n\n### Applications🔸\n\nWe test our typicality measure in two different approaches which we properly discuss in our paper.\n\n#### Clustering of Translated Visual Elements \nUsing our diffusion model, we can translate each image, e.g. in the case of geography, from one country to another.\nWe use [PnP](https://arxiv.org/abs/2211.12572) which is the only method we found that was relatively robust in keeping a consistency between translated objects (i.e., windows would remain windows).\nYou can launch this translation by running:\n```bash\nsource scripts/parallel.sh translate\n```\n\nAfterwards you need to compute typicality for all elements:\n```bash\nsource scripts/parallel.sh compute\n```\n\nand then cluster them using:\n```bash\nsource scripts/parallel.sh cluster\n```\n\n#### Emergent Disease Localization in X-rays 🩻\nAs typicality is connected to a binary classifier of the conditional vs the null conditioning, it can be used to \"spatialize\" information related to the condition on the input image.\nWe test this on X-ray images and show how typicality is improved after finetuning. To reproduce our results and evaluations run:\n```bash\nsource scripts/xray.sh\n```\n\n### Comparing with Doersch et al. 2012 🥐\nWe provide a minimal optimized implementation of the algorithm of [\"What makes Paris look like Paris?\"](http://graphics.cs.cmu.edu/projects/whatMakesParis/) under `doersch/`.\nRunning the code should only require:\n```bash\npython doersch.py --which geo --category 'Italy'\n```\nyet you will probably have to adjust it to the dataset of choice.\n\n### Citing 💫\n\n```bibtex\n  @article{diff-mining,\n    title = {Diffusion Models as Data Mining Tools},\n    author = {Siglidis, Ioannis and Holynski, Aleksander and Efros, A. Alexei and Aubry, Mathieu and Ginosar, Shiry},\n    journal = {ECCV},\n    year = {2024},\n  }\n```\n\n### Acknowledgements \nThis work was partly supported by the European Research Council (ERC project DISCOVER, number 101076028) and leveraged the HPC resources of IDRIS under the allocation AD011012905R1, AD0110129052 made\nby GENCI. We would like to thank Grace Luo for data, code, and discussion; Loic Landreu and David Picard for insights on geographical representations and diffusion; Karl Doersch, for project advice and implementation insights; Sophia Koepke for feedback on our manuscript.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fysig%2Fdiff-mining","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fysig%2Fdiff-mining","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fysig%2Fdiff-mining/lists"}