Ecosyste.ms: Awesome
An open API service indexing awesome lists of open source software.
https://github.com/castorini/daam
Diffusion attentive attribution maps for interpreting Stable Diffusion.
https://github.com/castorini/daam
diffusion explainable-ai generative-ai huggingface pytorch stable-diffusion
Last synced: 3 days ago
JSON representation
Diffusion attentive attribution maps for interpreting Stable Diffusion.
- Host: GitHub
- URL: https://github.com/castorini/daam
- Owner: castorini
- License: mit
- Created: 2022-10-08T23:02:12.000Z (over 2 years ago)
- Default Branch: main
- Last Pushed: 2024-04-05T20:02:28.000Z (10 months ago)
- Last Synced: 2024-04-25T01:01:36.076Z (9 months ago)
- Topics: diffusion, explainable-ai, generative-ai, huggingface, pytorch, stable-diffusion
- Language: Jupyter Notebook
- Homepage:
- Size: 2.15 MB
- Stars: 601
- Watchers: 12
- Forks: 61
- Open Issues: 10
-
Metadata Files:
- Readme: README.md
- License: LICENSE
Awesome Lists containing this project
- awesome-stable-diffusion - DAAM - Diffusion attention attribution maps, generating heatmaps modelling the impact of specific terms and tokens in the prompt on the final diffusion result. (Complementary Models and Tools / Potentially Stale/Less active branches)
README
# What the DAAM: Interpreting Stable Diffusion Using Cross Attention
[](https://huggingface.co/spaces/tetrisd/Diffusion-Attentive-Attribution-Maps) [](https://gist.github.com/daemon/639de6fea584d7df1a62f04a2ea0cdad) [](https://pypi.org/project/daam) [](https://pepy.tech/project/daam)
### Updated to support Stable Diffusion XL (SDXL) and Diffusers 0.21.1!
I regularly update this codebase. Please submit an issue if you have any questions.
In [our paper](https://aclanthology.org/2023.acl-long.310), we propose diffusion attentive attribution maps (DAAM), a cross attention-based approach for interpreting Stable Diffusion.
Check out our demo: https://huggingface.co/spaces/tetrisd/Diffusion-Attentive-Attribution-Maps.
See our [documentation](https://castorini.github.io/daam/), hosted by GitHub pages, and [our Colab notebook](https://colab.research.google.com/drive/1miGauqa07uHnDoe81NmbmtTtnupmlipv?usp=sharing), updated for v0.1.0.
## Getting Started
First, install [PyTorch](https://pytorch.org) for your platform.
Then, install DAAM with `pip install daam`, unless you want an editable version of the library, in which case do `git clone https://github.com/castorini/daam && pip install -e daam`.
Finally, login using `huggingface-cli login` to get many stable diffusion models -- you'll need to get a token at [HuggingFace.co](https://huggingface.co/).
### Running the Website Demo
Simply run `daam-demo` in a shell and navigate to http://localhost:8080.
The same demo as the one on HuggingFace Spaces will show up.
### Using DAAM as a CLI Utility
DAAM comes with a simple generation script for people who want to quickly try it out.
Try running
```bash
$ mkdir -p daam-test && cd daam-test
$ daam "A dog running across the field."
$ ls
a.heat_map.png field.heat_map.png generation.pt output.png seed.txt
dog.heat_map.png running.heat_map.png prompt.txt
```
Your current working directory will now contain the generated image as `output.png` and a DAAM map for every word, as well as some auxiliary data.
You can see more options for `daam` by running `daam -h`.
To use Stable Diffusion XL as the backend, run `daam --model xl-base-1.0 "Dog jumping"`.
### Using DAAM as a Library
Import and use DAAM as follows:
```python
from daam import trace, set_seed
from diffusers import DiffusionPipeline
from matplotlib import pyplot as plt
import torch
model_id = 'stabilityai/stable-diffusion-xl-base-1.0'
device = 'cuda'
pipe = DiffusionPipeline.from_pretrained(model_id, use_auth_token=True, torch_dtype=torch.float16, use_safetensors=True, variant='fp16')
pipe = pipe.to(device)
prompt = 'A dog runs across the field'
gen = set_seed(0) # for reproducibility
with torch.no_grad():
with trace(pipe) as tc:
out = pipe(prompt, num_inference_steps=50, generator=gen)
heat_map = tc.compute_global_heat_map()
heat_map = heat_map.compute_word_heat_map('dog')
heat_map.plot_overlay(out.images[0])
plt.show()
```
You can also serialize and deserialize the DAAM maps pretty easily:
```python
from daam import GenerationExperiment, trace
with trace(pipe) as tc:
pipe('A dog and a cat')
exp = tc.to_experiment('experiment-dir')
exp.save() # experiment-dir now contains all the data and heat maps
exp = GenerationExperiment.load('experiment-dir') # load the experiment
```
We'll continue adding docs.
In the meantime, check out the `GenerationExperiment`, `GlobalHeatMap`, and `DiffusionHeatMapHooker` classes, as well as the `daam/run/*.py` example scripts.
You can download the COCO-Gen dataset from the paper at http://ralphtang.com/coco-gen.tar.gz.
If clicking the link doesn't work on your browser, copy and paste it in a new tab, or use a CLI utility such as `wget`.
## See Also
- [DAAM-i2i](https://github.com/RishiDarkDevil/daam-i2i), an extension of DAAM to image-to-image attribution.
- [Furkan's video](https://www.youtube.com/watch?v=XiKyEKJrTLQ) on easily getting started with DAAM.
- [1littlecoder's video](https://www.youtube.com/watch?v=J2WtkA1Xfew) for a code demonstration and Colab notebook of an older version of DAAM.
## Citation
```
@inproceedings{tang2023daam,
title = "What the {DAAM}: Interpreting Stable Diffusion Using Cross Attention",
author = "Tang, Raphael and
Liu, Linqing and
Pandey, Akshat and
Jiang, Zhiying and
Yang, Gefei and
Kumar, Karun and
Stenetorp, Pontus and
Lin, Jimmy and
Ture, Ferhan",
booktitle = "Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)",
year = "2023",
url = "https://aclanthology.org/2023.acl-long.310",
}
```