Ecosyste.ms: Awesome

An open API service indexing awesome lists of open source software.

Awesome Lists | Featured Topics | Projects

https://github.com/castorini/daam

Diffusion attentive attribution maps for interpreting Stable Diffusion.
https://github.com/castorini/daam

diffusion explainable-ai generative-ai huggingface pytorch stable-diffusion

Last synced: about 5 hours ago
JSON representation

Diffusion attentive attribution maps for interpreting Stable Diffusion.

Awesome Lists containing this project

README 

        
# What the DAAM: Interpreting Stable Diffusion Using Cross Attention



[![HF Spaces](https://img.shields.io/badge/HuggingFace%20Space-online-green.svg)](https://huggingface.co/spaces/tetrisd/Diffusion-Attentive-Attribution-Maps) [![Citation](https://img.shields.io/badge/Citation-ACL-orange.svg)](https://gist.github.com/daemon/639de6fea584d7df1a62f04a2ea0cdad) [![PyPi version](https://badgen.net/pypi/v/daam?color=blue)](https://pypi.org/project/daam) [![Downloads](https://static.pepy.tech/badge/daam)](https://pepy.tech/project/daam)



![example image](example.jpg)



### Updated to support Stable Diffusion XL (SDXL) and Diffusers 0.21.1!



I regularly update this codebase. Please submit an issue if you have any questions.



In [our paper](https://aclanthology.org/2023.acl-long.310), we propose diffusion attentive attribution maps (DAAM), a cross attention-based approach for interpreting Stable Diffusion.

Check out our demo: https://huggingface.co/spaces/tetrisd/Diffusion-Attentive-Attribution-Maps.

See our [documentation](https://castorini.github.io/daam/), hosted by GitHub pages, and [our Colab notebook](https://colab.research.google.com/drive/1miGauqa07uHnDoe81NmbmtTtnupmlipv?usp=sharing), updated for v0.1.0.



## Getting Started

First, install [PyTorch](https://pytorch.org) for your platform.

Then, install DAAM with `pip install daam`, unless you want an editable version of the library, in which case do `git clone https://github.com/castorini/daam && pip install -e daam`.

Finally, login using `huggingface-cli login` to get many stable diffusion models -- you'll need to get a token at [HuggingFace.co](https://huggingface.co/).



### Running the Website Demo

Simply run `daam-demo` in a shell and navigate to http://localhost:8080.

The same demo as the one on HuggingFace Spaces will show up.



### Using DAAM as a CLI Utility

DAAM comes with a simple generation script for people who want to quickly try it out.

Try running

```bash

$ mkdir -p daam-test && cd daam-test

$ daam "A dog running across the field."

$ ls

a.heat_map.png    field.heat_map.png    generation.pt   output.png  seed.txt

dog.heat_map.png  running.heat_map.png  prompt.txt

```

Your current working directory will now contain the generated image as `output.png` and a DAAM map for every word, as well as some auxiliary data.

You can see more options for `daam` by running `daam -h`.

To use Stable Diffusion XL as the backend, run `daam --model xl-base-1.0 "Dog jumping"`.  



### Using DAAM as a Library



Import and use DAAM as follows:



```python

from daam import trace, set_seed

from diffusers import DiffusionPipeline

from matplotlib import pyplot as plt

import torch



model_id = 'stabilityai/stable-diffusion-xl-base-1.0'

device = 'cuda'



pipe = DiffusionPipeline.from_pretrained(model_id, use_auth_token=True, torch_dtype=torch.float16, use_safetensors=True, variant='fp16')

pipe = pipe.to(device)



prompt = 'A dog runs across the field'

gen = set_seed(0)  # for reproducibility



with torch.no_grad():

    with trace(pipe) as tc:

        out = pipe(prompt, num_inference_steps=50, generator=gen)

        heat_map = tc.compute_global_heat_map()

        heat_map = heat_map.compute_word_heat_map('dog')

        heat_map.plot_overlay(out.images[0])

        plt.show()

```



You can also serialize and deserialize the DAAM maps pretty easily:



```python

from daam import GenerationExperiment, trace



with trace(pipe) as tc:

    pipe('A dog and a cat')

    exp = tc.to_experiment('experiment-dir')

    exp.save()  # experiment-dir now contains all the data and heat maps



exp = GenerationExperiment.load('experiment-dir')  # load the experiment

```



We'll continue adding docs.

In the meantime, check out the `GenerationExperiment`, `GlobalHeatMap`, and `DiffusionHeatMapHooker` classes, as well as the `daam/run/*.py` example scripts.

You can download the COCO-Gen dataset from the paper at http://ralphtang.com/coco-gen.tar.gz.

If clicking the link doesn't work on your browser, copy and paste it in a new tab, or use a CLI utility such as `wget`.



## See Also

- [DAAM-i2i](https://github.com/RishiDarkDevil/daam-i2i), an extension of DAAM to image-to-image attribution.



- [Furkan's video](https://www.youtube.com/watch?v=XiKyEKJrTLQ) on easily getting started with DAAM.



- [1littlecoder's video](https://www.youtube.com/watch?v=J2WtkA1Xfew) for a code demonstration and Colab notebook of an older version of DAAM.



## Citation

```

@inproceedings{tang2023daam,

    title = "What the {DAAM}: Interpreting Stable Diffusion Using Cross Attention",

    author = "Tang, Raphael  and

      Liu, Linqing  and

      Pandey, Akshat  and

      Jiang, Zhiying  and

      Yang, Gefei  and

      Kumar, Karun  and

      Stenetorp, Pontus  and

      Lin, Jimmy  and

      Ture, Ferhan",

    booktitle = "Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)",

    year = "2023",

    url = "https://aclanthology.org/2023.acl-long.310",

}

```