{"id":15037305,"url":"https://github.com/csailvision/gandissect","last_synced_at":"2025-04-08T11:13:30.169Z","repository":{"id":37431098,"uuid":"158235160","full_name":"CSAILVision/gandissect","owner":"CSAILVision","description":"Pytorch-based tools for visualizing and understanding the neurons of a GAN.  https://gandissect.csail.mit.edu/","archived":false,"fork":false,"pushed_at":"2021-05-23T10:17:34.000Z","size":26641,"stargazers_count":1770,"open_issues_count":16,"forks_count":280,"subscribers_count":75,"default_branch":"master","last_synced_at":"2025-04-08T11:12:51.233Z","etag":null,"topics":["deep-learning","gan","generative-adversarial-network","image-manipulation","interactive-visualizations","interpretable-ml","pytorch"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/CSAILVision.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2018-11-19T14:16:08.000Z","updated_at":"2025-04-07T06:52:54.000Z","dependencies_parsed_at":"2022-08-19T06:10:52.342Z","dependency_job_id":null,"html_url":"https://github.com/CSAILVision/gandissect","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/CSAILVision%2Fgandissect","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/CSAILVision%2Fgandissect/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/CSAILVision%2Fgandissect/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/CSAILVision%2Fgandissect/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/CSAILVision","download_url":"https://codeload.github.com/CSAILVision/gandissect/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":247829512,"owners_count":21002997,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["deep-learning","gan","generative-adversarial-network","image-manipulation","interactive-visualizations","interpretable-ml","pytorch"],"created_at":"2024-09-24T20:34:14.003Z","updated_at":"2025-04-08T11:13:30.147Z","avatar_url":"https://github.com/CSAILVision.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# GANDissect \u003ca href=\"http://gandissect.res.ibm.com/ganpaint.html?project=churchoutdoor\u0026layer=layer4\"\u003e\u003cimg src='doc/ganpaint.gif' align=\"right\" height=\"259\"\u003e\u003c/a\u003e\n\n[**Project**](https://gandissect.csail.mit.edu/) | [**Demo**](http://gandissect.res.ibm.com/ganpaint.html?project=churchoutdoor\u0026layer=layer4) | [**Paper**](https://arxiv.org/pdf/1811.10597.pdf) | [**Video**](http://tiny.cc/gandissect) \u003cbr\u003e\n\n[GAN Dissection](https://gandissect.csail.mit.edu/) is a way to inspect the internal representations of a generative adversarial network (GAN) to understand how internal units align with human-interpretable concepts. It is part of [NetDissect](https://netdissect.csail.mit.edu/).\n\nThis repo allows you to dissect a GAN model. It provides the dissection results as a static summary or as an interactive visualization. Try our interactive [GANPaint demo](http://gandissect.res.ibm.com/ganpaint.html?project=churchoutdoor\u0026layer=layer4) to interact with GANs and draw images. \n\n## Overview\n\n\u003cimg src=\"doc/teaser.jpg\" width=\"800px\" clear=\"both\" /\u003e\n\n[Visualizing and Understanding Generative Adversarial Networks](http://gandissect.csail.mit.edu) \u003cbr\u003e\n[David Bau](http://people.csail.mit.eduÂ/davidbau/home/), [Jun-Yan Zhu](http://people.csail.mit.edu/junyanz/), [Hendrik Strobelt](http://hendrik.strobelt.com/), [Bolei Zhou](http://people.csail.mit.edu/bzhou/), [Joshua B. Tenenbaum](http://web.mit.edu/cocosci/josh.html), [William T. Freeman](https://billf.mit.edu/), [Antonio Torralba](http://web.mit.edu/torralba/www/) \u003cbr\u003e\nMIT CSAIL, MIT-IBM Watson AI Lab, CUHK, IBM Research \u003cbr\u003e\nIn arXiv, 2018.\n\n\n## Analysis and Applications\n### Interpretable Units in GANs\n\u003cimg src=\"doc/units.png\" width=\"800px\"/\u003e\n\n### Analyzing different layers\n\u003cimg src=\"doc/layers.png\" width=\"800px\"/\u003e\n\n\n### Diagnosing and improving GANs\n\u003cimg src=\"doc/artifacts.jpg\" width=\"800px\"/\u003e\n\n### Removing objects from conference rooms\n\u003cimg src=\"doc/ablation_confroom.jpg\" width=\"800px\"/\u003e\n\n### Removing windows from different natural scenes\n\u003cimg src=\"doc/ablation_window.jpg\" width=\"800px\"/\u003e\n\n### Inserting new objects into images\n\u003cimg src=\"doc/insert.jpg\" width=\"800px\"/\u003e\n\n\n## Release history\nv 0.9 alpha  - Nov 26,  2018 \u003cbr\u003e\n\n\n## Getting Started\nLet's set up the environment and dissect a `churchoutdoor` GAN. This requires some CUDA-enabled GPU and some disk space.\n\n### Setup\n\nTo install everything needed from this repo, have `conda` available,\nand run:\n\n```\nscript/setup_env.sh      # Create a conda environment with dependencies\nscript/make_dirs.sh      # Create the dataset and dissect directories\nscript/download_data.sh  # Download support data and demo GANs\nsource activate netd     # Enter the conda environment\npip install -v -e .      # Link the local netdissect package into the env\n```\n\nDetails.  The code depends on python 3, Pytorch 4.1, and several other\npackages.  For conda users, `script/environment.yml` provides the details\nof the dependencies.  For pip users, `setup.py` lists everything needed.\n\nData.  The `download_data.sh` script downloads the segmentation dataset\nused to dissect classifiers, the segmentation network used to dissect GANs,\nand several example GAN models to dissect.  The downloads will go into\nthe directories `dataset/` and `models/`.  If you do not wish to download\nthe example networks, `python -m netdissect --download` will download\njust the data and models needed for netdissect itself.\n\n\n### Dissecting a GAN\n\nGAN example: to dissect three layers of the LSUN living room progressive\nGAN trained by Karras:\n\n```\npython -m netdissect \\\n   --gan \\\n   --model \"netdissect.proggan.from_pth_file('models/karras/livingroom_lsun.pth')\" \\\n   --outdir \"dissect/livingroom\" \\\n   --layer layer1 layer4 layer7 \\\n   --size 1000\n```\n\nThe result is a static HTML page at `dissect/livingroom/dissect.html`, and\na JSON file of metrics at `dissect/livingroom/dissect.json`.\n\nYou can test your own model: the `--model` argument is a fully-qualified\npython function or constructor for loading the GAN to test.  The\n`--layer` names are fully-qualified (`state_dict`-style) names for layers.\n\nBy default, a scene-based segmentation is used but a different segmenter class\ncan be substituted by supplying an alternate class constructor to\n`--segmenter`.  See `netdissect/segmenter.py` for the segmenter base class.\n\n## Running a GAN editing server (alpha)\n\n\nOnce a GAN is dissected, you can run a web server that provides an API\nthat generates images with (optional) interventions.\n\n```\npython -m netdissect.server --address 0.0.0.0\n```\n\u003cimg src='doc/ganter_screenshot.png' width='40%' align='right' style=\"padding:3px;\"\u003e\n\n\n\nThe editing UI (right) is served at [http://localhost:5001/](http://localhost:5001/) .\n\nOther URLs:\n\n- [http://localhost:5001/api/ui](http://localhost:5001/api/ui) is the OpenAPI/swagger UI for directly\n    testing GAN interventions.\n- [http://localhost:5001/data/livingroom/dissect.html](http://localhost:5001/data/livingroom/dissect.html) static net\n    dissection reports.\n- [http://localhost:5001/data/livingroom/edit.html](http://localhost:5001/data/livingroom/edit.html) a dissection-based\n    interface for testing interventions.\n- TODO: [http://localhost:5001/ganpaint.html](http://localhost:5001/ganpaint.html) will serve GANpaint\n\n\n## Advanced Level\n\n### Dissecting a classifier (NetDissect)\n\nClassifier example: to dissect three layers of the pretrained `alexnet` in `torchvision`:\n```bash\npython -m netdissect \\\n   --model \"torchvision.models.alexnet(pretrained=True)\" \\\n   --layers features.6:conv3 features.8:conv4 features.10:conv5 \\\n   --imgsize 227 \\\n   --outdir dissect/alexnet-imagenet\n```\n\nNo special web server for a classifier.\n\n### Command Line Details\n\nDocumentation for the netdissect command-line utility.\n\n```\nusage: python -m netdissect [-h] [--model MODEL] [--pthfile PTHFILE]\n                            [--outdir OUTDIR] [--layers LAYERS [LAYERS ...]]\n                            [--segments SEGMENTS] [--segmenter SEGMENTER]\n                            [--download] [--imgsize IMGSIZE]\n                            [--netname NETNAME] [--meta META [META ...]]\n                            [--examples EXAMPLES] [--size SIZE]\n                            [--batch_size BATCH_SIZE]\n                            [--num_workers NUM_WORKERS]\n                            [--quantile_threshold {[0-1],iqr}] [--no-labels]\n                            [--maxiou] [--covariance] [--no-images]\n                            [--no-report] [--no-cuda] [--gen] [--gan]\n                            [--perturbation PERTURBATION] [--add_scale_offset]\n                            [--quiet]\n```\n\noptional arguments:\n\n```\n  -h, --help            show this help message and exit\n  --model MODEL         constructor for the model to test\n  --pthfile PTHFILE     filename of the .pth file for the model\n  --outdir OUTDIR       directory for dissection output\n  --layers LAYERS [LAYERS ...]\n                        space-separated list of layer names to dissect, in the\n                        form layername[:reportedname]\n  --segments SEGMENTS   directory containing segmentation dataset\n  --segmenter SEGMENTER\n                        constructor for a segmenter class\n  --download            downloads Broden dataset if needed\n  --imgsize IMGSIZE     input image size to use\n  --netname NETNAME     name for the network in generated reports\n  --meta META [META ...]\n                        json files of metadata to add to report\n  --examples EXAMPLES   number of image examples per unit\n  --size SIZE           dataset subset size to use\n  --batch_size BATCH_SIZE\n                        batch size for a forward pass\n  --num_workers NUM_WORKERS\n                        number of DataLoader workers\n  --quantile_threshold {[0-1],iqr}\n                        quantile to use for masks\n  --no-labels           disables labeling of units\n  --maxiou              enables maxiou calculation\n  --covariance          enables covariance calculation\n  --no-images           disables generation of unit images\n  --no-report           disables generation report summary\n  --no-cuda             disables CUDA usage\n  --gen                 test a generator model (e.g., a GAN)\n  --gan                 synonym for --gen\n  --perturbation PERTURBATION\n                        the filename of perturbation attack to apply\n  --add_scale_offset    offsets masks according to stride and padding\n  --quiet               silences console output\n```\n\n### API, for classifiers\n\nIt can be used from code as a function, as follows:\n\n1. Load up the convolutional model you wish to dissect, and call\n   `imodel = InstrumentedModel(model)` and then\n   `imodel.retain_layers([layernames,..])` to instrument the model.\n2. Load the segmentation dataset using the BrodenDataset class;\n   use the `transform_image` argument to normalize images to\n   be suitable for the model, and the `size` argument to truncate the dataset.\n3. Choose a directory in which to write the output, and call\n   `dissect(outdir, imodel, dataset)`.\n\nA quick approximate dissection can be done by reducing the `size`\nof the `BrodenDataset`.  Generating example images can be time-consuming\nand the number of images can be set via `examples_per_unit`.\n\nExample:\n\n```\n    from netdissect import InstrumentedModel, dissect\n    from netdissect import BrodenDataset\n\n    model = InstrumentedModel(load_my_model())\n    model.eval()\n    model.cuda()\n    model.retain_layers(['conv1', 'conv2', 'conv3', 'conv4', 'conv5'])\n    bds = BrodenDataset('dataset/broden1_227',\n            transform_image=transforms.Compose([\n                transforms.ToTensor(),\n                transforms.Normalize(IMAGE_MEAN, IMAGE_STDEV)]),\n            size=10000)\n    dissect('result/dissect', model, bds,\n            batch_size=100,\n            examples_per_unit=10)\n```\n\nThe Broden dataset is oriented towards semantic objects, parts, material, colors, etc that are found in natural scene photographs.\nIf you want to analyze your model with a different semantic segmentation, you can substitute a different segmentation dataset and supply a `segrunner`, an argument that describes how to get segmentations and RGB images from the dataset.  See `ClassifierSegRunner` for the details.\n\n### API, for generators\n\nSimilarly:\n\n1. Load up the generator model wish to dissect, and call\n   `retain_layers(model, [layernames,..])` to instrument the model.\n2. Create a dataset of z input samples for testing.  If your model\n   uses a uniform normal distribution, z_dataset_for_model will make one.\n3. Choose a directory in which to write the output, and call\n   `dissect(outdir, model, dataset, segrunner=GeneratorSegRunner())`.\n\nThe time for the dissection is proportional to the number of samples\nin the dataset.\n\n```\n    from netdissect import InstrumentedModel, dissect\n    from netdissect import z_dataset_for_model, GeneratorSegRunner\n\n    model = InstrumentedModel(load_my_model())\n    model.eval()\n    model.cuda()\n    model.retain_layers(model, ['layer3', 'layer4', 'layer5'])\n    zds = z_dataset_for_model(size, model)\n    dissect('result/gandissect', model, zds,\n            segrunner=GeneratorSegRunner(),\n            batch_size=100,\n            examples_per_unit=10)\n```\n\nThe `GeneratorSegRunner` defaults to a running a semantic segmentation network oriented towards semantic objects, parts, and materials found in natural scene photographs.  To use a different semantic segmentation, you can supply a custom `Segmenter` subclass to the constructor of `GeneratorSegRunner`.\n\n## Citation\nIf you use this code for your research, please cite our [paper](https://arxiv.org/pdf/1811.10597.pdf):\n```\n@inproceedings{bau2019gandissect,\n title={GAN Dissection: Visualizing and Understanding Generative Adversarial Networks},\n author={Bau, David and Zhu, Jun-Yan and Strobelt, Hendrik and Zhou, Bolei and Tenenbaum, Joshua B. and Freeman, William T. and Torralba, Antonio},\n booktitle={Proceedings of the International Conference on Learning Representations (ICLR)},\n year={2019}\n}\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcsailvision%2Fgandissect","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fcsailvision%2Fgandissect","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcsailvision%2Fgandissect/lists"}