{"id":13585187,"url":"https://github.com/Mayukhdeb/torch-dreams","last_synced_at":"2025-04-07T06:32:46.184Z","repository":{"id":52808035,"uuid":"291035531","full_name":"Mayukhdeb/torch-dreams","owner":"Mayukhdeb","description":"Flexible Feature visualization on PyTorch, for research and art :mag_right: :computer: :brain: :art:","archived":false,"fork":false,"pushed_at":"2025-03-17T23:17:00.000Z","size":158542,"stargazers_count":239,"open_issues_count":8,"forks_count":19,"subscribers_count":7,"default_branch":"master","last_synced_at":"2025-03-29T02:05:57.807Z","etag":null,"topics":["colab","deep-dream","deep-learning","feature-visualization","pytorch"],"latest_commit_sha":null,"homepage":"https://pypi.org/project/torch-dreams/","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/Mayukhdeb.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2020-08-28T12:00:58.000Z","updated_at":"2025-03-17T23:17:04.000Z","dependencies_parsed_at":"2024-03-25T06:26:54.559Z","dependency_job_id":"81c55342-8ec6-4b39-89c9-0dbcc8461892","html_url":"https://github.com/Mayukhdeb/torch-dreams","commit_stats":{"total_commits":428,"total_committers":6,"mean_commits":71.33333333333333,"dds":0.2079439252336449,"last_synced_commit":"91904b2437106956aca1e546ce5956d32470f29f"},"previous_names":[],"tags_count":12,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Mayukhdeb%2Ftorch-dreams","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Mayukhdeb%2Ftorch-dreams/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Mayukhdeb%2Ftorch-dreams/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Mayukhdeb%2Ftorch-dreams/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/Mayukhdeb","download_url":"https://codeload.github.com/Mayukhdeb/torch-dreams/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":247280190,"owners_count":20912966,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["colab","deep-dream","deep-learning","feature-visualization","pytorch"],"created_at":"2024-08-01T15:04:47.183Z","updated_at":"2025-04-07T06:32:46.178Z","avatar_url":"https://github.com/Mayukhdeb.png","language":"Python","funding_links":[],"categories":["Python"],"sub_categories":[],"readme":"# Torch-Dreams\nMaking neural networks more interpretable, for research and art. \n\n\n[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/Mayukhdeb/torch-dreams-notebooks/blob/main/docs_notebooks/hello_torch_dreams.ipynb)\n[![build](https://github.com/Mayukhdeb/torch-dreams/actions/workflows/main.yml/badge.svg)](https://github.com/Mayukhdeb/torch-dreams/actions/workflows/main.yml)\n[![codecov](https://codecov.io/gh/Mayukhdeb/torch-dreams/branch/master/graph/badge.svg?token=krU6dNleoJ)](https://codecov.io/gh/Mayukhdeb/torch-dreams)\n\u003c!-- [![](https://img.shields.io/twitter/url?label=Docs\u0026style=flat-square\u0026url=https%3A%2F%2Fapp.gitbook.com%2F%40mayukh09%2Fs%2Ftorch-dreams%2F)](https://app.gitbook.com/@mayukh09/s/torch-dreams/) --\u003e\n\n\n\u003cimg src = \"https://github.com/Mayukhdeb/torch-dreams/blob/master/images/banner_segmentation_model.png?raw=true\"\u003e\n\n```\npip install torch-dreams \n```\n\n## Contents:\n\n* [Minimal example](https://github.com/Mayukhdeb/torch-dreams#minimal-example)\n* [Not so minimal example](https://github.com/Mayukhdeb/torch-dreams#not-so-minimal-example)\n* [Visualizing individual channels with `custom_func`](https://github.com/Mayukhdeb/torch-dreams#visualizing-individual-channels-with-custom_func)\n* [Caricatures](https://github.com/Mayukhdeb/torch-dreams#caricatures)\n* [Visualize features from multiple models simultaneously](https://github.com/Mayukhdeb/torch-dreams#visualize-features-from-multiple-models-simultaneously)\n* [Use custom transforms](https://github.com/Mayukhdeb/torch-dreams#using-custom-transforms)\n* [Feedback loops](https://github.com/Mayukhdeb/torch-dreams#you-can-also-use-outputs-of-one-render-as-the-input-of-another-to-create-feedback-loops)\n* [Custom images](https://github.com/Mayukhdeb/torch-dreams#using-custom-images)\n* [Working on models with different image normalizations](https://github.com/Mayukhdeb/torch-dreams#working-on-models-with-different-image-normalizations)\n* [Masked image parametrs](https://github.com/Mayukhdeb/torch-dreams#masked-image-parameters)\n* [Other conveniences](https://github.com/Mayukhdeb/torch-dreams#other-conveniences)\n* [Development](https://github.com/Mayukhdeb/torch-dreams#development)\n\n## Minimal example\n\u003e Make sure you also check out the [quick start colab notebook](https://colab.research.google.com/github/Mayukhdeb/torch-dreams-notebooks/blob/main/docs_notebooks/hello_torch_dreams.ipynb) \n\n\n```python\nimport matplotlib.pyplot as plt\nimport torchvision.models as models\nfrom torch_dreams import Dreamer\n\nmodel = models.inception_v3(pretrained=True)\ndreamy_boi = Dreamer(model, device = 'cuda')\n\nimage_param = dreamy_boi.render(\n    layers = [model.Mixed_5b],\n)\n\nplt.imshow(image_param)\nplt.show()\n```\n\n## Not so minimal example\n```python\nmodel = models.inception_v3(pretrained=True)\ndreamy_boi = Dreamer(model, device = 'cuda', quiet = False)\n\nimage_param = dreamy_boi.render(\n    layers = [model.Mixed_5b],\n    width = 256,\n    height = 256,\n    iters = 150,\n    lr = 9e-3,\n    rotate_degrees = 15,\n    scale_max = 1.2,\n    scale_min =  0.5,\n    translate_x = 0.2,\n    translate_y = 0.2,\n    custom_func = None,\n    weight_decay = 1e-2,\n    grad_clip = 1.,\n)\n\nplt.imshow(image_param)\nplt.show()\n```\n\n## Visualizing individual channels with `custom_func`\n\n```python\nmodel = models.inception_v3(pretrained=True)\ndreamy_boi = Dreamer(model, device = 'cuda')\n\nlayers_to_use = [model.Mixed_6b.branch1x1.conv]\n\ndef make_custom_func(layer_number = 0, channel_number= 0): \n    def custom_func(layer_outputs):\n        loss = layer_outputs[layer_number][:, channel_number].mean()\n        return -loss\n    return custom_func\n\nmy_custom_func = make_custom_func(layer_number= 0, channel_number = 119)\n\nimage_param = dreamy_boi.render(\n    layers = layers_to_use,\n    custom_func = my_custom_func,\n)\nplt.imshow(image_param)\nplt.show()\n```\n\n## Batched generation for large scale experiments\n\nThe `BatchedAutoImageParam` paired with the `BatchedObjective` can be used to generate multiple feature visualizations in parallel. This takes up more memory based on the batch size, but is also faster than generating one visualization at a time.\n\n```python\nfrom torch_dreams import Dreamer\nimport torchvision.models as models\nfrom torch_dreams.batched_objective import BatchedObjective\nfrom torch_dreams.batched_image_param import BatchedAutoImageParam\n\nmodel = models.inception_v3(pretrained=True)\ndreamy_boi = Dreamer(model, device=\"cuda\")\n\n## specify list of neuron indices to visualize\nbatch_neuron_indices = [i for i in range(10,20)]\n\n## set up a batch of trainable image parameters\nbap = BatchedAutoImageParam(\n    batch_size=len(batch_neuron_indices), \n    width=256, \n    height=256, \n    standard_deviation=0.01\n)\n\n## objective generator for each neuron\ndef make_custom_func(layer_number=0, channel_number=0):\n    def custom_func(layer_outputs):\n        loss = layer_outputs[layer_number][:, channel_number].norm()\n        return -loss\n\n    return custom_func\n\n## prepare objective functions for each neuron index\nbatched_objective = BatchedObjective(\n    objectives=[make_custom_func(channel_number=i) for i in batch_neuron_indices]\n)\n\n## render activation maximization signals\nresult_batch = dreamy_boi.render(\n    layers=[model.Mixed_5b],\n    image_parameter=bap,\n    iters=120,\n    custom_func=batched_objective,\n)\n\n## save results in a folder\nfor i in batch_neuron_indices:\n    result_batch[batch_neuron_indices.index(i)].save(f\"results/{i}.jpg\")\n```\n\n## Caricatures\n\nCaricatures create a new image that has a similar but more extreme activation pattern to the input image at a given layer (or multiple layers at a time). It's inspired from [this issue](https://github.com/tensorflow/lucid/issues/121)\n\n\u003cimg src = \"https://raw.githubusercontent.com/Mayukhdeb/torch-dreams/master/images/caricature.png\" width = \"70%\"\u003e\n\nIn this case, let's use googlenet \n\n```python\nmodel = models.googlenet(pretrained = True)\ndreamy_boi = Dreamer(model = model, quiet= False, device= 'cuda')\n\nimage_param = dreamy_boi.caricature(\n    input_tensor = image_tensor, \n    layers = [model.inception4c],   ## feel free to append more layers for more interesting caricatures \n    power= 1.2,                     ## higher -\u003e more \"exaggerated\" features\n)\n\nplt.imshow(image_param)\nplt.show()\n```\n## Visualize features from multiple models on a single image parameter\n\nFirst, let's pick 2 models and specify which layers we'd want to work with\n\n```python\nfrom torch_dreams.model_bunch import ModelBunch\n\nbunch = ModelBunch(\n    model_dict = {\n        'inception': models.inception_v3(pretrained=True).eval(),\n        'resnet':    models.resnet18(pretrained= True).eval()\n    }\n)\n\nlayers_to_use = [\n            bunch.model_dict['inception'].Mixed_6a,\n            bunch.model_dict['resnet'].layer2[0].conv1\n        ]\n\ndreamy_boi = Dreamer(model = bunch, quiet= False, device= 'cuda')\n```\n\nThen define a `custom_func` which determines which exact activations of the models we have to optimize\n\n```python\ndef custom_func(layer_outputs):\n    loss =   layer_outputs[0].mean()*2.0 + layer_outputs[1][:, 89].mean() \n    return -loss\n```\n\nRun the optimization\n\n```python\nimage_param = dreamy_boi.render(\n    layers = layers_to_use,\n    custom_func= custom_func,\n    iters= 100\n)\n\nplt.imshow(image_param)\nplt.show()\n```\n\n\n## Using custom transforms:\n\n```python\nimport torchvision.transforms as transforms\n\nmodel = models.inception_v3(pretrained=True)\ndreamy_boi = Dreamer(model,  device = 'cuda', quiet =  False)\n\nmy_transforms = transforms.Compose([\n    transforms.RandomAffine(degrees = 10, translate = (0.5,0.5)),\n    transforms.RandomHorizontalFlip(p = 0.3)\n])\n\ndreamy_boi.set_custom_transforms(transforms = my_transforms)\n\nimage_param = dreamy_boi.render(\n    layers = [model.Mixed_5b],\n)\n\nplt.imshow(image_param)\nplt.show()\n```\n\n## You can also use outputs of one `render()` as the input of another to create feedback loops.\n\n```python\nimport matplotlib.pyplot as plt\nimport torchvision.models as models\nfrom torch_dreams import Dreamer\n\nmodel = models.inception_v3(pretrained=True)\ndreamy_boi = Dreamer(model,  device = 'cuda', quiet =  False)\n\nimage_param = dreamy_boi.render(\n    layers = [model.Mixed_6c],\n)\n\nimage_param = dreamy_boi.render(\n    image_parameter= image_param,\n    layers = [model.Mixed_5b],\n    iters = 20\n)\n\nplt.imshow(image_param)\nplt.show()\n```\n\n## Using custom images\n\nNote that you might have to use smaller values for certain hyperparameters like `lr` and `grad_clip`.\n\n```python\nfrom torch_dreams.custom_image_param import CustomImageParam\nparam = CustomImageParam(image = 'images/sample_small.jpg', device= 'cuda')  ## image could either be a filename or a torch.tensor of shape NCHW\n\nimage_param = dreamy_boi.render(\n    image_parameter= param,\n    layers = [model.Mixed_6c],\n    lr = 2e-4,\n    grad_clip = 0.1,\n    weight_decay= 1e-1,\n    iters = 120\n)\n```\n\n## Working on models with different image normalizations\n\n`torch-dreams` generally works with models trained on images normalized with imagenet `mean` and `std`, but that can be easily overriden to support any other normalization. For example, if you have a model with `mean = [0.5, 0.5, 0.5]` and `std = [0.5, 0.5, 0.5]`: \n\n```python \nt = torchvision.transforms.Normalize(\n                mean = [0.5, 0.5, 0.5],\n                std =  [0.5, 0.5, 0.5]\n            )\n\ndreamy_boi.set_custom_normalization(normalization_transform = t) ## normalization_transform could be any instance of torch.nn.Module\n```\n\n## Masked Image parameters\n\nCan be used to optimize only certain parts of the image using a mask whose values are clipped between `[0,1]`.\n\n\u003cimg src = \"https://raw.githubusercontent.com/Mayukhdeb/torch-dreams/master/images/masked_param.png\" width = \"80%\"\u003e\n\nHere's an example with a vertical gradient \n\n```python \nfrom torch_dreams.masked_image_param import MaskedImageParam\n\nmask = torch.ones(1,1,512,512)\n\nfor i in range(0, 512, 1):  ## vertical gradient\n    mask[:,:,i,:] = (i/512)\n\nparam = MaskedImageParam(\n    image= 'images/sample_small.jpg',  ## optional\n    mask_tensor = mask,\n    device = 'cuda'\n)\n\nparam = dreamy_boi.render(\n    layers = [model.inception4c],\n    image_parameter= param,\n    lr = 1e-4,\n    grad_clip= 0.1,\n    weight_decay= 1e-1,\n    iters= 200,\n)\n\nparam.save('masked_param_output.jpg')\n```\n\nIt's also possible to update the mask on the fly with `param.update_mask(some_mask)`\n\n```python\n\nparam.update_mask(mask = torch.flip(mask, dims = (2,)))\n\nparam = dreamy_boi.render(\n    layers = [model.inception4a],\n    image_parameter= param,\n    lr = 1e-4,\n    grad_clip= 0.1,\n    weight_decay= 1e-1,\n    iters= 200,\n)\n\nparam.save('masked_param_output_2.jpg')\n```\n\n\n## Other conveniences \n\nThe following methods are handy for an [`auto_image_param`](https://github.com/Mayukhdeb/torch-dreams/blob/master/torch_dreams/auto_image_param.py) instance:\n\n1. Saving outputs as images:\n\n```python\nimage_param.save('output.jpg')\n```\n\n2. Torch Tensor of dimensions `(height, width, color_channels)`\n\n```python\ntorch_image = image_param.to_hwc_tensor(device = 'cpu')\n```\n\n3. Torch Tensor of dimensions `(color_channels, height, width)`\n\n```python\ntorch_image_chw = image_param.to_chw_tensor(device = 'cpu')\n```\n\n4. Displaying outputs on matplotlib. \n\n```python\nplt.imshow(image_param)\nplt.show()\n```\n\n5. For instances of `custom_image_param`, you can set any NCHW tensor as the image parameter: \n\n```python\nimage_tensor = image_param.to_nchw_tensor()\n\n## do some stuff with image_tensor\nt = transforms.Compose([\n    transforms.RandomRotation(5)\n])\ntransformed_image_tensor = t(image_tensor) \n\nimage_param.set_param(tensor = transformed_image_tensor)\n```\n\n## Args for `render()`\n\n* `layers` (`iterable`): List of the layers of model(s)'s layers to work on. `[model.layer1, model.layer2...]`\n* `image_parameter` (`auto_image_param`, optional): Instance of `torch_dreams.auto_image_param.auto_image_param`\n\n* `width` (`int`, optional): Width of image to be optimized \n* `height` (`int`, optional): Height of image to be optimized \n* `iters` (`int`, optional): Number of iterations, higher -\u003e stronger visualization\n* `lr` (`float`, optional): Learning rate\n* `rotate_degrees` (`int`, optional): Max rotation in default transforms\n* `scale_max` (`float`, optional): Max image size factor. Defaults to 1.1.\n* `scale_min` (`float`, optional): Minimum image size factor. Defaults to 0.5.\n* `translate_x` (`float`, optional): Maximum translation factor in x direction\n* `translate_y` (`float`, optional): Maximum translation factor in y direction\n* `custom_func` (`function`, optional): Can be used to define custom optimiziation conditions to `render()`. Defaults to None.\n* `weight_decay` (`float`, optional): Weight decay for default optimizer. Helps prevent high frequency noise. Defaults to 0.\n* `grad_clip` (`float`, optional): Maximum value of the norm of gradient. Defaults to 1.\n\n## Args for `Dreamer.__init__()`\n * `model` (`nn.Module` or  `torch_dreams.model_bunch.Modelbunch`): Almost any PyTorch model which was trained on imagenet `mean` and `std`, and supports variable sized images as inputs. You can pass multiple models into this argument as a `torch_dreams.model_bunch.Modelbunch` instance.\n * `quiet` (`bool`): Set to `True` if you want to disable any progress bars\n * `device` (`str`): `cuda` or `cpu` depending on your runtime \n\n ## Development\n\n1. Clone the repo and navigate into the folder\n```\ngit clone git@github.com:Mayukhdeb/torch-dreams.git\ncd torch-dreams/\n```\n\n2. Install dependencies\n```\npip install -r requirements.txt\n```\n\n3. Install `torch-dreams` as an editable module\n```\npython3 setup.py develop\n```\n\n## Citation\n```\n@misc{mayukhdebtorchdreams,\n  title={Feature Visualization library for PyTorch},\n  author={Mayukh Deb},\n  year={2021},\n  publisher={GitHub},\n  howpublished={\\url{https://github.com/Mayukhdeb/torch-dreams}},\n}\n```\n\n## Acknowledgements\n\n* [amFOSS](https://amfoss.in/)\n* [Gene Kogan](https://github.com/genekogan) \n\n## Recommended Reading \n\n* [Feature Visualization](https://distill.pub/2017/feature-visualization/)\n* [Google AI blog on Deepdreams](https://ai.googleblog.com/2015/06/inceptionism-going-deeper-into-neural.html)\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FMayukhdeb%2Ftorch-dreams","html_url":"https://awesome.ecosyste.ms/projects/github.com%2FMayukhdeb%2Ftorch-dreams","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FMayukhdeb%2Ftorch-dreams/lists"}