{"id":13617621,"url":"https://github.com/davidbau/rewriting","last_synced_at":"2025-04-04T19:11:33.986Z","repository":{"id":47227464,"uuid":"283545320","full_name":"davidbau/rewriting","owner":"davidbau","description":"Rewriting a Deep Generative Model, ECCV 2020 (oral).  Interactive tool to directly edit the rules of a GAN to synthesize scenes with objects added, removed, or altered.  Change StyleGANv2 to make extravagant eyebrows, or horses wearing hats.","archived":false,"fork":false,"pushed_at":"2023-10-29T09:30:42.000Z","size":105167,"stargazers_count":535,"open_issues_count":8,"forks_count":77,"subscribers_count":18,"default_branch":"master","last_synced_at":"2025-03-28T18:12:09.141Z","etag":null,"topics":["deep-learning","gans","graphics","hci","machine-learning","research","vision"],"latest_commit_sha":null,"homepage":"https://rewriting.csail.mit.edu/","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/davidbau.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null}},"created_at":"2020-07-29T16:12:38.000Z","updated_at":"2025-01-17T09:41:16.000Z","dependencies_parsed_at":"2024-04-16T23:01:56.699Z","dependency_job_id":null,"html_url":"https://github.com/davidbau/rewriting","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/davidbau%2Frewriting","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/davidbau%2Frewriting/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/davidbau%2Frewriting/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/davidbau%2Frewriting/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/davidbau","download_url":"https://codeload.github.com/davidbau/rewriting/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":247234921,"owners_count":20905854,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["deep-learning","gans","graphics","hci","machine-learning","research","vision"],"created_at":"2024-08-01T20:01:44.876Z","updated_at":"2025-04-04T19:11:28.971Z","avatar_url":"https://github.com/davidbau.png","language":"Python","funding_links":[],"categories":["Python","5. Generation of synthetic content"],"sub_categories":["5.3 Generation Images"],"readme":"# Rewriting a Deep Generative Model\n\nIn this paper, we ask if a deep network can be reprogrammed to follow different rules, by enabling a user to directly change the weights, instead of training with a data set.\n\n\n\u003ctable\u003e\u003ctr\u003e\u003ctd\u003e\u003ca href=\"http://rewriting.csail.mit.edu/\"\u003e\u003cimg src='images/horse-hat-edit.gif'\u003e\u003c/a\u003e\u003cbr\u003e\nDirectly rewriting the weights of a StyleGANv2 to reprogram horses to have hats.\u003c/td\u003e\u003c/tr\u003e\u003c/table\u003e\n\n### What is model rewriting?\nWe present the task of *model rewriting*, which aims to add, remove, and alter the semantic and physical rules of a pre-trained deep network.  While modern image editing tools achieve a user-specified goal by manipulating individual input images, we enable a user to synthesize an unbounded number of new images by editing a generative model to carry out modified rules.\n\n\n### Why rewrite a model?\nThere are two reasons to want to rewrite a deep network directly:\n  1. To gain insight about how a deep network organizes its knowledge.\n  2. To enable creative users to quickly make novel models for which there is no existing data set.\n\nModel rewriting envisions a way to construct deep networks according to a user's intentions. Rather than limiting networks to imitating data that we already have, rewriting allows deep networks to model a world that follows new rules that a user wishes to have.\n\n\n[**Rewriting a Deep Generative Model**](https://rewriting.csail.mit.edu/).\u003cbr\u003e\n[David Bau](https://people.csail.mit.edu/davidbau/home/), [Steven Liu](http://people.csail.mit.edu/stevenliu/), [Tongzhou Wang](https://ssnl.github.io/), [Jun-Yan Zhu](https://www.cs.cmu.edu/~junyanz/), [Antonio Torralba](http://web.mit.edu/torralba/www/). \u003cbr\u003e\nECCV 2020 (oral).\u003cbr\u003e\nMIT CSAIL and Adobe Research.\n\n\u003ctable\u003e\u003ctr\u003e\n\u003ctd\u003e\u003ccenter\u003e\u003ca href=\"https://www.youtube.com/watch?v=i2_-zNqtEPk\" target=\"_blank\" class=\"d-inline-block p-3 align-bottom\"\u003e\u003cimg height=\"78\" width=\"136\" src=\"images/short-video-thumb.png\" style=\"border:1px solid\" data-nothumb=\"\"\u003e\u003cbr\u003e2 minute\u003cbr\u003eVideo Preview\u003c/a\u003e\u003c/center\u003e\u003c/td\u003e\n\u003ctd\u003e\u003ccenter\u003e\u003ca href=\"https://rewriting.csail.mit.edu/paper/\"\u003e\u003cimg height=\"100\" width=\"78\" src=\"images/paper-thumb.png\" style=\"border:1px solid\" data-nothumb=\"\"\u003e\u003cbr\u003eECCV 2020\u003cbr\u003ePreprint\u003c/a\u003e\u003c/center\u003e\u003c/td\u003e\n\u003ctd\u003e\u003ccenter\u003e\u003ca href=\"https://rewriting.csail.mit.edu/video/\" class=\"d-inline-block p-3 align-bottom\"\u003e\u003cimg height=\"78\" width=\"136\" src=\"images/video-thumb.png\" style=\"border:1px solid\" data-nothumb=\"\"\u003e\u003cbr\u003eECCV 2020\u003cbr\u003eTalk Video\u003c/a\u003e\u003c/center\u003e\u003c/td\u003e\n\u003ctd\u003e\u003ccenter\u003e\u003ca href=\"https://rewriting.csail.mit.edu/slides/rewriting-slides.pptx\" class=\"d-inline-block p-3 align-bottom\"\u003e\u003cimg height=\"78\" width=\"136\" src=\"images/slides-thumb.png\" style=\"border:1px solid\" data-nothumb=\"\"\u003e\u003cbr\u003ePresentation\u003cbr\u003eSlides\u003c/a\u003e\u003c/center\u003e\u003c/td\u003e\n\u003ctd\u003e\u003ccenter\u003e\u003ca href=\"https://rewriting.csail.mit.edu/\" class=\"d-inline-block p-3 align-top\"\u003e\u003cimg height=\"100\" width=\"78\" src=\"images/website-thumb.png\" style=\"border:1px solid\" data-nothumb=\"\"\u003e\u003cbr\u003eWebsite\u003c/a\u003e\u003c/center\u003e\u003c/td\u003e\n\u003ctd\u003e\u003ccenter\u003e\u003ca href=\"https://colab.research.google.com/github/davidbau/rewriting/blob/master/notebooks/rewriting-interface.ipynb\" class=\"d-inline-block p-3 align-bottom\"\u003e\u003cimg height=\"78\" width=\"136\" src=\"images/colab-thumb.png\" style=\"border:1px solid\" data-nothumb=\"\"\u003e\u003cbr\u003eDemo Colab\u003cbr\u003eNotebook\u003c/a\u003e\u003c/center\u003e\u003c/td\u003e\u003c/tr\u003e\u003c/table\u003e\n\n\u003cimg src='images/rewriting_teaser.gif' width=\"800px\" /\u003e\n\nOur method rewrites the weights of a generator to change generative rules.\nInstead of editing individual images, our method edits the generator, so an infinite set of images can be potentially synthesized and manipulated using the altered rules.  Rules can be changed in various ways, such as *removing* patterns like watermarks, *adding* objects such as people, or *replacing* definitions such as making trees grow out of towers.\n\n\u003cimg src='images/method.png' width=\"800px\" /\u003e\n\nOur method is based on the hypothesis that the weights of a generator act as linear associative memory. A layer stores a map between keys, which denote meaningful context, and values, which determine output.\n\n## Example Results\n\nThe results below show changes of a single rule within StyleGANv2.  In each case,\nfour examples chosen by the user (center of the top row) establish the context\nfor the rule begin rewritten, and the \"copy and paste\" examples (left and right of\ntop row) indicate how the user wishes to change the model.\n\nThe grid below shows pairs of outputs: for each pair, the first is the output of\nthe original unmodified StyleGANv2.  The second is the output of the modified\nStyleGANv2, applying the user's intention using our method.\n\n\u003chr\u003e\n\nFirst: changing the rule defining kids' eyebrows to make them look like a bushy mustache.\n\n\u003cimg src=\"images/example-eyebrows.png\" width=800\u003e\n\n\u003chr\u003e\n\nAltering the rule for pointy tower tops to make them into trees.\n\n\u003cimg src=\"images/example-tree-towers.png\" width=800\u003e\n\n\u003chr\u003e\n\nChanging the rule for tops of horses heads, to put hats on horses.\n\n\u003cimg src=\"images/example-horsehats.png\" width=800\u003e\n\n\u003chr\u003e\n\nChanging frowns into smiles.\n\n\u003cimg src=\"images/example-smiles.png\" width=800\u003e\n\n\u003chr\u003e\n\nRemoving the main window in a building by changing the rule to draw a blank wall.\n\n\u003cimg src=\"images/example-erasewindows.png\" width=800\u003e\n\n## Tips\n\nThe code runs using PyTorch.\n\n* The method and interface can be found in `/rewrite`\n* Notebooks are in `/notebooks`: see `rewriting-interface.ipynb` for the demonstration UI.\n* Quantitative experiments in `/metrics`, dissection utilities in `/utils`.\n  Experiments from section 5.1 can be repoduced by `/experiments.sh`,\n  and section 5.2 can be reproduced by `/watermarks.sh`.  The experiment\n  in section 5.3 runs in ithe notebook `/notebooks/reflection-rule-change.ipynb`.\n\n## Setup\n\nIt's designed to use a recent version of PyTorch (1.4+) on python (3.6), using\ncuda 10.1 and cudnn 7.6.0.  Run `setup/setup_renv.sh` to create a conda environment\nthat has the needed dependencies.\n\n## Rewriting your own models\n\nTo edit your own models, do the following:\n * Train your GAN using the standard StyleGAN v2 tensorflow release, found\n   at https://github.com/NVlabs/stylegan2\n * Convert your trained weights to pytorch using the `convert_weight.py` utility \n   by rosinality, found in https://github.com/rosinality/stylegan2-pytorch\n * Then those weight files can be directly loaded by our sequentialiized\n   StyleGAN port, as follows:\n```\n# Resolution (size) and style dimensionality (style_dim and n_mlp) are\n# the architecture dimensions as you trained them.  The truncation trick can be\n# applied here if desired (truncation=1.0 if not).\n# Note that mconv='seq' splits apart the optimized modulated convolution into\n# separate operations that the rewriter can examine the underlying\n# convolution directly.\nmodel = SeqStyleGAN2(size=256, style_dim=512, n_mlp=8, truncation=0.5, mconv='seq')\n\n# load the exponential moving average model weights, put it on the GPU.\nstate_dict = torch.load('your_model.pt')\nmodel.load_state_dict(state_dict['g_ema'], latent_avg=state_dict['latent_avg'])\nmodel.cuda()\n```\n * Create a `ganrewrite.SeqStyleGanRewriter` instance to edit your model\n```\nlayernum = 8 # or which ever layer you wish to modify\nsample_size = 1000 # a small sample of images for computing statistics\nzds = zdataset.z_dataset_for_model(model, size=sample_size)\ngw = SeqStyleGanRewriter(\n    model, zds, layernum,\n    cachedir='experiments')\n```\n * Finally, to specify and create changes, you can use the\n   `rewriteapp.GanRewriteApp` interface (assumes you are running\n   in a notebook.  This interface can be used to try edits\n   and save and load json files with stored model edit specifications.\n```\nsavedir = 'masks'\ninterface = rewriteapp.GanRewriteApp(gw, size=256, mask_dir=savedir, num_canvases=32)\n```\n * To bulk-generate images from an edited model (notebook UI not needed),\n   you can do the following, then sample the output of the modified model\n   as usual.  See `metrics/sample_edited.py` for an example.\n```\nsaved_edit = 'masks/my_edit.json'\ngw.apply_edit(json.load(saved_edit), rank=1)\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdavidbau%2Frewriting","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fdavidbau%2Frewriting","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdavidbau%2Frewriting/lists"}