{"id":19647080,"url":"https://github.com/idsia/fpainter","last_synced_at":"2026-02-14T12:02:24.032Z","repository":{"id":79448572,"uuid":"546048920","full_name":"IDSIA/fpainter","owner":"IDSIA","description":"Official repository for the paper \"Images as Weight Matrices: Sequential Image Generation Through Synaptic Learning Rules\" (ICLR 2023)","archived":false,"fork":false,"pushed_at":"2023-04-25T15:10:44.000Z","size":328,"stargazers_count":12,"open_issues_count":0,"forks_count":3,"subscribers_count":4,"default_branch":"main","last_synced_at":"2025-04-28T15:41:51.476Z","etag":null,"topics":["fast-weight-programmers","fast-weights","generative-adversarial-network","image-generation","neural-learning-rules","transformers"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"other","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/IDSIA.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2022-10-05T12:40:42.000Z","updated_at":"2023-06-15T15:57:33.000Z","dependencies_parsed_at":"2023-07-20T10:46:14.734Z","dependency_job_id":null,"html_url":"https://github.com/IDSIA/fpainter","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/IDSIA/fpainter","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/IDSIA%2Ffpainter","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/IDSIA%2Ffpainter/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/IDSIA%2Ffpainter/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/IDSIA%2Ffpainter/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/IDSIA","download_url":"https://codeload.github.com/IDSIA/fpainter/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/IDSIA%2Ffpainter/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":29443468,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-02-14T10:51:12.367Z","status":"ssl_error","status_checked_at":"2026-02-14T10:50:52.088Z","response_time":53,"last_error":"SSL_read: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["fast-weight-programmers","fast-weights","generative-adversarial-network","image-generation","neural-learning-rules","transformers"],"created_at":"2024-11-11T14:42:18.120Z","updated_at":"2026-02-14T12:02:24.015Z","avatar_url":"https://github.com/IDSIA.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Fast Weight Painters (FPAs)\n\nThis is the official repository containing code for the paper:\n\n[Images as Weight Matrices: Sequential Image Generation Through Synaptic Learning Rules (ICLR 2023)](https://arxiv.org/abs/2210.06184)\n\n### Illustrations\n\nIterative generation steps from left to right. *Top row: rank-1 update, bottom row: (partially) generated image*\n\n- Generation of 64x64 images through 16 rank-1 updates (MetFaces \u0026 AFHQ-Cat)\n\n\u003cimg src=\"./images/metfaces_16steps.jpg\" width=\"500px\"\u003e\u003c/img\u003e\n\n\u003cimg src=\"./images/afhq_cat_16steps.jpg\" width=\"500px\"\u003e\u003c/img\u003e\n\n- Generation of 64x64 images through 64 rank-1 updates (AFHQ-Wild \u0026 CelebA)\n\n\u003cimg src=\"./images/afhq_wild_full.jpg\" width=\"40000px\"\u003e\u003c/img\u003e\n\n\u003cimg src=\"./images/celeba_full.jpg\" width=\"40000px\"\u003e\u003c/img\u003e\n\n### Preliminary notes\n\nThis repository is originally forked from [lucidrains/lightweight-gan](https://github.com/lucidrains/lightweight-gan).\nIn addition to the implementation of our generators/FPAs, we also applied a few crucial modifications to consistently compute FID scores (see also Appendix B.4 of the paper):\n* The image format can be specified via `--image_save_format`. We store generated and resized real images in the same format to compute FIDs.\n* Following the standard practice, we use all real images to compute an FID.\nNB: the implementation of FID computation in the current code is suboptimal: the same statistics for the real images are recomputed at every FID computation. Instead, we should compute the stastistics only for the first time (or pre-compute them), store and reuse them. However, this would also require changes in the actual function that computes the FID (we use [mseitzer/pytorch-fid](https://github.com/mseitzer/pytorch-fid) that does not support pre-computed statistics, unlike [GaParmar/clean-fid](https://github.com/GaParmar/clean-fid)). In the end, we left this as is, but ideally this should be improved.\n\nWe use the U-Net implemetation from [lucidrains/denoising-diffusion-pytorch](https://github.com/lucidrains/denoising-diffusion-pytorch) by the same author. We include the corresponding licences in `LICENSE` and in the relevant files.\n\nWe also found it useful to look into the official \"Lightweight GAN\" implementation ([odegeasslbc/FastGAN-pytorch](https://github.com/odegeasslbc/FastGAN-pytorch)).\n\nFor the StyleGAN2 baselines reported in the paper, we used the official StyleGAN3 implementation ([NVlabs/stylegan3](https://github.com/NVlabs/stylegan3)).\n\nWe thank all these authors for making their code publicly available.\n\n## Requirements\nThe packages we used for our experiments can be found in the `requirements.txt` file.\nWe used python \u003e= 3.8 and PyTorch \u003e= 1.9.\n\n## Generating images from pre-trained models\nNB: there is no need to prepare/download any image data to generate images from a pre-trained model.\n\nPre-trained models can be found [here](https://drive.google.com/file/d/1mtD3wATtGJfkX6IjRPH-G1ZxOlbcpVax/view?usp=sharing).\nThe same folder contains example scripts for training and image generation from the corresponding model.\n\n## Training \u0026 evaluation/generation\n\n### Data\nTo train a model or to evaluate a model by computing its FID score, the data has to be downloaded in advance.\nThe code in this repository does not take care of downloading data: this has to be done manually.\nHere are some useful links:\n- AFHQ [clovaai/stargan-v2](https://github.com/clovaai/stargan-v2)\n- MetFaces [NVlabs/metfaces-dataset](https://github.com/NVlabs/metfaces-dataset)\n- LSUN [fyu/lsun](https://github.com/fyu/lsun)\n\n### Training\nWe provide example training scripts (with the pre-trained models) [here](https://drive.google.com/file/d/1mtD3wATtGJfkX6IjRPH-G1ZxOlbcpVax/view?usp=sharing).\nA typical example looks as follows:\n\n```\n## path to the data downloaded in advance\nDATA=\"/mypath/data/celeba/img_align_celeba\"\n\n## this is where resized real images will be stored\nREAL=\"/mypath/real_data/celeba_64\"\n\npython code/main.py \\\n  --num_train_steps 450_000 \\\n  --image_size 64 \\\n  --num_decoding_steps 64 \\\n  --data_dir ${DATA} \\\n  --fid_real_path ${REAL} \\\n  --batch_size 20 \\\n  --grad_cummulate_every 1 \\\n  --latent_dim 512 \\\n  --v2_input_sigmoid \\\n  --use_latent_to_init \\\n  --mini_latent_dim 8 \\\n  --mini_input_size 128 \\\n  --rnn_hidden_size 1024 \\\n  --num_layers 1 \\\n  --use_softmax \\\n  --out_tanh \\\n  --calculate_fid_every 5_000 \\\n  --model_type 'lstm_delta_split_v2' \\\n  --result_dir 'results' \\\n  --model_dir 'models' \\\n  --project_name 'just_test' \\\n  --use_wandb \n```\n\n`--use_wandb` activates training monitoring using Weights \u0026 Biases. Remove it to disable it.\n\nThe model architecture can be speficied via `--model_type`.\nThe models used in the paper have the following names in this code:\n* `base`: LightGAN baseline\n* `lstm_delta`: FPA with v1 input generator\n* `lstm_delta_unet`: U-Net extension of the model above\n* `lstm_delta_split_v2`: FPA with v2 input generator (the main FPA architecture)\n* `lstm_delta_split_v2_unet`: U-Net extension of the model above\n\nOther models that are not reported in the paper may not be well tested.\n\n### Evaluation\nTo evaluate a trained model, add the following flags to the training script (and remove `--use_wandb`):\n```\n  --eval_fid \\\n  --load_from 1 \\\n  --image_save_format \"jpg\" \\\n  --calculate_fid_num_images 50_000 \\\n```\n\n* `--load_from 1` evaluates the best checkpoint. If this is set to `0`, the latest checkpoint is evaluated.\n* `--image_save_format` should be adapted depending on the format in which the resized real images are stored (we used jpg for all datasets; again see Appendix B.4 for more details).\n\n### Generation\nSimilarly, to generate images from a trained model, add the following flags to the training script (and remove `--use_wandb`):\n```\n  --generate \\\n  --load_from 1 \\\n  --num_image_tiles 64 \\\n  --generate_types 'ema' \\\n```\n\n### Training U-Net models\nTo train an FPA/U-Net model using a pre-trained FPA checkpoint, add the following flags:\n```\n  --load_pre_trained path_to_my_fpa/models/default/model_1.pt \\\n  --frozen_base \\\n  --stop_grad_unet \\\n  --reset_g_optimizer \\\n  --unet_tanh \\\n  --ema_only_unet \\\n```\nand modify `--model_type` to specify the corresponding FPA/U-Net architecture.\n\n* `--load_pre_trained` has to be adapted to the path of a pre-trained FPA model.\n* `--unet_tanh` can be removed if the pre-trained FPA is trained without `--out_tanh` option.\n* The version above initializes the discriminator from the pre-trained parameters. `--load_only_g` and `--reset_d_optimizer` should be further added to train the discriminator from scratch. In general, we only observed minor differences between these two options in term of the final FID.\n\n## Links\n* Fast Weight Painters are Fast Weight Programmers (FWPs) applied to image generation. See also our previous works on FWPs: \n    * [Linear Transformers are Secretly Fast Weight Programmers (ICML 2021)](https://arxiv.org/abs/2102.11174)\n    * [Going Beyond Linear Transformers with Recurrent Fast Weight Programmers (NeurIPS 2021)](https://arxiv.org/abs/2106.06295)\n    * [A Modern Self-Referential Weight Matrix That Learns to Modify Itself (ICML 2022)](https://arxiv.org/abs/2202.05780)\n    * [Neural Differential Equations for Learning to Program Neural Nets Through Continuous Learning Rules (NeurIPS 2022)](https://arxiv.org/abs/2206.01649)\n* [Jürgen Schmidhuber's AI blog post on Fast Weight Programmers (March 26, 2021)](https://people.idsia.ch/~juergen/fast-weight-programmer-1991-transformer.html).\n\n## BibTex\n```\n@inproceedings{irie2023image,\n  title={Images as Weight Matrices: Sequential Image Generation Through Synaptic Learning Rules},\n  author={Kazuki Irie and J{\\\"u}rgen Schmidhuber},\n  booktitle={Int. Conf. on Learning Representations (ICLR)},\n  address = {Kigali, Rwanda}, \n  month = may,\n  year={2023}\n}\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fidsia%2Ffpainter","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fidsia%2Ffpainter","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fidsia%2Ffpainter/lists"}