{"id":13563601,"url":"https://github.com/kpandey008/DiffuseVAE","last_synced_at":"2025-04-03T20:31:21.914Z","repository":{"id":38126900,"uuid":"391273014","full_name":"kpandey008/DiffuseVAE","owner":"kpandey008","description":"Official implementation of \"DiffuseVAE: Efficient, Controllable and High-Fidelity Generation from Low-Dimensional Latents\"","archived":false,"fork":false,"pushed_at":"2022-09-10T04:22:30.000Z","size":10790,"stargazers_count":332,"open_issues_count":6,"forks_count":30,"subscribers_count":7,"default_branch":"main","last_synced_at":"2024-08-01T13:29:34.958Z","etag":null,"topics":["controllable-generation","ddpm","diffusion-models","generative-model","latent-variable-models","vae"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/kpandey008.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2021-07-31T06:22:18.000Z","updated_at":"2024-07-17T19:00:34.000Z","dependencies_parsed_at":"2022-07-08T01:41:20.869Z","dependency_job_id":null,"html_url":"https://github.com/kpandey008/DiffuseVAE","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/kpandey008%2FDiffuseVAE","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/kpandey008%2FDiffuseVAE/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/kpandey008%2FDiffuseVAE/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/kpandey008%2FDiffuseVAE/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/kpandey008","download_url":"https://codeload.github.com/kpandey008/DiffuseVAE/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":223030658,"owners_count":17076470,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["controllable-generation","ddpm","diffusion-models","generative-model","latent-variable-models","vae"],"created_at":"2024-08-01T13:01:21.225Z","updated_at":"2024-11-04T16:31:03.992Z","avatar_url":"https://github.com/kpandey008.png","language":"Python","funding_links":[],"categories":["Python"],"sub_categories":[],"readme":"# DiffuseVAE: Efficient, Controllable and High-Fidelity Generation from Low-Dimensional Latents\n\nThis repo contains the official implementation of the paper: [DiffuseVAE: Efficient, Controllable and High-Fidelity Generation from Low-Dimensional Latents](https://arxiv.org/abs/2201.00308) by [Kushagra Pandey](https://kpandey008.github.io/), [Avideep Mukherjee](https://www.cse.iitk.ac.in/users/avideep/), [Piyush Rai](https://www.cse.iitk.ac.in/users/piyush/), [Abhishek Kumar](http://www.abhishek.umiacs.io/)\n\n---\n## Overview\n\n DiffuseVAE is a novel generative framework that integrates a standard VAE within a diffusion model by conditioning the diffusion model samples on the VAE generated reconstructions. The resulting model can significantly improve upon the blurry samples generated from a standard VAE while at the same time equipping diffusion models with the low-dimensional VAE inferred latent code which can be used for downstream tasks like controllable synthesis and image attribute manipulation. In short, DiffuseVAE presents a generative model which combines the benefits of both VAEs and Diffusion models.\n\n![architecture!](./assets/diffusevae_tmlr-methodology.png)\n\nOur core contributions are as follows:\n\n1. We propose a generic DiffuseVAE conditioning framework and show that our framework can be reduced to a simple *generator-refiner* framework in which blurry samples generated from a VAE are refined using a conditional DDPM formulation.\n\n1. **Controllable synthesis** from a low-dimensional latent using diffusion models.\n\n1. **Better speed vs quality tradeoffs**: We show that DiffuseVAE inherently provides a better speed vs quality tradeoff as compared to standard DDPM/DDIM models on several image benchmarks\n\n1. **State-of-the-art synthesis**:  We show that DiffuseVAE exhibits synthesis quality comparable to recent state-of-the-art on standard image synthesis benchmarks like CIFAR-10, CelebA-64 and CelebA-HQ while maintaining access to a low-dimensional latent code representation.\n\n1. **Generalization to noisy conditioning signals**: We show that a pre-trained DiffuseVAE model exhibits generalization to different noise types in the DDPM conditioning signal exhibiting the effectiveness of our conditioning framework.\n\n![High res samples!](./assets/diffusevae_tmlr-main.png)\n\n---\n\n## Code overview\n\nThis repo uses [PyTorch Lightning](https://www.pytorchlightning.ai/) for training and [Hydra](https://hydra.cc/docs/intro/) for config management so basic familiarity with both these tools is expected. Please clone the repo with `DiffuseVAE` as the working directory for any downstream tasks like setting up the dependencies, training and inference.\n\n## Setting up the dependencies\n\nWe use `pipenv` for a project-level dependency management. Simply [install](https://pipenv.pypa.io/en/latest/#install-pipenv-today) `pipenv` and run the following command:\n\n```\npipenv install\n```\n\n## Config Management\nWe manage `train` and `test` configurations separately for each benchmark/dataset used in this work. All configs are present in the `main/configs` directory. This directory has subfolders named according to the dataset. Each dataset subfolder contains the training and evaluation configs as `train.yaml` and `test.yaml`. \n\n**Note**: The configuration files consists of many command line options. The meaning of each of these options is explained in the config for CIFAR-10.\n\n## Training\nPlease refer to the scripts provided in the table corresponding to some training tasks possible using the code.\n\n|          **Task**          \t|      **Reference**      \t|\n|:--------------------------:\t|:-----------------------:\t|\n|  Training First stage VAE  \t|  `scripts/train_ae.sh`  \t|\n| Training Second stage DDPM \t| `scripts/train_ddpm.sh` \t|\n\n## Inference\n\nPlease refer to the scripts provided in the table corresponding to some inference tasks possible using the code.\n\n|                          **Task**                         \t|         **Reference**         \t|\n|:---------------------------------------------------------:\t|:-----------------------------:\t|\n|            Sample/Reconstruct from Baseline VAE           \t|      `scripts/test_ae.sh`     \t|\n|                   Sample from DiffuseVAE                  \t|     `scripts/test_ddpm.sh`    \t|\n|          Generate reconstructions from DiffuseVAE         \t| `scripts/test_recons_ddpm.sh` \t|\n| Interpolate in the VAE/DDPM latent space using DiffuseVAE \t|    `scripts/interpolate.sh`   \t|\n\nFor computing the evaluation metrics (FID, IS etc.), we use the [torch-fidelity](https://github.com/toshas/torch-fidelity) package. See `scripts/fid.sh` for some sample usage examples.\n\n\n## Pretrained checkpoints\nAll pretrained checkpoints have been organized by dataset and can be accessed [here](https://drive.google.com/drive/folders/1GzIh75NnpgPa4A1hSb_viPowuaSHnL7R?usp=sharing).\n\n## Citing\nTo cite DiffuseVAE please use the following BibTEX entries:\n\n```\n@misc{pandey2022diffusevae,\n      title={DiffuseVAE: Efficient, Controllable and High-Fidelity Generation from Low-Dimensional Latents}, \n      author={Kushagra Pandey and Avideep Mukherjee and Piyush Rai and Abhishek Kumar},\n      year={2022},\n      eprint={2201.00308},\n      archivePrefix={arXiv},\n      primaryClass={cs.LG}\n}\n```\n```\n@inproceedings{\npandey2021vaes,\ntitle={{VAE}s meet Diffusion Models: Efficient and High-Fidelity Generation},\nauthor={Kushagra Pandey and Avideep Mukherjee and Piyush Rai and Abhishek Kumar},\nbooktitle={NeurIPS 2021 Workshop on Deep Generative Models and Downstream Applications},\nyear={2021},\nurl={https://openreview.net/forum?id=-J8dM4ed_92}\n}\n```\n\nSince our model uses diffusion models please consider citing the original [Diffusion model](https://arxiv.org/abs/1503.03585), [DDPM](https://arxiv.org/abs/2006.11239) and [VAE](https://arxiv.org/abs/1312.6114) papers.\n\n## Contact\nKushagra Pandey (@kpandey008)\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fkpandey008%2FDiffuseVAE","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fkpandey008%2FDiffuseVAE","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fkpandey008%2FDiffuseVAE/lists"}