{"id":13476857,"url":"https://github.com/LukasRinder/normalizing-flows","last_synced_at":"2025-03-27T04:31:26.366Z","repository":{"id":37636142,"uuid":"237158627","full_name":"LukasRinder/normalizing-flows","owner":"LukasRinder","description":"Implementation of normalizing flows in TensorFlow 2 including a small tutorial.","archived":false,"fork":false,"pushed_at":"2024-08-30T06:12:31.000Z","size":6765,"stargazers_count":141,"open_issues_count":1,"forks_count":37,"subscribers_count":7,"default_branch":"master","last_synced_at":"2024-10-30T09:35:43.646Z","etag":null,"topics":["deep-learning","density-estimation","generative-models","normalizing-flows"],"latest_commit_sha":null,"homepage":null,"language":"Jupyter Notebook","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"gpl-3.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/LukasRinder.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2020-01-30T07:10:10.000Z","updated_at":"2024-08-30T06:12:29.000Z","dependencies_parsed_at":"2024-10-30T09:41:01.439Z","dependency_job_id":null,"html_url":"https://github.com/LukasRinder/normalizing-flows","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/LukasRinder%2Fnormalizing-flows","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/LukasRinder%2Fnormalizing-flows/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/LukasRinder%2Fnormalizing-flows/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/LukasRinder%2Fnormalizing-flows/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/LukasRinder","download_url":"https://codeload.github.com/LukasRinder/normalizing-flows/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":245784903,"owners_count":20671621,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["deep-learning","density-estimation","generative-models","normalizing-flows"],"created_at":"2024-07-31T16:01:35.414Z","updated_at":"2025-03-27T04:31:23.985Z","avatar_url":"https://github.com/LukasRinder.png","language":"Jupyter Notebook","funding_links":[],"categories":["Jupyter Notebook","🧑‍💻 Repos \u003csmall\u003e(18)\u003c/small\u003e"],"sub_categories":["\u003cimg src=\"assets/tensorflow.svg\" alt=\"TensorFlow\" height=\"20px\"\u003e \u0026nbsp;TensorFlow Repos"],"readme":"# Normalizing Flows\n\nIn this project, we implemented various normalizing flows in Tensorflow 2.0 and tested them on different datasets. \nCurrently implemented flows are:\n \n- Planar Flow [1]\n- Radial Flow [1]\n- Real NVP [2]\n- Masked Autoregressive Flow (MAF) [3]\n- Inverse Autoregressive Flow (IAF) [4]\n- Neural Spline Flow [5] \n\nThe implementations are located in [normalizingflows/flow_catalog.py](/normalizingflows/flow_catalog.py). \nFor MAF, the Tensorflow implementation is used [6]. \nThe MADE [7] network for modelling the shift and scale parameters can also be loaded from the flow catalog. \nNote that IAF can be obtained with the Tensorflow bijector inverse function, as described in the Tensorflow documentation [6].\nAlso important to note is that planar and radial flow do not admit an algebraic inverse.\n\n\nThe normalizing flows can be tested in terms of estimating the density on various datasets.\nIf an algebraic inverse is available, the flows can also be used as flow-based generative model.\n[data/toy_data.py](/data/toy_data.py) contains various 2D toy data distributions on which the flows can be tested.\nAdditionally, data loading functions for three datasets from the UCI machine learning repository [8] (GAS, POWER, MINIBOONE),\nMNIST, and CelebA are implemented in [data/dataset_loader.py](/data/dataset_loader.py). \nDetails about how to download the data can be found in the respective files. \nThe Dataset class in [data/data_manager.py](/data/data_manager.py) can be used as general data loader.\n\n[example_training.ipynb](/example_training.ipynb) describes how to use normalizing flows to estimate the probability density function in a small tutorial.\nAs example Masked Autoregressive Flow is used to estimate the density of a 2D toy data distribution. \n\nThe experiments folder contains experiments that we performed, including example heatmaps of approximated distributions and generated images from training on MNIST and CelebA.\nBelow we show some examples.\n\n- Density estimation on toy data (left: ground truth, right: estimated density with MAF:\n\n\u003cimg src=\"visualizations/ground_truth_toy_data/tum_1000.png\" width=\"200\" height=\"200\" /\u003e \u003cimg src=\"/experiments/maf/example_results/toy_data/tum_density_800_64_[128, 128].png\" width=\"250\" height=\"200\" /\u003e\n\n\n- Visualization how samples from a Gaussian base distribution are transformed through 8 layers of Real NVP to match the target distribution:\n\n\u003cimg src=\"visualizations/gif_moons.gif\" width=\"200\" height=\"200\" /\u003e\n\n- Generated images from training on the classes \"2\", \"4\", \"7\", and \"8\" of the MNIST dataset (from left to right: ground truth, Real NVP, MAF)\n\n\u003cimg src=\"visualizations/mnist_realnvp_maf.png\" width=\"350\" height=\"200\" /\u003e\n\n\n- Generated images from training on the CelebA dataset (from left to right: ground truth, Real NVP, MAF)\n\n\u003cimg src=\"visualizations/celeba_all.png\" width=\"350\" height=\"200\" /\u003e\n\n\nAs next steps, it would by interesting to implement more flows such as Neural Autoregressive Flows [9], Glow [10], and FFJORD [11].\n\nThis work was done during the large-scale machine learning practical course at the TUM Data Analytics and Machine Learning Group.\nIn case of any questions, feel free to reach out to us.\n\nMurat Can Karacabey, muratcan.karacabey@tum.de\n\nMarkus Kittel, markus.kittel@tum.de\n\nLukas Rinder, lukas.rinder@tum.de\n\n\n### References\n\n[1] D. J. Rezende and S. Mohamed, “Variational inference with normalizing flows,” 32nd Int. Conf. Mach. Learn. ICML 2015, vol. 2, pp. 1530–1538, 2015.\n\n[2] L. Dinh, J. Sohl-Dickstein, and S. Bengio, “Density estimation using Real NVP,” 2016.\n\n[3] G. Papamakarios, T. Pavlakou, and I. Murray, “Masked autoregressive flow for density estimation,” Adv. Neural Inf. Process. Syst., vol. 2017-Decem, pp. 2339–2348, 2017.\n\n[4] D. P. Kingma, T. Salimans, R. Jozefowicz, X. Chen, I. Sutskever, and M. Welling, “Improved variational inference with inverse autoregressive flow,” Adv. Neural Inf. Process. Syst., no. Nips, pp. 4743–4751, 2016.\n\n[5] C. Durkan, A. Bekasov, I. Murray, and G. Papamakarios, “Neural Spline Flows,” no. NeurIPS, 2019.\n\n[6] TensorFlow, “TensorFlow Documentation Masked Autoregressive Flow,” 2020. [Online]. Available: https://www.tensorflow.org/probability/api_docs/python/tfp/bijectors/MaskedAutoregressiveFlow. [Accessed: 28-Feb-2020].\n\n[7] M. Germain, K. Gregor, I. Murray, and H. Larochelle, “MADE: Masked autoencoder for distribution estimation,” 32nd Int. Conf. Mach. Learn. ICML 2015, vol. 2, pp. 881–889, 2015.\n\n[8] D. Dua and E. K. Taniskidou, “UCI: Machine Learning Repository.” [Online]. Available: http://archive.ics.uci.edu/ml/index.php. [Accessed: 29-Jan-2020].\n\n[9] C. Huang, D. Krueger, A. Lacoste, and A. Courville, “Neural Autoregressive Flows,” arXiv Prepr., no. Icml, 2018.\n\n[10] D. P. Kingma and P. Dhariwal, “Glow: Generative Flow with Invertible 1x1 Convolutions,” arXiv Prepr., 2018.\n\n[11] W. Grathwohl, R. T. Q. Chen, J. Bettencourt, I. Sutskever, and D. Duvenaud, “FFJORD: Free-form Continuous Dynamics for Scalable Reversible Generative Models,” arXiv Prepr., 2018.\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FLukasRinder%2Fnormalizing-flows","html_url":"https://awesome.ecosyste.ms/projects/github.com%2FLukasRinder%2Fnormalizing-flows","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FLukasRinder%2Fnormalizing-flows/lists"}