{"id":16510774,"url":"https://github.com/janosh/torch-mnf","last_synced_at":"2025-07-08T08:35:33.712Z","repository":{"id":42466526,"uuid":"239540780","full_name":"janosh/torch-mnf","owner":"janosh","description":"Multiplicative Normalizing Flows in PyTorch.","archived":false,"fork":false,"pushed_at":"2025-07-07T16:31:17.000Z","size":1443,"stargazers_count":24,"open_issues_count":1,"forks_count":2,"subscribers_count":6,"default_branch":"main","last_synced_at":"2025-07-07T17:51:17.383Z","etag":null,"topics":["generative-modeling","normalizing-flows","pytorch"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/janosh.png","metadata":{"files":{"readme":"readme.md","changelog":null,"contributing":null,"funding":null,"license":"license","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2020-02-10T15:10:10.000Z","updated_at":"2025-05-18T19:21:20.000Z","dependencies_parsed_at":"2024-10-27T11:09:07.800Z","dependency_job_id":"ea7d6783-3b2f-4d18-9b98-1c21e16b8602","html_url":"https://github.com/janosh/torch-mnf","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/janosh/torch-mnf","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/janosh%2Ftorch-mnf","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/janosh%2Ftorch-mnf/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/janosh%2Ftorch-mnf/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/janosh%2Ftorch-mnf/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/janosh","download_url":"https://codeload.github.com/janosh/torch-mnf/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/janosh%2Ftorch-mnf/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":264232338,"owners_count":23576820,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["generative-modeling","normalizing-flows","pytorch"],"created_at":"2024-10-11T15:57:13.805Z","updated_at":"2025-07-08T08:35:33.686Z","avatar_url":"https://github.com/janosh.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Torch MNF\n\n[![Tests](https://github.com/janosh/torch-mnf/actions/workflows/test.yml/badge.svg)](https://github.com/janosh/torch-mnf/actions/workflows/test.yml)\n[![pre-commit.ci status](https://results.pre-commit.ci/badge/github/janosh/torch-mnf/master.svg)](https://results.pre-commit.ci/latest/github/janosh/torch-mnf/master)\n![GitHub Repo Size](https://img.shields.io/github/repo-size/janosh/torch-mnf?label=Repo+Size)\n\nPyTorch implementation of Multiplicative Normalizing Flows [[1]](#mnf-bnn).\n\nWith flow implementations courtesy of [Andrej Karpathy](https://github.com/karpathy/pytorch-normalizing-flows).\n\n## Files of Interest\n\nNew here? Check out the example notebooks:\n\n- [`examples/half_moons.ipynb`](examples/half_moons.ipynb)\n- [`examples/mnf_mnist.ipynb`](examples/mnf_mnist.ipynb)\n\nInterested in the implementation? See\n\n- [`models/mnf_lenet.py`](torch_mnf/models/mnf_lenet.py)\n- [`flows/*.py`](torch_mnf/flows)\n- [`layers/*.py`](torch_mnf/layers)\n\n## MNF Results\n\n### MNIST\n\nRotating an MNIST 9 by 180° in steps of 20°, the MNF LeNet (left) does not produce overconfident predictions on out-of-sample data unlike the regular LeNet (right), indicating it captures its own uncertainty well. The violin distributions in the top plot were generated by the MNF LeNet predicting each image 500 times. The predictions run in parallel so this is fast. Both models trained for 3 epochs on MNIST with Adam. The MNF model has 696,950 trainable parameters, the regular LeNet 258,582.\n\n| MNF Lenet                                                  | Regular LeNet                                                 |\n| ---------------------------------------------------------- | ------------------------------------------------------------- |\n| ![RNVP Point Flow](assets/mnf/mnist/rot-9-mnf-lenet-s.png) | ![RNVP x to 2 and z to x](assets/mnf/mnist/rot-9-lenet-s.png) |\n\n## Flow Results\n\n### Real Non-Volume Preserving Flows\n\nFlow: `[RNVP, RNVP, RNVP, RNVP, RNVP, RNVP, RNVP, RNVP, RNVP]`\n\nFinal loss: 0.47\n\n| Trained for 1400 steps with Adam (`lr=1e-4, wd=1e-5`) | Parameters: 22,914                                       |\n| ----------------------------------------------------- | -------------------------------------------------------- |\n| ![RNVP Point Flow](assets/rnvp/moons/point-flow.png)  | ![RNVP x to 2 and z to x](assets/rnvp/moons/z2x+x2z.png) |\n\n### Masked Autoregressive Flow\n\nFlow: `[MAF, MAF, MAF, MAF, MAF, MAF, MAF, MAF, MAF]`\n\nFinal loss: 36.21\n\n| Trained for 1400 steps with Adam (`lr=1e-4, wd=1e-5`) | Parameters: 12,348                                     |\n| ----------------------------------------------------- | ------------------------------------------------------ |\n| ![MAF Point Flow](assets/maf/moons/point-flow.png)    | ![MAF x to 2 and z to x](assets/maf/moons/z2x+x2z.png) |\n\n### Neural Spline Flow Autoregressive Layer\n\nFlow: `[ActNormFlow, Glow, NSF_AR, ActNormFlow, Glow, NSF_AR, ActNormFlow, Glow, NSF_AR]`\n\nFinal loss: 19.13\n\n| Trained for 1400 steps with Adam (`lr=1e-4, wd=1e-5`)    | Parameters: 3,012                                            |\n| -------------------------------------------------------- | ------------------------------------------------------------ |\n| ![NSF-AR Point Flow](assets/nsf_ar/moons/point-flow.png) | ![NSF-AR x to 2 and z to x](assets/nsf_ar/moons/z2x+x2z.png) |\n\n### Neural Spline Flow Coupling Layer\n\nFlow: `[ActNormFlow, Glow, NSF_CL, ActNormFlow, Glow, NSF_CL, ActNormFlow, Glow, NSF_CL]`\n\nFinal loss: 6.06\n\n| Trained for 1400 steps with Adam (`lr=1e-4, wd=1e-5`)    | Parameters: 5,844                                            |\n| -------------------------------------------------------- | ------------------------------------------------------------ |\n| ![NSF-CL Point Flow](assets/nsf_cl/moons/point-flow.png) | ![NSF-CL x to 2 and z to x](assets/nsf_cl/moons/z2x+x2z.png) |\n\n## References\n\n1. \u003ca id=\"mnf-bnn\"\u003e\u003c/a\u003e **MNF**: _Multiplicative Normalizing Flows for Variational Bayesian Neural Networks_ | Christos Louizos, Max Welling (Mar 2017) | [1703.01961](https://arxiv.org/abs/1703.01961)\n\n2. \u003ca id=\"vi-nf\"\u003e\u003c/a\u003e **VI-NF**: _Variational Inference with Normalizing Flows_ | Danilo Rezende, Shakir Mohamed (May 2015) | [1505.05770](https://arxiv.org/abs/1505.05770)\n\n3. \u003ca id=\"made\"\u003e\u003c/a\u003e **MADE**: _Masked Autoencoder for Distribution Estimation_ | Mathieu Germain, Karol Gregor, Iain Murray, Hugo Larochelle (Jun 2015) | [1502.03509](https://arxiv.org/abs/1502.03509)\n\n4. \u003ca id=\"nice\"\u003e\u003c/a\u003e **NICE**: _Non-linear Independent Components Estimation_ | Laurent Dinh, David Krueger, Yoshua Bengio (Oct 2014) | [1410.8516](https://arxiv.org/abs/1410.8516)\n\n5. \u003ca id=\"rnvp\"\u003e\u003c/a\u003e **RNVP**: _Density estimation using Real NVP_ | Laurent Dinh, Jascha Sohl-Dickstein, Samy Bengio (May 2016) | [1605.08803](https://arxiv.org/abs/1605.08803)\n\n6. \u003ca id=\"maf\"\u003e\u003c/a\u003e **MAF**: _Masked Autoregressive Flow for Density Estimation_ | George Papamakarios, Theo Pavlakou, Iain Murray (Jun 2018) | [1705.07057](https://arxiv.org/abs/1705.07057)\n\n7. \u003ca id=\"iaf\"\u003e\u003c/a\u003e **IAF**: _Improving Variational Inference with Inverse Autoregressive Flow_ | Diederik Kingma et al. (Jun 2016) | [1606.04934](https://arxiv.org/abs/1606.04934)\n\n8. \u003ca id=\"nsf\"\u003e\u003c/a\u003e **NSF**: _Neural Spline Flows_ | Conor Durkan, Artur Bekasov, Iain Murray, George Papamakarios (Jun 2019) | [1906.04032](https://arxiv.org/abs/1906.04032)\n\n## Debugging Tips\n\nA great method of checking for infinite or `NaN` gradients is\n\n```py\nfor name, param in model.named_parameters():\n    print(name, torch.isfinite(param.grad).all())\n    print(name, torch.isnan(param.grad).any())\n```\n\nThere's also [`torch.autograd.detect_anomaly()`](https://pytorch.org/docs/stable/autograd.html#torch.autograd.detect_anomaly) used as context manager:\n\n```py\nwith torch.autograd.detect_anomaly():\n    x = torch.rand(10, 10, requires_grad=True)\n    out = model(x)\n    out.backward()\n```\n\nand [`torch.autograd.set_detect_anomaly(True)`](https://pytorch.org/docs/stable/autograd.html#torch.autograd.set_detect_anomaly). See [here](https://discuss.pytorch.org/t/87594) for an issue that used these tools.\n\n## Requirements\n\n[`requirements.txt`](requirements.txt) created with [`pipreqs .`](https://github.com/bndr/pipreqs). Find new dependencies manually with `pipreqs --diff requirements.txt`.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjanosh%2Ftorch-mnf","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fjanosh%2Ftorch-mnf","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjanosh%2Ftorch-mnf/lists"}