{"id":13784234,"url":"https://github.com/mlvlab/MonotoneFlows","last_synced_at":"2025-05-11T19:32:34.837Z","repository":{"id":141431264,"uuid":"571396684","full_name":"mlvlab/MonotoneFlows","owner":"mlvlab","description":"Official PyTorch implementation of NeurIPS 2022 paper \"Invertible Monotone Operators for Normalizing Flows\"","archived":false,"fork":false,"pushed_at":"2022-11-28T03:06:44.000Z","size":70,"stargazers_count":12,"open_issues_count":0,"forks_count":1,"subscribers_count":3,"default_branch":"main","last_synced_at":"2024-08-03T19:08:40.763Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/mlvlab.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null}},"created_at":"2022-11-28T03:04:53.000Z","updated_at":"2024-07-19T11:50:58.000Z","dependencies_parsed_at":"2024-01-17T02:15:51.191Z","dependency_job_id":"5177201e-25e2-4bde-8449-a216d3c523f6","html_url":"https://github.com/mlvlab/MonotoneFlows","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mlvlab%2FMonotoneFlows","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mlvlab%2FMonotoneFlows/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mlvlab%2FMonotoneFlows/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mlvlab%2FMonotoneFlows/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/mlvlab","download_url":"https://codeload.github.com/mlvlab/MonotoneFlows/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":225086576,"owners_count":17418753,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-08-03T19:00:37.959Z","updated_at":"2024-11-17T20:31:40.165Z","avatar_url":"https://github.com/mlvlab.png","language":"Python","funding_links":[],"categories":["📝 Publications \u003csmall\u003e(60)\u003c/small\u003e"],"sub_categories":[],"readme":"# Invertible Monotone Operators for Normalizing Flows\n\nCode for [Monotone Flows](https://arxiv.org/abs/2210.08176).\n\nThis work is inspired by [Invertible Residual Networks](https://arxiv.org/abs/1811.00995), [Residual Flows](https://arxiv.org/abs/1906.02735), [Implicit Normalizing Flows](https://arxiv.org/abs/2103.09527), and [i-DenseNets](https://arxiv.org/abs/2102.02694). The source is largely adapted from [i-DenseNets](https://github.com/yperugachidiaz/invertible_densenets) with fixed-point routines adapted from [Implicit Normalizing Flows](https://github.com/thu-ml/implicit-normalizing-flows/).\n\nA BibTeX entry for LaTeX users:\n```\n@misc{ahn2022invertible,\n      title={Invertible Monotone Operators for Normalizing Flows}, \n      author={Byeongkeun Ahn and Chiyoon Kim and Youngjoon Hong and Hyunwoo J. Kim},\n      year={2022},\n      booktitle={NeurIPS},\n      eprint={2210.08176}\n}\n```\n\n\n## Download datasets\n* CIFAR10 is automatically downloaded. \n* The pre-processing steps and downloading of ImageNet32 are described in [Residual Flows](https://github.com/rtqichen/residual-flows).\n\n\n## Setting up the environment\n1. Create a new conda environment.\n2. Install packages with the following command:\n```\nconda install pytorch=1.11.0 cudnn cudatoolkit=11.3 torchvision numpy pandas scikit-learn tqdm yacs matplotlib tabulate colorama ninja scipy termcolor -c pytorch -c nvidia -c conda-forge\n```\n3. Run the commands below.\n\n\n## 1D toy experiments\n\n* Experiment (a)\n```\nCUDA_VISIBLE_DEVICES=0 python train_1dexample.py --data multiplestepfunc --nblocks 2 --arch iresnet --save experiments/toy-1d/multiplestepfunc_rf_woLS --atol 1e-4 --rtol 1e-4 --densenet True --learnable_concat True --start_learnable_concat 0 --act CReLU --densenet_depth 4 --densenet_growth 128 --monotone_resolvent False --brute-force True --lr 0.01 --coeff 0.99 --lip_coeff 0.99 --batch_size 5000 --niters 15000 --weight-decay 0 --actnorm False\n```\n\n* Experiment (b)\n```\nCUDA_VISIBLE_DEVICES=0 python train_1dexample.py --data multiplestepfunc --nblocks 2 --arch iresnet --save experiments/toy-1d/multiplestepfunc_rf_wLS --atol 1e-4 --rtol 1e-4 --densenet True --learnable_concat True --start_learnable_concat 0 --act CReLU --densenet_depth 4 --densenet_growth 128 --monotone_resolvent False --brute-force True --lr 0.01 --coeff 0.99 --lip_coeff 0.99 --batch_size 5000 --niters 15000 --weight-decay 0 --actnorm True\n```\n\n* Experiment (c)\n```\nCUDA_VISIBLE_DEVICES=0 python train_1dexample.py --data multiplestepfunc --nblocks 1 --arch impflow --save experiments/toy-1d/multiplestepfunc_if_wLS --atol 1e-4 --rtol 1e-4 --densenet True --learnable_concat True --start_learnable_concat 0 --act CReLU --densenet_depth 4 --densenet_growth 128 --monotone_resolvent False --brute-force True --lr 0.01 --coeff 0.99 --lip_coeff 0.99 --batch_size 5000 --niters 15000 --weight-decay 0 --actnorm True\n```\n\n* Experiment (d)\n```\nCUDA_VISIBLE_DEVICES=0 python train_1dexample.py --data multiplestepfunc --nblocks 2 --save experiments/toy-1d/multiplestepfunc_mf_wLS --atol 1e-4 --rtol 1e-4 --densenet True --learnable_concat True --start_learnable_concat 0 --act CReLU --densenet_depth 4 --densenet_growth 128 --monotone_resolvent True --brute-force True --lr 0.01 --coeff 0.99 --lip_coeff 0.99 --batch_size 5000 --niters 15000 --weight-decay 0 --actnorm True\n```\n\n\n## 2D toy experiments\n\n* 2 Spirals, i-DenseNets\n```\npython train_toy.py --data 2spirals --nblocks 10 --save experiments/toy-2d/2spirals_idensenet --densenet True --learnable_concat True --start_learnable_concat 25000 --act CLipSwish --densenet_depth 3 --densenet_growth 16 --monotone_resolvent False --brute-force True\n```\n\n* 2 Spirals, Monotone Flows\n```\npython train_toy.py --data 2spirals --nblocks 10 --save experiments/toy-2d/2spirals_monotoneflow --densenet True --learnable_concat True --start_learnable_concat 25000 --act CPila --densenet_depth 3 --densenet_growth 16 --monotone_resolvent True --brute-force True\n```\n\n* 8 Gaussians, i-DenseNets\n```\npython train_toy.py --data 8gaussians --nblocks 10 --save experiments/toy-2d/8gaussians_idensenet --densenet True --learnable_concat True --start_learnable_concat 25000 --act CLipSwish --densenet_depth 3 --densenet_growth 16 --monotone_resolvent False --brute-force True\n```\n\n* 8 Gaussians, Monotone Flows\n```\npython train_toy.py --data 8gaussians --nblocks 10 --save experiments/toy-2d/8gaussians_monotoneflow --densenet True --learnable_concat True --start_learnable_concat 25000 --act CPila --densenet_depth 3 --densenet_growth 16 --monotone_resolvent True --brute-force True\n```\n\n* Checkerboard, i-DenseNets\n```\npython train_toy.py --data checkerboard --nblocks 10 --save experiments/toy-2d/checkerboard_idensenet --densenet True --learnable_concat True --start_learnable_concat 25000 --act CLipSwish --densenet_depth 3 --densenet_growth 16 --monotone_resolvent False --brute-force True\n```\n\n* Checkerboard, Monotone Flows\n```\npython train_toy.py --data checkerboard --nblocks 10 --save experiments/toy-2d/checkerboard_monotoneflow --densenet True --learnable_concat True --start_learnable_concat 25000 --act CPila --densenet_depth 3 --densenet_growth 16 --monotone_resolvent True --brute-force True\n```\n\n* Circles, i-DenseNets\n```\npython train_toy.py --data circles --nblocks 10 --save experiments/toy-2d/circles_idensenet --densenet True --learnable_concat True --start_learnable_concat 25000 --act CLipSwish --densenet_depth 3 --densenet_growth 16 --monotone_resolvent False --brute-force True\n```\n\n* Circles, Monotone Flows\n```\npython train_toy.py --data circles --nblocks 10 --save experiments/toy-2d/circles_monotoneflow --densenet True --learnable_concat True --start_learnable_concat 25000 --act CPila --densenet_depth 3 --densenet_growth 16 --monotone_resolvent True --brute-force True\n```\n\n* Moons, i-DenseNets\n```\npython train_toy.py --data moons --nblocks 10 --save experiments/toy-2d/moons_idensenet --densenet True --learnable_concat True --start_learnable_concat 25000 --act CLipSwish --densenet_depth 3 --densenet_growth 16 --monotone_resolvent False --brute-force True\n```\n\n* Moons, Monotone Flows\n```\npython train_toy.py --data moons --nblocks 10 --save experiments/toy-2d/moons_monotoneflow --densenet True --learnable_concat True --start_learnable_concat 25000 --act CPila --densenet_depth 3 --densenet_growth 16 --monotone_resolvent True --brute-force True\n```\n\n* Pinwheel, i-DenseNets\n```\npython train_toy.py --data pinwheel --nblocks 10 --save experiments/toy-2d/pinwheel_idensenet --densenet True --learnable_concat True --start_learnable_concat 25000 --act CLipSwish --densenet_depth 3 --densenet_growth 16 --monotone_resolvent False --brute-force True\n```\n\n* Pinwheel, Monotone Flows\n```\npython train_toy.py --data pinwheel --nblocks 10 --save experiments/toy-2d/pinwheel_monotoneflow --densenet True --learnable_concat True --start_learnable_concat 25000 --act CPila --densenet_depth 3 --densenet_growth 16 --monotone_resolvent True --brute-force True\n```\n\n* Rings, i-DenseNets\n```\npython train_toy.py --data rings --nblocks 10 --save experiments/toy-2d/rings_idensenet --densenet True --learnable_concat True --start_learnable_concat 25000 --act CLipSwish --densenet_depth 3 --densenet_growth 16 --monotone_resolvent False --brute-force True\n```\n\n* Rings, Monotone Flows\n```\npython train_toy.py --data rings --nblocks 10 --save experiments/toy-2d/rings_monotoneflow --densenet True --learnable_concat True --start_learnable_concat 25000 --act CPila --densenet_depth 3 --densenet_growth 16 --monotone_resolvent True --brute-force True\n```\n\n* Swissroll, i-DenseNets\n```\npython train_toy.py --data swissroll --nblocks 10 --save experiments/toy-2d/swissroll_idensenet --densenet True --learnable_concat True --start_learnable_concat 25000 --act CLipSwish --densenet_depth 3 --densenet_growth 16 --monotone_resolvent False --brute-force True\n```\n\n* Swissroll, Monotone Flows\n```\npython train_toy.py --data swissroll --nblocks 10 --save experiments/toy-2d/swissroll_monotoneflow --densenet True --learnable_concat True --start_learnable_concat 25000 --act CPila --densenet_depth 3 --densenet_growth 16 --monotone_resolvent True --brute-force True\n```\n\n\n## Image density modeling experiments\n\n* MNIST (batch size 64, distributed, 4 GPUs)\n```\nCUDA_VISIBLE_DEVICES=0,1,2,3 python train_img.py --data mnist --nblocks 16-16-16 --save experiments/mnist --densenet True --learnable_concat True --start_learnable_concat 25 --act CPila --densenet_depth 3 --densenet_growth 172 --monotone_resolvent True --n-exact-terms 10 --nworkers 0 --batchsize 64 --update-freq 4 --print-freq 20 --distributed False --mem-eff False --imagesize 28\n```\n\n* CIFAR-10 (batch size 64, distributed, 4 GPUs)\n```\nCUDA_VISIBLE_DEVICES=0,1,2,3 python train_img.py --data cifar10 --nblocks 16-16-16 --save experiments/cifar10 --densenet True --learnable_concat True --start_learnable_concat 25 --act CPila --densenet_depth 3 --densenet_growth 172 --monotone_resolvent True --n-exact-terms 10 --nworkers 4 --batchsize 64 --update-freq 1 --print-freq 5 --distributed True\n```\n\n* ImageNet32 (batch size 256, distributed, 8 GPUs)\n```\nCUDA_VISIBLE_DEVICES=0,1,2,3,4,5,6,7 python train_img.py --data imagenet32 --nblocks 32-32-32 --save experiments/imagenet32 --densenet True --learnable_concat True --start_learnable_concat 0 --act CPila --densenet_depth 3 --densenet_growth 172 --squeeze-first False --factor-out False --fc-end True --lr 0.004 --monotone_resolvent True --n-exact-terms 10 --nworkers 4 --batchsize 256 --update-freq 1 --print-freq 5 --distributed True\n```\n\n* ImageNet64 (batch size 256, distributed, 8 GPUs)\n```\nCUDA_VISIBLE_DEVICES=0,1,2,3,4,5,6,7 python train_img.py --data imagenet64 --nblocks 32-32-32 --save experiments/imagenet64 --densenet True --learnable_concat True --start_learnable_concat 0 --act CPila --densenet_depth 3 --densenet_growth 172 --squeeze-first True --factor-out True --fc-end True --lr 0.004 --monotone_resolvent True --n-exact-terms 10 --nworkers 4 --batchsize 256 --update-freq 1 --print-freq 5 --distributed True\n```\n\n\n## Ablation experiments on CIFAR-10 image density modeling\n\n* Row #1 (ablates both)\n```\nCUDA_VISIBLE_DEVICES=0,1,2,3 python train_img.py --data cifar10 --nblocks 16-16-16 --save experiments/cifar10 --densenet True --learnable_concat True --start_learnable_concat 25 --act CLipSwish --densenet_depth 3 --densenet_growth 172 --monotone_resolvent False --n-exact-terms 10 --nworkers 4 --batchsize 64 --update-freq 1 --print-freq 5 --distributed True\n```\n\n* Row #2 (ablates the monotone formulation)\n```\nCUDA_VISIBLE_DEVICES=0,1,2,3 python train_img.py --data cifar10 --nblocks 16-16-16 --save experiments/cifar10 --densenet True --learnable_concat True --start_learnable_concat 25 --act CPila --densenet_depth 3 --densenet_growth 172 --monotone_resolvent False --n-exact-terms 10 --nworkers 4 --batchsize 64 --update-freq 1 --print-freq 5 --distributed True\n```\n\n* Row #3 (ablates CPila)\n```\nCUDA_VISIBLE_DEVICES=0,1,2,3 python train_img.py --data cifar10 --nblocks 16-16-16 --save experiments/cifar10 --densenet True --learnable_concat True --start_learnable_concat 25 --act CLipSwish --densenet_depth 3 --densenet_growth 172 --monotone_resolvent True --n-exact-terms 10 --nworkers 4 --batchsize 64 --update-freq 1 --print-freq 5 --distributed True\n```\n\n* Row #4 (the same as the command for the CIFAR-10 density modeling experiment)\n```\nCUDA_VISIBLE_DEVICES=0,1,2,3 python train_img.py --data cifar10 --nblocks 16-16-16 --save experiments/cifar10 --densenet True --learnable_concat True --start_learnable_concat 25 --act CPila --densenet_depth 3 --densenet_growth 172 --monotone_resolvent True --n-exact-terms 10 --nworkers 4 --batchsize 64 --update-freq 1 --print-freq 5 --distributed True\n```\n\n\n## Variational dequantization experiments on CIFAR-10 and ImageNet32\n\n* CIFAR-10\n```\npython train_img.py --data cifar10 --nblocks 16-16-16 --save experiments/cifar10_vdq --densenet True --learnable_concat True --start_learnable_concat 25 --act CPila --densenet_depth 3 --densenet_growth 172 --monotone_resolvent True --n-exact-terms 10 --nworkers 4 --batchsize 64 --lr 0.001 --update-freq 1 --print-freq 1 --distributed True --val-batchsize 64 --var-deq True --var-deq-nblocks 3 --var-deq-act CPila --var-deq-mf False\n```\n\n* ImageNet32\n```\npython train_img.py --data imagenet32 --nblocks 32-32-32 --save experiments/imagenet32_vdq --densenet True --learnable_concat True --start_learnable_concat 2 --act CPila --densenet_depth 3 --densenet_growth 172 --squeeze-first False --factor-out False --fc-end False --lr 0.002 --monotone_resolvent True --n-exact-terms 10 --nworkers 4 --batchsize 128 --update-freq 1 --print-freq 1 --distributed True --var-deq True --var-deq-nblocks 3 --var-deq-act CPila --var-deq-mf False --neumann-grad True --mem-eff True\n```\n\n\n## Classification experiments on CIFAR-10\n\n* Tiny (k=1), i-DenseNets\n```\nCUDA_VISIBLE_DEVICES=0,1 python train_img.py --data cifar10 --nblocks 1-1-1 --save experiments/cifar10_cls_tiny_monotoneflow --densenet True --learnable_concat True --start_learnable_concat 25 --act CLipSwish --densenet_depth 3 --densenet_growth 80 --monotone_resolvent False --n-exact-terms 10 --nworkers 4 --batchsize 128 --update-freq 1 --print-freq 5 --distributed True --fc-end False --mem-eff False --lr 0.001 --task classification --clip_grad_norm False --nepochs 200\n```\n\n* Tiny (k=1), Monotone Flows\n```\nCUDA_VISIBLE_DEVICES=0,1 python train_img.py --data cifar10 --nblocks 1-1-1 --save experiments/cifar10_cls_tiny_monotoneflow --densenet True --learnable_concat True --start_learnable_concat 25 --act CPila --densenet_depth 3 --densenet_growth 80 --monotone_resolvent True --n-exact-terms 10 --nworkers 4 --batchsize 128 --update-freq 1 --print-freq 5 --distributed True --fc-end False --mem-eff False --lr 0.001 --task classification --clip_grad_norm False --nepochs 200\n```\n\n* Small (k=4), i-DenseNets\n```\nCUDA_VISIBLE_DEVICES=0,1 python train_img.py --data cifar10 --nblocks 4-4-4 --save experiments/cifar10_cls_tiny_monotoneflow --densenet True --learnable_concat True --start_learnable_concat 25 --act CLipSwish --densenet_depth 3 --densenet_growth 80 --monotone_resolvent False --n-exact-terms 10 --nworkers 4 --batchsize 128 --update-freq 1 --print-freq 5 --distributed True --fc-end False --mem-eff False --lr 0.001 --task classification --clip_grad_norm False --nepochs 200\n```\n\n* Small (k=4), Monotone Flows\n```\nCUDA_VISIBLE_DEVICES=0,1 python train_img.py --data cifar10 --nblocks 4-4-4 --save experiments/cifar10_cls_tiny_monotoneflow --densenet True --learnable_concat True --start_learnable_concat 25 --act CPila --densenet_depth 3 --densenet_growth 80 --monotone_resolvent True --n-exact-terms 10 --nworkers 4 --batchsize 128 --update-freq 1 --print-freq 5 --distributed True --fc-end False --mem-eff False --lr 0.001 --task classification --clip_grad_norm False --nepochs 200\n```\n\n* Large (k=16), i-DenseNets\n```\nCUDA_VISIBLE_DEVICES=0,1 python train_img.py --data cifar10 --nblocks 16-16-16 --save experiments/cifar10_cls_tiny_monotoneflow --densenet True --learnable_concat True --start_learnable_concat 25 --act CLipSwish --densenet_depth 3 --densenet_growth 80 --monotone_resolvent False --n-exact-terms 10 --nworkers 4 --batchsize 128 --update-freq 1 --print-freq 5 --distributed True --fc-end False --mem-eff False --lr 0.001 --task classification --clip_grad_norm False --nepochs 200\n```\n\n* Large (k=16), Monotone Flows\n```\nCUDA_VISIBLE_DEVICES=0,1 python train_img.py --data cifar10 --nblocks 16-16-16 --save experiments/cifar10_cls_tiny_monotoneflow --densenet True --learnable_concat True --start_learnable_concat 25 --act CPila --densenet_depth 3 --densenet_growth 80 --monotone_resolvent True --n-exact-terms 10 --nworkers 4 --batchsize 128 --update-freq 1 --print-freq 5 --distributed True --fc-end False --mem-eff False --lr 0.001 --task classification --clip_grad_norm False --nepochs 200\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmlvlab%2FMonotoneFlows","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fmlvlab%2FMonotoneFlows","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmlvlab%2FMonotoneFlows/lists"}