{"id":13784478,"url":"https://github.com/openai/iaf","last_synced_at":"2025-04-05T08:08:29.816Z","repository":{"id":48124508,"uuid":"61248329","full_name":"openai/iaf","owner":"openai","description":"Code for reproducing key results in the paper \"Improving Variational Inference with Inverse Autoregressive Flow\"","archived":false,"fork":false,"pushed_at":"2018-11-22T00:39:33.000Z","size":138,"stargazers_count":520,"open_issues_count":14,"forks_count":133,"subscribers_count":186,"default_branch":"master","last_synced_at":"2025-03-29T07:08:36.631Z","etag":null,"topics":["paper"],"latest_commit_sha":null,"homepage":"https://arxiv.org/abs/1606.04934","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/openai.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2016-06-15T23:46:20.000Z","updated_at":"2025-03-21T18:18:40.000Z","dependencies_parsed_at":"2022-09-12T04:20:55.023Z","dependency_job_id":null,"html_url":"https://github.com/openai/iaf","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/openai%2Fiaf","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/openai%2Fiaf/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/openai%2Fiaf/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/openai%2Fiaf/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/openai","download_url":"https://codeload.github.com/openai/iaf/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":247305935,"owners_count":20917208,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["paper"],"created_at":"2024-08-03T19:00:45.256Z","updated_at":"2025-04-05T08:08:29.791Z","avatar_url":"https://github.com/openai.png","language":"Python","funding_links":[],"categories":["📝 Publications \u003csmall\u003e(60)\u003c/small\u003e"],"sub_categories":[],"readme":"**Status:** Archive (code is provided as-is, no updates expected)\n\n# Improve Variational Inference with Inverse Autoregressive Flow\n\nCode for reproducing key results in the paper [Improving Variational Inference with Inverse Autoregressive Flow](http://arxiv.org/abs/1606.04934) by Diederik P. Kingma, Tim Salimans, Rafal Jozefowicz, Xi Chen, Ilya Sutskever, and Max Welling.\n\n## Prerequisites\n\n1. Make sure that recent versions installed of:\n    - Python (version 2.7 or higher)\n    - Numpy (e.g. `pip install numpy`)\n    - Theano (e.g. `pip install Theano`)\n\n2. Set `floatX = float32` in the `[global]` section of Theano config (usually `~/.theanorc`). Alternatively you could prepend `THEANO_FLAGS=floatX=float32 ` to the python commands below. \n\n3. Clone this repository, e.g.:\n```sh\ngit clone https://github.com/openai/iaf.git\n```\n\n4. Download the [CIFAR-10 dataset](https://www.cs.toronto.edu/~kriz/cifar.html) (get the *Python* version) and create an environment variable `CIFAR10_PATH` that points to the subdirectory with CIFAR-10 data. For example:\n```sh\nexport CIFAR10_PATH=\"$HOME/cifar-10\"\n```\n\n## Syntax of train.py\n\nExample:\n```sh\npython train.py with problem=cifar10 n_z=32 n_h=64 depths=[2,2,2] margs.depth_ar=1 margs.posterior=down_iaf2_NL margs.kl_min=0.25\n```\n\n`problem` is the problem (dataset) to train on. I only tested `cifar10` for this release.\n\n`n_z` is the number of stochastic featuremaps in each layer.\n\n`n_h` is the number of deterministic featuremaps used throughout the model.\n\n`depths` is an array of integers that denotes the depths of the *levels* in the model. Each level is a sequence of layers. Each subsequent level operates over spatially smaller featuremaps. In case of CIFAR-10, the first level operates over 16x16 featuremaps, the second over 8x8 featuremaps, etc.\n\nSome possible choices for `margs.posterior` are:\n- `up_diag`: bottom-up factorized Gaussian\n- `up_iaf1_nl`: bottom-up IAF, mean-only perturbation\n- `up_iaf2_nl`: bottom-up IAF\n- `down_diag`: top-down factorized Gaussian\n- `down_iaf1_nl`: top-down IAF, mean-only perturbation\n- `down_iaf2_nl`: top-down IAF\n\n`margs.depth_ar` is the number of hidden layers within IAF, and can be any non-negative integer.\n\n`margs.kl_min`: the minimum information constraint. Should be a non-negative float (where 0 is no constraint).\n\n## Results of Table 3\n\n(3.28 bits/dim)\n\n```sh\npython train.py with problem=cifar10 n_h=160 depths=[10,10] margs.depth_ar=2 margs.posterior=down_iaf2_nl margs.prior=diag margs.kl_min=0.25\n```\n\nMore instructions will follow.\n\n\n## Multi-GPU TensorFlow implementation\n\n### Prerequisites\n\nMake sure that recent versions installed of:\n- Python (version 2.7 or higher)\n- TensorFlow\n- tqdm\n   \n`CIFAR10_PATH` environment variable should point to the dataset location.\n\n### Syntax of tf_train.py\n\nTraining script:\n```sh\npython tf_train.py --logdir \u003clogdir\u003e --hpconfig depth=1,num_blocks=20,kl_min=0.1,learning_rate=0.002,batch_size=32 --num_gpus 8 --mode train\n```\n\nIt will run the training procedure on a given number of GPUs. Model checkpoints will be stored in `\u003clogdir\u003e/train` directory along with TensorBoard summaries that are useful for monitoring and debugging issues.\n\nEvaluation script:\n```sh\npython tf_train.py --logdir \u003clogdir\u003e --hpconfig depth=1,num_blocks=20,kl_min=0.1,learning_rate=0.002,batch_size=32 --num_gpus 1 --mode eval_test\n```\n\nIt will run the evaluation on the test set using a single GPU and will produce TensorBoard summary with the results and generated samples.\n\nTo start TensorBoard:\n```sh\ntensorboard --logdir \u003clogdir\u003e\n```\n\nFor the description of hyper-parameters, take a look at `get_default_hparams` function in `tf_train.py`.\n\n\n### Loading from the checkpoint\n\nThe best IAF model trained on CIFAR-10 reached 3.15 bits/dim when evaluated with a single sample. With 10,000 samples, the estimation of log likelihood is 3.111 bits/dim.\nThe checkpoint is available at [link](https://drive.google.com/file/d/0B-pv8mYT4p0OOXFfWElyeUs0bUk/view?usp=sharing).\nSteps to use it:\n- download the file\n- create directory `\u003clogdir\u003e/train/` and copy the checkpoint there\n- run the following command:\n```sh\npython tf_train.py --logdir \u003clogdir\u003e --hpconfig depth=1,num_blocks=20,kl_min=0.1,learning_rate=0.002,batch_size=32 --num_gpus 1 --mode eval_test\n```\n\nThe script will run the evaluation on the test set and generate samples stored in TensorFlow events file that can be accessed using TensorBoard.","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fopenai%2Fiaf","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fopenai%2Fiaf","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fopenai%2Fiaf/lists"}