{"id":20832397,"url":"https://github.com/ermongroup/fast_feedforward_computation","last_synced_at":"2025-05-08T01:21:26.134Z","repository":{"id":45102944,"uuid":"376141511","full_name":"ermongroup/fast_feedforward_computation","owner":"ermongroup","description":"Official code for \"Accelerating Feedforward Computation via Parallel Nonlinear Equation Solving\", ICML 2021","archived":false,"fork":false,"pushed_at":"2021-09-25T18:25:36.000Z","size":1632,"stargazers_count":27,"open_issues_count":0,"forks_count":1,"subscribers_count":7,"default_branch":"main","last_synced_at":"2025-03-31T16:09:54.165Z","etag":null,"topics":["acceleration","autoregressive-models","densenet","icml-2021","made","numerical-methods","pixelcnn","rnn"],"latest_commit_sha":null,"homepage":"","language":"Jupyter Notebook","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/ermongroup.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2021-06-11T21:01:45.000Z","updated_at":"2025-01-27T11:03:14.000Z","dependencies_parsed_at":"2022-07-13T15:30:42.648Z","dependency_job_id":null,"html_url":"https://github.com/ermongroup/fast_feedforward_computation","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ermongroup%2Ffast_feedforward_computation","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ermongroup%2Ffast_feedforward_computation/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ermongroup%2Ffast_feedforward_computation/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ermongroup%2Ffast_feedforward_computation/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/ermongroup","download_url":"https://codeload.github.com/ermongroup/fast_feedforward_computation/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":252979486,"owners_count":21835061,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["acceleration","autoregressive-models","densenet","icml-2021","made","numerical-methods","pixelcnn","rnn"],"created_at":"2024-11-18T00:11:42.801Z","updated_at":"2025-05-08T01:21:26.105Z","avatar_url":"https://github.com/ermongroup.png","language":"Jupyter Notebook","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Accelerating Feedforward Computation via Parallel Nonlinear Equation Solving\n\nThis repo contains the official implementation for the ICML 2021 paper [Accelerating Feedforward Computation via Parallel Nonlinear Equation Solving](https://arxiv.org/abs/2002.03629), \n\nby [Yang Song](https://yang-song.github.io), [Chenlin Meng](https://chenlin9.github.io/), [Renjie Liao](http://www.cs.toronto.edu/~rjliao/), and [Stefano Ermon](https://cs.stanford.edu/~ermon/).\n\n----------------------\n\nWe show that by viewing feedforward computation as a system of nonlinear equations, we can leverage parallel equation solvers  to improve its speed. Our methods are particularly efficient when the computation graph contains many long skip connections, and can be used to accelerate the backpropagation of recurrent neural networks, inference of DenseNets, and autoregressive sampling from MADE and PixelCNN++.\n\n![schematic](assets/schematic.png)\n\n## Dependencies\n\n```\njax==0.2.7\njaxlib==0.1.57+cuda101\nflax==0.3.2\ntorch==1.7.1\ntorchvision==0.8.2\n```\n\n## Running experiments\n\nWe run experiments by calling  `main.py`  with the following arguments.\n\n```bash\nusage: main.py [--runner RUNNER] [--config CONFIG]\narguments:\n  --runner RUNNER       The runner to execute\n  --config CONFIG       Path to the config file\n```\n\n`runner` and `config` can be specified below\n\n| Experiment | --runner | --config |\n|:---------------|:---------------:|:------------:|\n| RNN backpropagation | `BackpropRunner` | `backprop.yml` |\n| DenseNet | `DenseNetRunner` | `densenet.yml` |\n| MADE sampling | `MADESamplerRunner` | `made_sampler.yml` |\n| PixelCNN++ sequential \u0026 Jacobi sampling| `PixelCNNPPSamplerRunner` | `pixelcnnpp_sampler.yml` |\n| PixelCNN++ sequential sampling w/ cache | `CachedPixelCNNPPSamplerRunner` | `cached_pixelcnnpp_sampler.yml` |\n| PixelCNN++ Jacobi-GS sampling | `JacobiGSPixelCNNPPSamplerRunner` | `jacobi_gs_pixelcnnpp_sampler.yml` |\n| PixelCNN++ GS-Jacobi sampling | `GSJacobiPixelCNNPPSamplerRunner` | `gs_jacobi_pixelcnnpp_sampler.yml` |\n\nThe configurations of each experiment, such as dataset, block size, and algorithm, are provided in the corresponding config file, which should be straightforward to modify.\n\n## Retrieving experimental results\n\nAfter running `main.py`, we can retrieve and plot experimental results by running `*.ipynb` files with Jupyter Notebook. We assume all experimental data are stored in folder `plot_data/`. Here is the correspondence between each experiment and the `.ipynb` file.\n\n| Experiment | Jupyter Notebook file |\n|:----------|:---------------------:|\n| RNN backpropagation           |  `rnn_backprop.ipynb` |\n| DenseNet | `densenet.ipynb` |\n| MADE sampling | `made.ipynb` |\n| PixelCNN++ sampling | `pixelcnnpp.ipynb` |\n\n## Pretrained checkpoints and raw experimental data\n\nWe provide pretrained checkpoints (in subfolder `runs/pretarined`) and raw experimental data (in subfolder `plot_data`) via the following link:\n\nhttps://drive.google.com/file/d/1cxV3pKUmLETt9veWRnEKt3SBCb-6sZDc/view?usp=sharing\n\n## Reference\n\nIf you find the idea or code useful for your research, please consider citing\n```bibtex\n@inproceedings{song2021accelerating,\n  title={Accelerating Feedforward Computation via Parallel Nonlinear Equation Solving},\n  author={Song, Yang and Meng, Chenlin and Liao, Renjie, and Ermon, Stefano},\n  booktitle = {International Conference on Machine Learning (ICML)},\n  year={2021},\n}\n```\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fermongroup%2Ffast_feedforward_computation","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fermongroup%2Ffast_feedforward_computation","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fermongroup%2Ffast_feedforward_computation/lists"}