{"id":13738592,"url":"https://github.com/allenai/dnw","last_synced_at":"2025-10-24T11:42:39.590Z","repository":{"id":75564506,"uuid":"194745223","full_name":"allenai/dnw","owner":"allenai","description":"Discovering Neural Wirings (https://arxiv.org/abs/1906.00586)","archived":false,"fork":false,"pushed_at":"2020-01-28T22:51:16.000Z","size":12659,"stargazers_count":138,"open_issues_count":5,"forks_count":17,"subscribers_count":8,"default_branch":"master","last_synced_at":"2024-11-15T07:34:59.929Z","etag":null,"topics":["computer-vision","machine-learning","neural-networks"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"other","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/allenai.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null}},"created_at":"2019-07-01T21:27:51.000Z","updated_at":"2024-11-07T13:10:07.000Z","dependencies_parsed_at":"2023-06-06T22:15:46.201Z","dependency_job_id":null,"html_url":"https://github.com/allenai/dnw","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/allenai%2Fdnw","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/allenai%2Fdnw/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/allenai%2Fdnw/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/allenai%2Fdnw/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/allenai","download_url":"https://codeload.github.com/allenai/dnw/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":253105712,"owners_count":21855084,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["computer-vision","machine-learning","neural-networks"],"created_at":"2024-08-03T03:02:28.468Z","updated_at":"2025-10-13T15:57:47.385Z","avatar_url":"https://github.com/allenai.png","language":"Python","funding_links":[],"categories":["Python"],"sub_categories":[],"readme":"# [Discovering Neural Wirings](https://arxiv.org/abs/1906.00586)\n\nBy Mitchell Wortsman, Ali Farhadi and Mohammad Rastegari.\n\n\n[Preprint](https://arxiv.org/abs/1906.00586) | [Blog](https://mitchellnw.github.io/blog/2019/dnw/) | [BibTex](#citing)\n\n![](fig/dnw.gif)\n\nIn this work we propose a method for discovering neural wirings.\nWe relax the typical notion of layers and instead enable channels\nto form connections independent of each other.\nThis allows for a much larger space of possible networks.\nThe wiring of our network is not fixed during training --\nas we learn the network parameters we also learn the structure itself.\n\nThe folder `imagenet_sparsity_experiments` contains the code for training sparse neural networks. \n\n## Citing\n\nIf you find this project useful in your research, please consider citing:\n\n```\n@article{Wortsman2019DiscoveringNW,\n  title={Discovering Neural Wirings},\n  author={Mitchell Wortsman and Ali Farhadi and Mohammad Rastegari},\n  journal={ArXiv},\n  year={2019},\n  volume={abs/1906.00586}\n}\n```\n\n\n## Set Up\n0. Clone this repository.\n1. Using `python 3.6`, create a `venv` with  `python -m venv venv` and run `source venv/bin/activate`.\n2. Install requirements with `pip install -r requirements.txt`.\n3. Create a **data directory** `\u003cdata-dir\u003e`.\nIf you wish to run ImageNet experiments there must be a folder `\u003cdata-dir\u003e/imagenet`\nthat contains the ImageNet `train` and `val`. By running experiments on CIFAR-10 a\nfolder `\u003cdata-dir\u003e/cifar10` will automatically be created with the dataset.\n\n## Small Scale Experiments\n\nTo test a tiny (41k parameters) classifier on CIFAR-10 in static and dynamic settings, see `apps/small_scale`.\nThere are 6 experiment files in total -- 3 for random graphs and 3 for discovering neural wirings (DNW).\n\nYou may run an experiment with\n```bash\npython runner.py app:apps/small_scale/\u003cexperiment-file\u003e --gpus 0 --data-dir \u003cdata-dir\u003e\n```\n\nWe recommend running the static and discrete time experiments on a single GPU (as above), though you will need to use\nmultiple GPUs for the continuous time experiments. To do this you may use `--gpus 0 1`.\n\nYou should expect the following result:\n\n| Model  | Accuracy (CIFAR-10) |\n| :-------------: | :-------------: |\nStatic (Random Graph) | 76.1 \u0026pm;  0.5|\nStatic (DNW) | 80.9  \u0026pm;  0.6 |\nDiscrete Time (Random Graph) | 77.3 \u0026pm;  0.7|\nDiscrete Time (DNW) | 82.3 \u0026pm;  0.6 |\nContinuous (Random Graph) | 78.5  \u0026pm;  1.2  |\nContinuous (DNW) | 83.1  \u0026pm;  0.3 |\n\n## ImageNet Experiments and Pretrained Models\n\nThe experiment files for the ImageNet experiments in the paper may be found in `apps/large_scale`.\nTo train your own model you may run\n```bash\npython runner.py app:apps/large_scale/\u003cexperiment-file\u003e --gpus 0 1 2 3 --data-dir \u003cdata-dir\u003e\n```\nand to evaluate a pretrained model which matches the experiment file use.\n```bash\npython runner.py app:apps/large_scale/\u003cexperiment-file\u003e --gpus 0 1 --data-dir \u003cdata-dir\u003e --resume \u003cpath-to-pretrained-model\u003e --evaluate\n```\n\n\n| Model  | Params | FLOPs | Accuracy (ImageNet) |\n| :-------------: | :-------------: | :-------------: | :-------------: |\n| [MobileNet V1 (x 0.25)](https://arxiv.org/abs/1704.04861)  |  0.5M  | 41M  | 50.6  |\n| [ShuffleNet V2 (x 0.5)](https://arxiv.org/abs/1807.11164)  |  1.4M | 41M  | 60.3 |\n| [MobileNet V1 (x 0.5)](https://arxiv.org/abs/1704.04861)  |  1.3M | 149M  | 63.7 |\n| [ShuffleNet V2 (x 1)](https://arxiv.org/abs/1807.11164)  |  2.3M | 146M  | 69.4 |\n| [MobileNet V1 Random Graph (x 0.225)](https://prior-datasets.s3.us-east-2.amazonaws.com/dnw/pretrained-models/rg_x225.pt)  |  1.2M | 55.7M  | 53.3 |\n| [MobileNet V1 DNW Small (x 0.15)](https://prior-datasets.s3.us-east-2.amazonaws.com/dnw/pretrained-models/dnw_small_x15.pt)  |  0.24M | 22.1M  | 50.3 |\n| [MobileNet V1 DNW Small (x 0.225)](https://prior-datasets.s3.us-east-2.amazonaws.com/dnw/pretrained-models/dnw_small_x225.pt)  |  0.4M | 41.2M  | 59.9 |\n| [MobileNet V1 DNW (x 0.225)](https://prior-datasets.s3.us-east-2.amazonaws.com/dnw/pretrained-models/dnw_x225.pt)  |  1.1M | 42.1M | 60.9 |\n| [MobileNet V1 DNW (x 0.3)](https://prior-datasets.s3.us-east-2.amazonaws.com/dnw/pretrained-models/dnw_x3.pt)  | 1.3M | 66.7M | 65.0 |\n| [MobileNet V1 Random Graph (x 0.49)](https://prior-datasets.s3.us-east-2.amazonaws.com/dnw/pretrained-models/rg_x49.pt)  |  1.8M | 170M  | 64.1 |\n| [MobileNet V1 DNW (x 0.49)](https://prior-datasets.s3.us-east-2.amazonaws.com/dnw/pretrained-models/dnw_x49.pt)  | 1.8M  | 154M  | 70.4 |\n\nYou may also add the flag `--fast_eval` to make the model smaller and speed up inference. Adding `--fast_eval` removes the neurons which _die_.\nAs a result, the first conv, last linear layer, and all operations throughout have much fewer input and output channels. You may add both\n`--fast_eval` and `--use_dgl` to obtain a model for evaluation that matches the theoretical FLOPs by using a graph implementation via\n[https://www.dgl.ai/](https://www.dgl.ai/). You must then install the version of `dgl` which matches your CUDA and Python version\n(see [this](https://www.dgl.ai/pages/start.html) for more details). For example, we run\n```bash\npip uninstall dgl\npip install https://s3.us-east-2.amazonaws.com/dgl.ai/wheels/cuda9.2/dgl-0.3-cp36-cp36m-manylinux1_x86_64.whl\n```\nand finally\n```bash\npython runner.py app:apps/large_scale/\u003cexperiment-file\u003e --gpus 0 --data-dir \u003cdata-dir\u003e --resume \u003cpath-to-pretrained-model\u003e --evaluate --fast_eval --use_dgl --batch_size 256\n```\n\n## Other Methods of Discovering Neural Wirings\n\nTo explore other methods of discovering neural wirings see `apps/medium_scale`.\n\nYou may run an experiment with\n```bash\npython runner.py app:apps/medium_scale/\u003cexperiment-file\u003e --gpus 0 --data-dir \u003cdata-dir\u003e\n```\n\nTo replicate the one-shot pruning or fine tuning experiments, you must first use `mobilenetv1_complete_graph.yml` to obtain the initialization `init.pt` and the final epoch.  \n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fallenai%2Fdnw","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fallenai%2Fdnw","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fallenai%2Fdnw/lists"}