{"id":20663802,"url":"https://github.com/vita-group/weaknas","last_synced_at":"2026-03-04T14:32:26.939Z","repository":{"id":107047327,"uuid":"339641725","full_name":"VITA-Group/WeakNAS","owner":"VITA-Group","description":"[NeurIPS 2021] “Stronger NAS with Weaker Predictors“, Junru Wu, Xiyang Dai, Dongdong Chen, Yinpeng Chen, Mengchen Liu, Ye Yu, Zhangyang Wang, Zicheng Liu, Mei Chen and Lu Yuan","archived":false,"fork":false,"pushed_at":"2022-09-23T01:01:53.000Z","size":24613,"stargazers_count":27,"open_issues_count":1,"forks_count":6,"subscribers_count":9,"default_branch":"master","last_synced_at":"2025-10-28T10:43:17.652Z","etag":null,"topics":["deep-learning","mobilenet-space","nas-bench-101","nas-bench-201","nasbench","neural-architecture-search","pytorch"],"latest_commit_sha":null,"homepage":"","language":"Jupyter Notebook","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/VITA-Group.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2021-02-17T07:20:53.000Z","updated_at":"2024-03-14T20:17:30.000Z","dependencies_parsed_at":"2023-04-13T15:47:15.151Z","dependency_job_id":null,"html_url":"https://github.com/VITA-Group/WeakNAS","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/VITA-Group/WeakNAS","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/VITA-Group%2FWeakNAS","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/VITA-Group%2FWeakNAS/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/VITA-Group%2FWeakNAS/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/VITA-Group%2FWeakNAS/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/VITA-Group","download_url":"https://codeload.github.com/VITA-Group/WeakNAS/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/VITA-Group%2FWeakNAS/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":30083761,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-03-04T13:22:36.021Z","status":"ssl_error","status_checked_at":"2026-03-04T13:20:45.750Z","response_time":59,"last_error":"SSL_read: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["deep-learning","mobilenet-space","nas-bench-101","nas-bench-201","nasbench","neural-architecture-search","pytorch"],"created_at":"2024-11-16T19:19:54.911Z","updated_at":"2026-03-04T14:32:26.903Z","avatar_url":"https://github.com/VITA-Group.png","language":"Jupyter Notebook","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Stronger NAS with Weaker Predictors\n[[NeurIPS'21] Stronger NAS with Weaker Predictors](https://arxiv.org/abs/2102.10490).\n\nJunru Wu, Xiyang Dai, Dongdong Chen, Yinpeng Chen, Mengchen Liu, Ye Yu, Zhangyang Wang, Zicheng Liu, Mei Chen and Lu Yuan\n\n## Overview\n\nOur WeakNAS Pipeline\n\u003cp align=\"center\"\u003e\n  \u003cimg src=\"assets/process_raw.png\" alt=\"drawing\" width=\"900\"/\u003e\n\u003c/p\u003e\n\nSearch Dynamic Visualization of WeakNAS in t-SNE\n\u003cp align=\"center\"\u003e\n  \u003cimg src=\"assets/tsne.png\" alt=\"drawing\" width=\"900\"/\u003e\n\u003c/p\u003e\nReproduce the aboved figure:\n\n```bash\nvisualize_search_dynamic.ipynb\n```\n\n## Implementation\n\n- [x] NAS-Bench Search Space\n  - [x] NAS-Bench-101 Search Space (CIFAR10)\n  - [x] NAS-Bench-201 Search Space (CIFAR10, CIFAR100, ImageNet16-120)\n- [ ] Open Domain Search Space\n  - [ ] NASNet Search Space (ImageNet)\n  - [x] MobileNet Search Space (ImageNet)\n- [x] Interpretation\n  - [x] Search Dynamic Visualization in t-SNE (NAS-Bench-201)\n\n### Environment\n```bash\npip install -r requirements.txt\n```\n\n## NASBench Search Space\n\n### NAS-Bench-101\n\nDownload pre-processed NAS-Bench-101 from this [Link](https://drive.google.com/file/d/1v0tvvh3yi_S2oDTJMvqFgFimsH1rCz1H/view?usp=sharing), Replace $BENCH_PATH with the file path\n\nReplace --save_dir with your own log path, repeat at least 100 times for stable result\n\n  - ### run WeakNAS\n  - ```bash\n    python WeakNAS.py --rand_seed -1 --repeat 100 --train_set valid --test_set test \\\n    --save_dir OUTPUT/nasbench101/init_100_sample_10/top_start_100_end_100/MLP/onehot_size_1000_1000_1000_1000_iter_100/acq_uniform/deter_False \\\n    --bench_path $BENCH_PATH --bench nasbench101 --dataset cifar10 --deterministic False \\\n    --top_start 100 --top_end 100 --init_sample 100 --sample_each_iter 10 --sampling_method uniform \\\n    --predictor MLP --max_sample 1000 --mlp_size 1000 1000 1000 1000 --mlp_iter 100\n    ```\n  - ### run WeakNAS + EI Variant\n  - ```bash\n    python WeakNAS.py --rand_seed -1 --repeat 100 --train_set valid --test_set test \\\n    --save_dir OUTPUT/nasbench101/init_100_sample_10/top_start_100_end_100/MLP/onehot_size_1000_1000_1000_1000_iter_100/acq_ei/deter_False \\\n    --bench_path $BENCH_PATH --bench nasbench101 --dataset cifar10 --deterministic False \\\n    --top_start 100 --top_end 100 --init_sample 100 --sample_each_iter 10 --sampling_method ei \\\n    --predictor MLP --max_sample 1000 --mlp_size 1000 1000 1000 1000 --mlp_iter 100 \n    ```\n    Note: EI calculation took very long time (~10 hrs each run)\n  - ###  Plot Figure\n  - ```bash\n    plot_nasbench101.ipynb\n    ```\n    \n### NAS-Bench-201\n\n- CIFAR10 Subset\n\n  - Download pre-processed NAS-Bench-201 CIFAR10 Subset from this [Link](https://drive.google.com/uc?export=download\u0026id=1-BG879iwZJpgdKlSYM6cgnr5DHerRLs-), Replace $BENCH_PATH with the file path\n\n  - Replace --save_dir with your own log path, repeat at least 100 times for stable result\n\n  - ### run WeakNAS\n  - ```bash\n    python WeakNAS.py --rand_seed -1 --repeat 100 --train_set x-valid --test_set ori-test \\\n    --save_dir OUTPUT/nasbench201/cifar10/init_10_sample_10/top_start_100_end_100/MLP/onehot_size_1000_1000_1000_1000_iter_100/acq_uniform/deter_False \\\n    --bench_path $BENCH_PATH --bench nasbench201 --dataset cifar10-valid --deterministic False \\\n    --top_start 100 --top_end 100 --init_sample 10 --sample_each_iter 10 --sampling_method uniform \\\n    --predictor MLP --max_sample 1000 --mlp_size 1000 1000 1000 1000 --mlp_iter 100\n    ```\n  - ### run WeakNAS + EI Variant\n  - ```bash\n    python WeakNAS.py --rand_seed -1 --repeat 100 --train_set x-valid --test_set ori-test \\\n    --save_dir OUTPUT/nasbench201/cifar10/init_10_sample_10/top_start_100_end_100/MLP/onehot_size_1000_1000_1000_1000_iter_100/acq_ei/deter_False \\\n    --bench_path $BENCH_PATH --bench nasbench201 --dataset cifar10-valid --deterministic False \\\n    --top_start 100 --top_end 100 --init_sample 10 --sample_each_iter 10 --sampling_method ei \\\n    --predictor MLP --max_sample 1000 --mlp_size 1000 1000 1000 1000 --mlp_iter 100\n    ```\n    Note: EI calculation took very long time (~5 hrs each run)\n  \n\n- CIFAR100 Subset\n\n  - Download pre-processed NAS-Bench-201 CIFAR100 Subset from this [Link](https://drive.google.com/uc?export=download\u0026id=1RzeGdu_8BEpOKcDgQMY2uaoWTokqrUlc), Replace $BENCH_PATH with the file path\n\n  - Replace --save_dir with your own log path, repeat at least 100 times for stable result\n  \n  - ### run WeakNAS\n  - ```bash\n    python WeakNAS.py --rand_seed -1 --repeat 100 --train_set x-valid --test_set x-test \\\n    --save_dir OUTPUT/nasbench201/cifar100/init_10_sample_10/top_start_100_end_100/MLP/onehot_size_1000_1000_1000_1000_iter_100/acq_uniform/deter_False \\\n    --bench_path $BENCH_PATH --bench nasbench201 --dataset cifar100 --deterministic False \\\n    --top_start 100 --top_end 100 --init_sample 10 --sample_each_iter 10 --sampling_method uniform \\\n    --predictor MLP --max_sample 1000 --mlp_size 1000 1000 1000 1000 --mlp_iter 100\n    ```\n  - ### run WeakNAS + EI Variant\n  - ```bash\n    python WeakNAS.py --rand_seed -1 --repeat 100 --train_set x-valid --test_set x-test \\\n    --save_dir OUTPUT/nasbench201/cifar100/init_10_sample_10/top_start_100_end_100/MLP/onehot_size_1000_1000_1000_1000_iter_100/acq_ei/deter_False \\\n    --bench_path $BENCH_PATH --bench nasbench201 --dataset cifar100 --deterministic False \\\n    --top_start 100 --top_end 100 --init_sample 10 --sample_each_iter 10 --sampling_method ei \\\n    --predictor MLP --max_sample 1000 --mlp_size 1000 1000 1000 1000 --mlp_iter 100\n    ```\n    Note: EI calculation took very long time (~5 hrs each run)\n\n\n - ImageNet16-120 Subset\n\n   - Download pre-processed NAS-Bench-201 ImageNet16-120 Subset from this [Link](https://drive.google.com/uc?export=download\u0026id=1saTFD1-uIPtwuj9MPekw0wnNKS1p3sJE), Replace $BENCH_PATH with the file path\n\n   - Replace --save_dir with your own log path, repeat at least 100 times for stable result\n\n   - ### run WeakNAS\n   - ```bash\n     python WeakNAS.py --rand_seed -1 --repeat 100 --train_set x-valid --test_set x-test \\\n     --save_dir OUTPUT/nasbench201/ImageNet16-120/init_10_sample_10/top_start_100_end_100/MLP/onehot_size_1000_1000_1000_1000_iter_100/acq_uniform/deter_False \\\n     --bench_path $BENCH_PATH --bench nasbench201 --dataset ImageNet16-120 --deterministic False \\\n     --top_start 100 --top_end 100 --init_sample 10 --sample_each_iter 10 --sampling_method uniform \\\n     --predictor MLP --max_sample 1000 --mlp_size 1000 1000 1000 1000 --mlp_iter 100\n     ```\n   - ### run WeakNAS + EI Variant\n   - ```bash\n     python WeakNAS.py --rand_seed -1 --repeat 100 --train_set x-valid --test_set x-test \\\n     --save_dir OUTPUT/nasbench201/ImageNet16-120/init_10_sample_10/top_start_100_end_100/MLP/onehot_size_1000_1000_1000_1000_iter_100/acq_ei/deter_False \\\n     --bench_path $BENCH_PATH --bench nasbench201 --dataset ImageNet16-120 --deterministic False \\\n     --top_start 100 --top_end 100 --init_sample 10 --sample_each_iter 10 --sampling_method ei \\\n     --predictor MLP --max_sample 1000 --mlp_size 1000 1000 1000 1000 --mlp_iter 100\n     ```\n     Note: EI calculation took very long time (~5 hrs each run)\n \n  - ###  Plot Figure\n  - ```bash\n    plot_nasbench201.ipynb\n    ```\n\n## Open Domain Search Space\n  \n### ImageNet (MobileNet Setting)\n\n\u003cp align=\"center\"\u003e\n  \u003cimg src=\"assets/imagenet@mobilenet_compare.png\" alt=\"drawing\" width=\"540\"/\u003e\n\u003c/p\u003e\nBest architecture founded by WeakNAS\n\u003cp align=\"center\"\u003e\n  \u003cimg src=\"assets/imagenet@mobilenet_best.png\" alt=\"drawing\" width=\"750\"/\u003e\n\u003c/p\u003e\n\n- Train SuperNet\n  - We use the codebase [OFA](https://github.com/mit-han-lab/once-for-all) as our training pipeline, directly reuse the weight from pretrain SuperNet variant \"ofa_mbv3_d234_e346_k357_w1.2\".\n- Search\n  - More details in imagenet_mobilenet_search.ipynb, it will print out the founded best architecture(s)\n- Train from scratch\n  - We use the [pytorch-image-models](https://github.com/rwightman/pytorch-image-models) codebase as our training pipeline.\n  - Our run of best architecture founded by WeakNAS\n    - Best architecture @800 Queries\n    ```bash\n    cd pytorch-image-models;\n    bash distributed_train.sh $NUM_GPU $IMAGENET_PATH --model ofa_mbv3_800 -b 128 \\\n    --sched cosine --img-size 236 --epochs 300 --warmup-epochs 3 --decay-rate .97 \\\n    --opt rmsproptf --opt-eps .001 -j 10 --warmup-lr 1e-6 --weight-decay 1e-05 --drop 0.3 \\\n    --drop-path 0.0 --model-ema --model-ema-decay 0.9999 --aa rand-m9-mstd0.5 \\\n    --remode pixel --reprob 0.2 --lr 1e-02 --output $LOG_PATH \\\n    --experiment res_236/bs_128/cosine/lr_5e-03/wd_1e-05/epoch_300/dp_0.0 --log-interval 200\n    ```\n    - Best architecture @1000 Queries\n    ```bash\n    cd pytorch-image-models;\n    bash distributed_train.sh $NUM_GPU $IMAGENET_PATH --model ofa_mbv3_1000 -b 128 \\\n    --sched cosine --img-size 236 --epochs 600 --warmup-epochs 3 --decay-rate .97 \\\n    --opt rmsproptf --opt-eps .001 -j 10 --warmup-lr 1e-6 --weight-decay 1e-05 --drop 0.3 \\\n    --drop-path 0.0 --model-ema --model-ema-decay 0.9999 --aa rand-m9-mstd0.5 \\\n    --remode pixel --reprob 0.2 --lr 1e-02 --output $LOG_PATH \\\n    --experiment res_236/bs_128/cosine/lr_5e-03/wd_1e-05/epoch_600/dp_0.0 --log-interval 200\n    ```\n\n  - Adapt to your run of best architecture founded by WeakNAS\n    - Modify line 24 - line 53 pytorch-image-models/timm/models/ofa_mbv3.py, add the configuration of architecture founded in search stage as \"ofa_mbv3_custom\" to default_cfgs\n    ```bash\n    cd pytorch-image-models;\n    bash distributed_train.sh $NUM_GPU $IMAGENET_PATH --model ofa_mbv3_custom -b 128 \\\n    --sched cosine --img-size 236 --epochs 600 --warmup-epochs 3 --decay-rate .97 \\\n    --opt rmsproptf --opt-eps .001 -j 10 --warmup-lr 1e-6 --weight-decay 1e-05 --drop 0.3 \\\n    --drop-path 0.0 --model-ema --model-ema-decay 0.9999 --aa rand-m9-mstd0.5 \\\n    --remode pixel --reprob 0.2 --lr 1e-02 --output $LOG_PATH \\\n    --experiment res_236/bs_128/cosine/lr_5e-03/wd_1e-05/epoch_600/dp_0.0 --log-interval 200\n    ```\n  Previous Tensorboard.dev Logs: [Link](https://tensorboard.dev/experiment/YuDEyzRSQpOQT7ZEZa8tNg/#scalars)\n\n\n## Acknowledgement\nNASBench Codebase from [AutoDL-Projects](https://github.com/D-X-Y/AutoDL-Projects)  \nImageNet Codebase from [timm](https://github.com/rwightman/pytorch-image-models)\n\n## Citation\nif you find this repo is helpful, please cite\n```\n@article{wu2021weak,\n  title={Stronger NAS with Weaker Predictors},\n  author={Junru Wu and Xiyang Dai and Dongdong Chen and Yinpeng Chen and Mengchen Liu and Ye Yu and Zhangyang Wang and Zicheng Liu and Mei Chen and Lu Yuan},\n  journal={arXiv preprint arXiv:2102.10490},\n  year={2021}\n}\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fvita-group%2Fweaknas","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fvita-group%2Fweaknas","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fvita-group%2Fweaknas/lists"}