{"id":15425908,"url":"https://github.com/ultmaster/neuralpredictor.pytorch","last_synced_at":"2025-04-19T15:33:57.272Z","repository":{"id":83764887,"uuid":"236416111","full_name":"ultmaster/neuralpredictor.pytorch","owner":"ultmaster","description":"Open source reproduction in PyTorch of \"Neural Predictor for Neural Architecture Search\".","archived":false,"fork":false,"pushed_at":"2020-09-05T16:47:50.000Z","size":137,"stargazers_count":39,"open_issues_count":3,"forks_count":9,"subscribers_count":3,"default_branch":"master","last_synced_at":"2025-03-29T09:23:17.088Z","etag":null,"topics":["automl","neural-architecture-search"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/ultmaster.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2020-01-27T03:55:42.000Z","updated_at":"2025-03-27T16:08:37.000Z","dependencies_parsed_at":null,"dependency_job_id":"49975c60-8877-4fad-89f0-28e589fd72b4","html_url":"https://github.com/ultmaster/neuralpredictor.pytorch","commit_stats":{"total_commits":3,"total_committers":3,"mean_commits":1.0,"dds":0.6666666666666667,"last_synced_commit":"1d6d643236d3b0e23304758025029be7807c00b0"},"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ultmaster%2Fneuralpredictor.pytorch","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ultmaster%2Fneuralpredictor.pytorch/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ultmaster%2Fneuralpredictor.pytorch/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ultmaster%2Fneuralpredictor.pytorch/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/ultmaster","download_url":"https://codeload.github.com/ultmaster/neuralpredictor.pytorch/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":249220841,"owners_count":21232421,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["automl","neural-architecture-search"],"created_at":"2024-10-01T17:54:07.003Z","updated_at":"2025-04-16T08:33:00.899Z","avatar_url":"https://github.com/ultmaster.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Neural Predictor for Neural Architecture Search\n\nWei Wen, Hanxiao Liu, Hai Li, Yiran Chen, Gabriel Bender, Pieter-Jan Kindermans. \"Neural Predictor for Neural Architecture Search\". arXiv:1912.00848. [Paper link](https://arxiv.org/abs/1912.00848).\n\n**This is a open source reproduction in PyTorch.**\n\n## Reproduction Results\n\n![](assets/scatterplot.png)\n\nAll the results are run with the hyper-parameters provided in paper (default value in `train.py`), unless otherwise specified.\n\nThe following results are MSE. The lower, the better.\n\n| Train Split | Eval Split | Paper | Reproduction | Comments                |\n|-------------|------------|-------|--------------|-------------------------|\n| 172         | all        | 1.95  | 3.62         |                         |\n| 860         | all        | NA    | 2.94         |                         |\n| 172         | denoise-80 | NA    | 1.90         |                         |\n| 91-172      | denoise-91 | 0.66  | 0.74         | Paper used classifier to denoise |\n| 91-172      | denoise-91 | NA    | 0.56         | epochs = 600, lr = 2e-4 |\n\nNOTE: As the classifier is not ready, we cheated a little by directly filtering out all the architectures below 91%. The splits are called `91-`.\n\n## TODO Items\n\n- [ ] Classifier (first stage)\n- [ ] Cross validation\n- [ ] E2E Architecture Selection\n\n## Preparation\n\n### Dependencies\n\n* PyTorch (cuda)\n* [NasBench](https://github.com/google-research/nasbench/tree/master/nasbench)\n* h5py\n* matplotlib\n\n### Dataset\n\nDownload HDF5 version of NasBench from [here](https://drive.google.com/open?id=1x1EQCyClzHBVDHloUCtES_-M_E9o4MeF) and put it under `data`.\n\nThen generate train/eval split:\n\n```\npython tools/split_train_val.py\n```\n\n### Advanced: Build HDF5 Dataset from Scratch\n\nSkip this step if you have downloaded the data from last step.\n\nThis step is to convert the tfrecord into a hdf5 file, as the official asset Google has provided is too slow to read (and very large in volume).\n\nDownload `nasbench_full.tfrecord` from NasBench, and put it under `data`. Then run\n\n```\npython tools/nasbench_tfrecord_converter.py\n```\n\n### Splits\n\nThe following splits are provided for now:\n\n* `172`, `334`, `860`: Randomly sampled architectures from NasBench.\n* `91-172`, `91-334`, `91-860`: The splits above filtered with a threshold (validation accuracy 91% on seed 0).\n* `denoise-91`, `denoise-80`: All architectures filtered with threshold 91% and 80%.\n* `all`.\n\n## Train and Evaluation\n\nRefer to `python train.py -h` for options. Training and evaluation are very fast (about 90 seconds on P100).\n\n## Implementation Details\n\n### HDF5 Format\n\nThe HDF5 is quite self-explanatory. You can refer to `dataset.py` for how to read it. The only thing I believe should be highlighted is that `metrics` is a 423624 x 4 (epochs: 4, 12, 36, 108) x 3 (seed: 0, 1, 2) x 2 (halfway, total) x 4 (`training_time`, `train_accuracy`, `validation_accuracy`, `test_accuracy`) matrix.\n\n### Modeling and Training\n\n* The paper didn't mention where to put dropout. So dropout is added after every layer. Nevertheless, the model still tends to overfit.\n* Uses xavier uniform for initialization. Bias for linear layer is turned off.\n* Paper didn't mention hyper-parameters other than lr and weight decay in Adam. We follow default settings in tensorflow 1.15.\n* We drop the last samples (less than batch size) in every epoch.\n* We normalize the labels (validation accuracy) with MEAN (90.8) and STD (2.4).\n* Resample with different seed in case the validation accuracy belows (\u003c 15%).\n\n### Bad results for evaluation on \"all\"\n\nA brief case study reveals that the bad results are mainly due to the \"noise\" in NasBench. In NasBench, there are two types of noises:\n\n1. Some training (0.63%) \"blows\". The results are about ~10%. We resample on such case. We can handle 99.85% (422979/423624), with others using mean accuracy directly.\n2. The result diversifies. About 1.2% of the architectures get accuracy lower than 80%, but they contribute a lot to MSE. We found that without sampling them in testing, the results improve and are on par with paper.\n\n## References\n\n- [graph-cnn.pytorch](https://github.com/meliketoy/graph-cnn.pytorch)\n- [pygcn](https://github.com/tkipf/pygcn)\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fultmaster%2Fneuralpredictor.pytorch","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fultmaster%2Fneuralpredictor.pytorch","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fultmaster%2Fneuralpredictor.pytorch/lists"}