https://github.com/rrwen/assign-rmcs-litreview
Assignment titled "A Brief Review of Hyperparameter Optimization Methods for Machine Learning" for Research Methods in Computer Science course at Ryerson University
https://github.com/rrwen/assign-rmcs-litreview
automated automatic hyperparameter learning literature machine optimisation optimization parameter review survey
Last synced: 25 days ago
JSON representation
Assignment titled "A Brief Review of Hyperparameter Optimization Methods for Machine Learning" for Research Methods in Computer Science course at Ryerson University
- Host: GitHub
- URL: https://github.com/rrwen/assign-rmcs-litreview
- Owner: rrwen
- License: mit
- Created: 2017-11-03T04:16:57.000Z (over 7 years ago)
- Default Branch: master
- Last Pushed: 2017-11-27T03:00:25.000Z (over 7 years ago)
- Last Synced: 2025-02-08T21:11:19.837Z (3 months ago)
- Topics: automated, automatic, hyperparameter, learning, literature, machine, optimisation, optimization, parameter, review, survey
- Language: TeX
- Homepage: https://github.com/rrwen/assign-rmcs-litreview/blob/master/wen2017_reviewparamoptml.pdf
- Size: 2.3 MB
- Stars: 0
- Watchers: 2
- Forks: 1
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
- License: LICENSE
Awesome Lists containing this project
README
# A Brief Review of Hyperparameter Optimization Methods for Machine Learning
Richard Wen
[email protected]
November 02, 2017
Assignment for Research Methods in Computer Science Course
Instructed by Dr. Cherie Ding
- [PDF](https://github.com/rrwen/assign-rmcs-litreview/blob/master/wen2017_reviewparamoptml.pdf)
- [Revisions](https://github.com/rrwen/assign-rmcs-litreview/blob/master/revisions.md)
- [Presentation](https://github.com/rrwen/slides-rmcs-litreview)
## Abstract
In this review, five research papers from 2015 to 2017 were selected from the ACM and IEEE Xplore digital libraries by using journal citation reports, search queries, and manual selection criteria. These five papers were summarized for a range of commonly used hyperparameter optimization methods and their performance in experimental settings. It was found that the most effective hyperparameter optimization methods used the Sequential Model Based Optimization (SMBO) framework with hyperparameter initialization techniques and incorporation of conditional spaces. However, hyperparameter optimization techniques are heavily reliant on the machine learning algorithms and datasets used, which leads to issues in dimensionality and inconsistent results. Proposed future work included implementing improved sampling techniques, and combining machine learning models with hyperparameter optimization methods for automated machine learning.
## References
* N. Schilling, M. Wistuba, L. Drumond, and L. Schmidt-Thieme, “Joint model choice and hyperparameter optimization with factorized multilayer perceptrons,” in 2015 IEEE 27th International Conference on Tools with Artificial Intelligence (ICTAI), Nov 2015, pp. 72–79.
* M. Wistuba, N. Schilling, and L. Schmidt-Thieme, “Learning hyperparameter optimization initializations,” in 2015 IEEE International Conference on Data Science and Advanced Analytics (DSAA), Oct 2015, pp. 1–10.
* M. Wistuba, N. Schilling, and L. Schmidt-Thieme, “Hyperparameter optimization machines,” in 2016 IEEE International Conference on Data Science and Advanced Analytics (DSAA), Oct 2016, pp. 41–50.
* J. C. L\`evesque, A. Durand, C. Gagn\`e, and R. Sabourin, “Bayesian optimization for conditional hyperparameter spaces,” in 2017 International Joint Conference on Neural Networks (IJCNN), May 2017, pp. 286–293.
* A. Quitadamo, J. Johnson, and X. Shi, “Bayesian hyperparameter optimization for machine learning based eqtl analysis,” in Proceedings of the 8th ACM International Conference on Bioinformatics, Computational Biology,and Health Informatics, ser. ACM-BCB ’17. New York, NY, USA: ACM, 2017, pp. 98–106. [Online]. Available: http://doi.acm.org/10.1145/3107411.3107434
## Developer Notes
1. Install [TeX Live](https://www.tug.org/texlive/acquire-netinstall.html) or [Miktex](https://miktex.org/download)
2. Add bin paths to system or environmental paths
3. [Clone](https://git-scm.com/docs/git-clone) this repository
4. Generate **wen2017_reviewparamoptml.pdf** (see [make.bat](https://github.com/rrwen/assign-rmcs-litreview/blob/master/make.bat) or [make.sh](https://github.com/rrwen/assign-rmcs-litreview/blob/master/make.sh))
In windows:
```
git clone https://github.com/rrwen/assign-rmcs-litreview
cd assign-rmcs-litreview
make
```
In Linux/Mac:
```
git clone https://github.com/rrwen/assign-rmcs-litreview
cd assign-rmcs-litreview
./make.sh
```