{"id":13671187,"url":"https://github.com/LiYangHart/Hyperparameter-Optimization-of-Machine-Learning-Algorithms","last_synced_at":"2025-04-27T14:33:08.885Z","repository":{"id":37494945,"uuid":"281841206","full_name":"LiYangHart/Hyperparameter-Optimization-of-Machine-Learning-Algorithms","owner":"LiYangHart","description":"Implementation of hyperparameter optimization/tuning methods for machine learning \u0026 deep learning models (easy\u0026clear)","archived":false,"fork":false,"pushed_at":"2022-09-22T05:24:53.000Z","size":241,"stargazers_count":1287,"open_issues_count":0,"forks_count":303,"subscribers_count":88,"default_branch":"master","last_synced_at":"2025-04-08T20:18:17.501Z","etag":null,"topics":["artificial-neural-networks","automated-machine-learning","automl","bayesian-optimization","deep-learning","genetic-algorithm","grid-search","hpo","hyperparameter-optimization","hyperparameter-tuning","machine-learning","machine-learning-algorithms","machine-learning-optimization","optimization","particle-swarm-optimization","python-examples","python-samples","random-forest","random-search","tuning-parameters"],"latest_commit_sha":null,"homepage":"","language":"Jupyter Notebook","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/LiYangHart.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2020-07-23T03:30:47.000Z","updated_at":"2025-04-08T07:54:38.000Z","dependencies_parsed_at":"2023-01-17T14:45:55.340Z","dependency_job_id":null,"html_url":"https://github.com/LiYangHart/Hyperparameter-Optimization-of-Machine-Learning-Algorithms","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/LiYangHart%2FHyperparameter-Optimization-of-Machine-Learning-Algorithms","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/LiYangHart%2FHyperparameter-Optimization-of-Machine-Learning-Algorithms/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/LiYangHart%2FHyperparameter-Optimization-of-Machine-Learning-Algorithms/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/LiYangHart%2FHyperparameter-Optimization-of-Machine-Learning-Algorithms/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/LiYangHart","download_url":"https://codeload.github.com/LiYangHart/Hyperparameter-Optimization-of-Machine-Learning-Algorithms/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":251154345,"owners_count":21544480,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["artificial-neural-networks","automated-machine-learning","automl","bayesian-optimization","deep-learning","genetic-algorithm","grid-search","hpo","hyperparameter-optimization","hyperparameter-tuning","machine-learning","machine-learning-algorithms","machine-learning-optimization","optimization","particle-swarm-optimization","python-examples","python-samples","random-forest","random-search","tuning-parameters"],"created_at":"2024-08-02T09:01:02.362Z","updated_at":"2025-04-27T14:33:08.274Z","avatar_url":"https://github.com/LiYangHart.png","language":"Jupyter Notebook","readme":"# Hyperparameter Optimization of Machine Learning Algorithms\n\nThis code provides a hyper-parameter optimization implementation for machine learning algorithms, as described in the paper:  \nL. Yang and A. Shami, “[On hyperparameter optimization of machine learning algorithms: Theory and practice](https://arxiv.org/abs/2007.15745),” Neurocomputing, vol. 415, pp. 295–316, 2020, doi: https://doi.org/10.1016/j.neucom.2020.07.061.  \n\nTo fit a machine learning model into different problems, its hyper-parameters must be tuned. Selecting the best hyper-parameter configuration for machine learning models has a direct impact on the model's performance. In this paper, optimizing the hyper-parameters of common machine learning models is studied. We introduce several state-of-the-art optimization techniques and discuss how to apply them to machine learning algorithms. Many available libraries and frameworks developed for hyper-parameter optimization problems are provided, and some open challenges of hyper-parameter optimization research are also discussed in this paper. Moreover, experiments are conducted on benchmark datasets to compare the performance of different optimization methods and provide practical examples of hyper-parameter optimization.  \n\nThis paper and code will help industrial users, data analysts, and researchers to better develop machine learning models by identifying the proper hyper-parameter configurations effectively.\n\n- PS: A comprehensive **Automated Machine Learning (AutoML)** tutorial code can be found in: [AutoML-Implementation-for-Static-and-Dynamic-Data-Analytics](https://github.com/Western-OC2-Lab/AutoML-Implementation-for-Static-and-Dynamic-Data-Analytics)  \n   * Including **automated data pre-processing, automated feature engineering, automated model selection, hyperparameter optimization, and automated model updating** (concept drift adaptation).  \n\n## Paper\nOn Hyperparameter Optimization of Machine Learning Algorithms: Theory and Practice  \n[One-column version: arXiv](https://arxiv.org/abs/2007.15745)  \n[Two-column version: Elsevier](https://www.sciencedirect.com/science/article/pii/S0925231220311693)  \n### Quick Navigation\n**Section 3**: Important hyper-parameters of common machine learning algorithms  \n**Section 4**: Hyper-parameter optimization techniques introduction  \n**Section 5**: How to choose optimization techniques for different machine learning models  \n**Section 6**: Common Python libraries/tools for hyper-parameter optimization  \n**Section 7**: Experimental results (sample code in \"HPO_Regression.ipynb\" and \"HPO_Classification.ipynb\")  \n**Section 8**: Open challenges and future research directions  \n**Summary table for Sections 3-6**: Table 2:  A comprehensive overview of common ML models, their hyper-parameters, suitable optimization techniques, and available Python libraries  \n**Summary table for Sections 8**: Table 10:  The open challenges and future directions of HPO research  \n\n## Implementation\nSample code for hyper-parameter optimization implementation for machine learning algorithms is provided in this repository.  \n\n### Sample code for Regression problems  \n[HPO_Regression.ipynb](https://github.com/LiYangHart/Hyperparameter-Optimization-of-Machine-Learning-Algorithms/blob/master/HPO_Regression.ipynb)   \n**Dataset used:** [Boston-Housing](https://scikit-learn.org/stable/modules/generated/sklearn.datasets.load_boston.html)   \n### Sample code for Classification problems  \n[HPO_Classification.ipynb](https://github.com/LiYangHart/Hyperparameter-Optimization-of-Machine-Learning-Algorithms/blob/master/HPO_Classification.ipynb)   \n**Dataset used:** [MNIST](https://scikit-learn.org/stable/modules/generated/sklearn.datasets.load_digits.html#sklearn.datasets.load_digits)   \n\n### Machine Learning \u0026 Deep Learning Algorithms  \n* Random forest (RF)\n* Support vector machine (SVM)\n* K-nearest neighbor (KNN)  \n* Artificial Neural Networks (ANN)\n\n### Hyperparameter Configuration Space  \n|     ML Model          |     Hyper-parameter      |     Type           |     Search Space                            |\n|-----------------------|--------------------------|--------------------|---------------------------------------------|\n|     RF Classifier     |     n_estimators         |     Discrete       |     [10,100]                                |\n|                       |     max_depth            |     Discrete       |     [5,50]                                  |\n|                       |     min_samples_split    |     Discrete       |     [2,11]                                  |\n|                       |     min_samples_leaf     |     Discrete       |     [1,11]                                  |\n|                       |     criterion            |     Categorical    |     'gini', 'entropy'                     |\n|                       |     max_features         |     Discrete       |     [1,64]                                  |\n|     SVM Classifier    |     C                    |     Continuous     |     [0.1,50]                                |\n|                       |     kernel               |     Categorical    |     'linear', 'poly', 'rbf', 'sigmoid'    |\n|     KNN Classifier    |     n_neighbors          |     Discrete       |     [1,20]                                  |\n|     ANN Classifier    |     optimizer         |     Categorical       |     'adam', 'rmsprop', 'sgd'                                |\n|                       |     activation            |     Categorical       |     'relu', 'tanh'                                  |\n|                       |     batch_size    |     Discrete       |     [16,64]                                  |\n|                       |     neurons     |     Discrete       |     [10,100]                                  |\n|                       |     epochs            |     Discrete    |     [20,50]                     |\n|                       |     patience         |     Discrete       |     [3,20]                                  |\n|     RF Regressor      |     n_estimators         |     Discrete       |     [10,100]                                |\n|                       |     max_depth            |     Discrete       |     [5,50]                                  |\n|                       |     min_samples_split    |     Discrete       |     [2,11]                                  |\n|                       |     min_samples_leaf     |     Discrete       |     [1,11]                                  |\n|                       |     criterion            |     Categorical    |     'mse', 'mae'                          |\n|                       |     max_features         |     Discrete       |     [1,13]                                  |\n|     SVM Regressor     |     C                    |     Continuous     |     [0.1,50]                                |\n|                       |     kernel               |     Categorical    |     'linear', 'poly', 'rbf', 'sigmoid'    |\n|                       |     epsilon              |     Continuous     |     [0.001,1]                               |\n|     KNN Regressor     |     n_neighbors          |     Discrete       |     [1,20]                                  |\n|     ANN Regressor     |     optimizer         |     Categorical       |     'adam', 'rmsprop'                              |\n|                       |     activation            |     Categorical       |     'relu', 'tanh'                                  |\n|                       |     loss            |     Categorical       |     'mse', 'mae'                                  |\n|                       |     batch_size    |     Discrete       |     [16,64]                                  |\n|                       |     neurons     |     Discrete       |     [10,100]                                  |\n|                       |     epochs            |     Discrete    |     [20,50]                     |\n|                       |     patience         |     Discrete       |     [3,20]                                  |\n\n### HPO Algorithms  \n* Grid search\n* Random search\n* Hyperband\n* Bayesian Optimization with Gaussian Processes (BO-GP)\n* Bayesian Optimization with Tree-structured Parzen Estimator (BO-TPE)\n* Particle swarm optimization (PSO)\n* Genetic algorithm (GA)  \n\n### Requirements  \n* Python 3.5+ \n* [Keras](https://keras.io/) \n* [scikit-learn](https://scikit-learn.org/stable/)  \n* [hyperband](https://github.com/thuijskens/scikit-hyperband)  \n* [scikit-optimize](https://github.com/scikit-optimize/scikit-optimize)  \n* [hyperopt](https://github.com/hyperopt/hyperopt)  \n* [optunity](https://github.com/claesenm/optunity)  \n* [DEAP](https://github.com/rsteca/sklearn-deap)  \n* [TPOT](https://github.com/EpistasisLab/tpot)  \n\n## Contact-Info\nPlease feel free to contact me for any questions or cooperation opportunities. I'd be happy to help.\n* Email: [liyanghart@gmail.com](mailto:liyanghart@gmail.com)\n* GitHub: [LiYangHart](https://github.com/LiYangHart) and [Western OC2 Lab](https://github.com/Western-OC2-Lab/)\n* LinkedIn: [Li Yang](https://www.linkedin.com/in/li-yang-phd-65a190176/)  \n* Google Scholar: [Li Yang](https://scholar.google.com.eg/citations?user=XEfM7bIAAAAJ\u0026hl=en) and [OC2 Lab](https://scholar.google.com.eg/citations?user=oiebNboAAAAJ\u0026hl=en)\n\n## Citation\nIf you find this repository useful in your research, please cite this article as:  \n\nL. Yang and A. Shami, “On hyperparameter optimization of machine learning algorithms: Theory and practice,” *Neurocomputing*, vol. 415, pp. 295–316, 2020, doi: https://doi.org/10.1016/j.neucom.2020.07.061.\n\n```\n@article{YANG2020295,\ntitle = \"On hyperparameter optimization of machine learning algorithms: Theory and practice\",\nauthor = \"Li Yang and Abdallah Shami\",\nvolume = \"415\",\npages = \"295 - 316\",\njournal = \"Neurocomputing\",\nyear = \"2020\",\nissn = \"0925-2312\",\ndoi = \"https://doi.org/10.1016/j.neucom.2020.07.061\",\nurl = \"http://www.sciencedirect.com/science/article/pii/S0925231220311693\"\n}\n```\n","funding_links":[],"categories":["参数优化","Jupyter Notebook","Python","4.) Hyperparameter Optimization"],"sub_categories":["General-Purpose Machine Learning","**[Papers]**"],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FLiYangHart%2FHyperparameter-Optimization-of-Machine-Learning-Algorithms","html_url":"https://awesome.ecosyste.ms/projects/github.com%2FLiYangHart%2FHyperparameter-Optimization-of-Machine-Learning-Algorithms","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FLiYangHart%2FHyperparameter-Optimization-of-Machine-Learning-Algorithms/lists"}