{"id":13684871,"url":"https://github.com/DeepHiveMind/AutoML_AutoKeras_HPO","last_synced_at":"2025-05-01T00:33:59.724Z","repository":{"id":112362737,"uuid":"275619990","full_name":"DeepHiveMind/AutoML_AutoKeras_HPO","owner":"DeepHiveMind","description":":star: The Awesome world of Automated Machine Learning | The Next Gen of AI |","archived":false,"fork":false,"pushed_at":"2020-06-28T21:17:46.000Z","size":17241,"stargazers_count":6,"open_issues_count":0,"forks_count":1,"subscribers_count":2,"default_branch":"master","last_synced_at":"2024-08-03T14:08:07.269Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"https://deephivemind.io","language":null,"has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"bsd-3-clause","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/DeepHiveMind.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null}},"created_at":"2020-06-28T16:02:37.000Z","updated_at":"2024-01-07T07:47:58.000Z","dependencies_parsed_at":"2023-05-16T00:30:53.664Z","dependency_job_id":null,"html_url":"https://github.com/DeepHiveMind/AutoML_AutoKeras_HPO","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/DeepHiveMind%2FAutoML_AutoKeras_HPO","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/DeepHiveMind%2FAutoML_AutoKeras_HPO/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/DeepHiveMind%2FAutoML_AutoKeras_HPO/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/DeepHiveMind%2FAutoML_AutoKeras_HPO/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/DeepHiveMind","download_url":"https://codeload.github.com/DeepHiveMind/AutoML_AutoKeras_HPO/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":224230662,"owners_count":17277373,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-08-02T14:00:40.004Z","updated_at":"2024-11-12T06:30:52.266Z","avatar_url":"https://github.com/DeepHiveMind.png","language":null,"funding_links":[],"categories":["Other awesome AI lists","Libraries"],"sub_categories":[],"readme":"# The World of AutoML AutoKeras HPO\n\nThis repo **AutoML-AutoKeras-HPO** is a curated list of *automated machine learning* tehcniques and projects. **Star** this repository, and then you can keep abreast of the latest developments of this booming research field which I would keep updating it. Thanks to all the people who made contributions to this project. Join us and you are welcome to be a contributor.\n\n# Agenda \n- [Introduction to AutoML](#Introduction-to-AutoML)\n- [High Level technique needed in AutoML](#High-Level-technique-needed-in-AutoML)\n- [Comparative view of Commercial AutoML Offerings](#Comparative-view-of-Commercial-AutoML-Offerings)\n- [Comparative view of AutoML Projects TYPES NAS HPO AutoFE](#Comparative-view-of-AutoML-Projects-TYPES-NAS-HPO-AutoFE)\n- [Detailed Insight into the constructs of AutoML ecosystems](#Detailed-Insight-into-the-constructs-of-AutoML-ecosystems)\n```\n- Automated Feature Engineering\n  - [Expand Reduce]\n  - [Hierarchical Organization of Transformations]\n  - [Meta Learning]\n  - [Reinforcement Learning]\n- Architecture Search\n  - [Evolutionary Algorithms]\n  - [Local Search]\n  - [Meta Learning]\n  - [Reinforcement Learning]\n  - [Transfer Learning]\n- Hyperparameter Optimization\n  - [Bayesian Optimization]\n  - [Evolutionary Algorithms]\n  - [Lipschitz Functions]\n  - [Local Search]\n  - [Meta Learning]\n  - [Particle Swarm Optimization]\n  - [Random Search]\n  - [Transfer Learning]\n  ```\n\n# Introduction to AutoML\n*Automated Machine Learning* (AutoML) provides methods and processes to make Machine Learning available for non-Machine Learning experts, to improve efficiency of Machine Learning and to accelerate research on Machine Learning.\n\nMachine Learning (ML) has achieved considerable successes in recent years and an ever-growing number of disciplines rely on it. However, this success crucially relies on human machine learning experts to perform the following tasks:\n+ Preprocess the data,\n+ Select appropriate features,\n+ Select an appropriate model family,\n+ Optimize model hyperparameters,\n+ Postprocess machine learning models,\n+ Critically analyze the results obtained.\n\nAs the complexity of these tasks is often beyond non-ML-experts, the rapid growth of machine learning applications has created a demand for off-the-shelf machine learning methods that can be used easily and without expert knowledge. We call the resulting research area that targets progressive automation of machine learning *AutoML*. As a new sub-area in machine learning, *AutoML* has got more attention not only in machine learning but also in computer vision, natural language processing and graph computing.\n\nThere are no formal definition of *AutoML*. From the descriptions of most papers，the basic procedure of *AutoML* can be shown as the following.\n\n\u003cdiv style=\"text-align: center\"\u003e\n\u003cimg src=\"resources/procedure.jpg\" width=\"600px\" atl=\"figure1\"/\u003e\n\u003c/div\u003e\n\n\n# High Level technique needed in AutoML\n\nThis Repo includes very up-to-date overviews of the bread-and-butter techniques we need in *AutoML*:\n+ Automated Data Clean (Auto Clean)\n+ Automated Feature Engineering (Auto FE)\n   - Automated Feature Engineering for Predictive Modeling |  [`Download`](https://github.com/hibayesian/awesome-automl-papers/blob/master/resources/slides/%5Bslides%5D-automated-feature-engineering-for-predictive-modeling.pdf) |\n+ Hyperparameter Optimization (HPO)\n   - Bayesian Optimization for Hyperparameter Tuning | [`Link`](https://arimo.com/data-science/2016/bayesian-optimization-hyperparameter-tuning/) | [`Download`](https://github.com/hibayesian/awesome-automl-papers/blob/master/resources/slides/%5Bslides%5D-a-tutorial-on-bayesian-optimization-for-machine-learning.pdf) |\n+ Meta-Learning\n   -  Learning to learn | [`Link`](http://bair.berkeley.edu/blog/2017/07/18/learning-to-learn/) |\n   - Why Meta-learning is Crucial for Further Advances of Artificial Intelligence? | [`Link`](https://chatbotslife.com/why-meta-learning-is-crucial-for-further-advances-of-artificial-intelligence-c2df55959adf) |\n+ Neural Architecture Search (NAS)\n\n\n\u003cdiv style=\"text-align: center\"\u003e\n\u003cimg src=\"resources/automl.png\" atl=\"automl\"/\u003e\n\u003c/div\u003e\n\n\n# Comparative view of Commercial AutoML Offerings\n\n - *AutoML* approaches are already mature enough to rival and sometimes even outperform human machine learning experts. Put simply, *AutoML* can lead to improved performance while saving substantial amounts of time and money, as machine learning experts are both hard to find and expensive. \n - As a result, commercial interest in *AutoML* has grown dramatically in recent years, and several major tech companies and start-up companies are now developing their own *AutoML* systems. An overview comparison of some of them can be summarized to the following table.\n\n| Company    | AutoFE     | HPO        | NAS        |\n| :--------: | :--------: | :--------: | :--------: |\n| Google     |     √      |      √     |      √     |\n| H2O.ai     |     √      |      √     |      ×     |\n| Microsoft  |     ×      |      √     |      √     |\n| Alibaba    |     ×      |      √     |      ×     |\n| Baidu      |     ×      |      ×     |      √     |\n| RapidMiner |     √      |      √     |      ×     |\n\n\n\n\n\n# Comparative view of AutoML Projects TYPES NAS HPO AutoFE\n| Project    | Type       | Language   | License    | Link       |\n| :--------: | :--------: | :--------: | :--------: | :--------: |\n| AdaNet     | NAS        | Python     | Apache-2.0 | [`Github`](https://github.com/tensorflow/adanet) |\n| Advisor    | HPO        | Python     | Apache-2.0 | [`Github`](https://github.com/tobegit3hub/advisor) |\n| AMLA       | HPO, NAS   | Python     | Apache-2.0 | [`Github`](https://github.com/CiscoAI/amla) |\n| ATM        | HPO        | Python     | MIT        | [`Github`](https://github.com/HDI-Project/ATM) |\n| Auger      | HPO        | Python     | Commercial | [`Homepage`](https://auger.ai) |\n| auptimizer | HPO, NAS   | Python (support R script) | GPL-3.0  | [`Github`](https://github.com/LGE-ARC-AdvancedAI/auptimizer) |\n| Auto-Keras | NAS        | Python     | [`License`](https://github.com/keras-team/autokeras/blob/master/LICENSE.txt) | [`Github`](https://github.com/keras-team/autokeras) |\n| AutoML Vision | NAS     | Python     | Commercial | [`Homepage`](https://cloud.google.com/vision/) |\n| AutoML Video Intelligence            | NAS        | Python    | Commercial | [`Homepage`](https://cloud.google.com/video-intelligence/) |\n| AutoML Natural Language | NAS        | Python     | Commercial | [`Homepage`](https://cloud.google.com/natural-language/) |\n| AutoML Translation      | NAS        | Python     | Commercial | [`Homepage`](https://cloud.google.com/translate/) |\n| AutoML Tables           | AutoFE, HPO             | Python    | Commercial | [`Homepage`](https://cloud.google.com/automl-tables/) |\n| auto-sklearn | HPO      | Python     | [`License`](https://github.com/automl/auto-sklearn/blob/master/LICENSE.txt) | [`Github`](https://github.com/automl/auto-sklearn) |\n| auto_ml     | HPO       | Python     | MIT        | [`Github`](https://github.com/ClimbsRocks/auto_ml) |\n| BayesianOptimization | HPO | Python  | MIT        | [`Github`](https://github.com/fmfn/BayesianOptimization) |\n| BayesOpt    | HPO       | C++        | AGPL-3.0   | [`Github`](https://github.com/rmcantin/bayesopt) |\n| comet       | HPO       | Python     | Commercial | [`Homepage`](https://www.comet.ml) |\n| DataRobot   | HPO       | Python     | Commercial | [`Homepage`](https://www.datarobot.com/) |\n| DEvol       | NAS       | Python     | MIT        | [`Github`](https://github.com/joeddav/devol) |\n| DeepArchitect | NAS     | Python     | MIT        | [`Github`](https://github.com/negrinho/deep_architect) |\n| Driverless AI | AutoFE  | Python     | Commercial | [`Homepage`](https://www.h2o.ai/products/h2o-driverless-ai/) |\n| FAR-HO      | HPO       | Python     | MIT        | [`Github`](https://github.com/lucfra/FAR-HO) |\n| H2O AutoML  | HPO       | Python, R, Java, Scala | Apache-2.0 | [`Github`](https://github.com/h2oai/h2o-3/) |\n| HpBandSter  | HPO       | Python     | BSD-3-Clause | [`Github`](https://github.com/automl/HpBandSter) |\n| HyperBand   | HPO       | Python     | [`License`](https://github.com/zygmuntz/hyperband/blob/master/LICENSE) | [`Github`](https://github.com/zygmuntz/hyperband) |\n| Hyperopt    | HPO       | Python     | [`License`](https://github.com/hyperopt/hyperopt/blob/master/LICENSE.txt) | [`Github`](https://github.com/hyperopt/hyperopt) |\n| Hyperopt-sklearn | HPO  | Python    | [`License`](https://github.com/hyperopt/hyperopt-sklearn/blob/master/LICENSE.txt) | [`Github`](https://github.com/hyperopt/hyperopt-sklearn) |\n| Hyperparameter Hunter | HPO | Python | MIT        | [`Github`](https://github.com/HunterMcGushion/hyperparameter_hunter) |\n| Katib       | HPO       | Python     | Apache-2.0 | [`Github`](https://github.com/kubeflow/katib) |\n| MateLabs    | HPO       | Python     | Commercial | [`Github`](http://matelabs.in/) |\n| Milano      | HPO       | Python     | Apache-2.0 | [`Github`](https://github.com/NVIDIA/Milano) |\n| MLJAR       | HPO       | Python     | Commercial | [`Homepage`](https://mljar.com/) |\n| nasbot      | NAS       | Python     | MIT        | [`Github`](https://github.com/kirthevasank/nasbot) |\n| neptune     | HPO       | Python     | Commercial | [`Homepage`](https://neptune.ml/) |\n| NNI         | HPO, NAS  | Python     | MIT        | [`Github`](https://github.com/Microsoft/nni) |\n| Oboe    | HPO       | Python     | BSD-3-Clause | [`Github`](https://github.com/udellgroup/oboe) |\n| Optunity    | HPO       | Python     | [`License`](https://github.com/claesenm/optunity/blob/master/LICENSE.txt) | [`Github`](https://github.com/claesenm/optunity) |\n| R2.ai       | HPO       |            | Commercial | [`Homepage`](https://r2.ai/) |\n| RBFOpt      | HPO       | Python     | [`License`](https://github.com/coin-or/rbfopt/blob/master/LICENSE) | [`Github`](https://github.com/coin-or/rbfopt) |\n| RoBO        | HPO       | Python     | BSD-3-Clause | [`Github`](https://github.com/automl/RoBO) |\n| Scikit-Optimize | HPO   | Python     | [`License`](https://github.com/scikit-optimize/scikit-optimize/blob/master/LICENSE.md) | [`Github`](https://github.com/scikit-optimize/scikit-optimize) |\n| SigOpt      | HPO       | Python     | Commercial | [`Homepage`](https://sigopt.com/) |\n| SMAC3       | HPO       | Python     | [`License`](https://github.com/automl/SMAC3/blob/master/LICENSE) | [`Github`](https://github.com/automl/SMAC3) |\n| TPOT        | AutoFE, HPO | Python   | LGPL-3.0   | [`Github`](https://github.com/rhiever/tpot) |\n| TransmogrifAI | HPO     | Scala      | BSD-3-Clause | [`Github`](https://github.com/salesforce/TransmogrifAI) |\n| Tune        | HPO       | Python     | Apache-2.0 | [`Github`](https://github.com/ray-project/ray/tree/master/python/ray/tune) |\n| Xcessiv     | HPO       | Python     | Apache-2.0 | [`Github`](https://github.com/reiinakano/xcessiv) |\n| SmartML     | HPO       | R          | GPL-3.0    | [`Github`](https://github.com/DataSystemsGroupUT/SmartML) |\n| MLBox     | AutoFE, HPO    | Python          | BSD-3 License    | [`Github`](https://github.com/AxeldeRomblay/MLBox) |\n| AutoAI Watson     | AutoFE, HPO    |      | Commercial    | [`Homepage`](https://www.ibm.com/cloud/watson-studio/autoai) |\n\n\n# Detailed Insight into the constructs of AutoML ecosystems\n\n- [Surveys](#surveys)\n- [Automated Feature Engineering](#automated-feature-engineering)\n  - [Expand Reduce](#expand-reduce)\n  - [Hierarchical Organization of Transformations](#hierarchical-organization-of-transformations)\n  - [Meta Learning](#meta-learning)\n  - [Reinforcement Learning](#reinforcement-learning)\n- [Architecture Search](#architecture-search)\n  - [Evolutionary Algorithms](#evolutionary-algorithms)\n  - [Local Search](#local-search)\n  - [Meta Learning](#meta-learning)\n  - [Reinforcement Learning](#reinforcement-learning)\n  - [Transfer Learning](#transfer-learning)\n- [Hyperparameter Optimization](#hyperparameter-optimization)\n  - [Bayesian Optimization](#bayesian-optimization)\n  - [Evolutionary Algorithms](#evolutionary-algorithms)\n  - [Lipschitz Functions](#lipschitz-functions)\n  - [Local Search](#local-search)\n  - [Meta Learning](#meta-learning)\n  - [Particle Swarm Optimization](#particle-swarm-optimization)\n  - [Random Search](#random-search)\n  - [Transfer Learning](#transfer-learning)\n\n\n\n\n\n### Surveys\n+ 2019 | AutoML: A Survey of the State-of-the-Art |arXiv | [`PDF`](https://arxiv.org/pdf/1908.00709.pdf)\n+ 2019 | Survey on Automated Machine Learning | arXiv | [`PDF`](https://arxiv.org/pdf/1904.12054.pdf)\n+ 2019 | Automated Machine Learning: State-of-The-Art and Open Challenges |  arXiv | [`PDF`](https://arxiv.org/pdf/1906.02287.pdf)\n\n### Automated Feature Engineering\n+ #### Expand Reduce\n  - 2017 | AutoLearn — Automated Feature Generation and Selection |  [`PDF`](https://ieeexplore.ieee.org/document/8215494/)\n  - 2016 | Automating Feature Engineering | Udayan Khurana, et al. | NIPS | [`PDF`](http://workshops.inf.ed.ac.uk/nips2016-ai4datasci/papers/NIPS2016-AI4DataSci_paper_13.pdf)\n  - 2016 | ExploreKit: Automatic Feature Generation and Selection |[`PDF`](http://ieeexplore.ieee.org/document/7837936/)\n  - 2015 | Deep Feature Synthesis: Towards Automating Data Science Endeavors | [`PDF`](http://www.jmaxkanter.com/static/papers/DSAA_DSM_2015.pdf)\n+ #### Hierarchical Organization of Transformations\n  - 2016 | Cognito: Automated Feature Engineering for Supervised Learning | ICDMW | [`PDF`](http://ieeexplore.ieee.org/document/7836821/)\n+ #### Meta Learning\n  - 2017 | Learning Feature Engineering for Classification |  [`PDF`](https://www.ijcai.org/proceedings/2017/0352.pdf)\n+ #### Reinforcement Learning\n  - 2017 | Feature Engineering for Predictive Modeling using Reinforcement Learning | Udayan Khurana, et al. | arXiv | [`PDF`](https://arxiv.org/pdf/1709.07150.pdf)\n\n### Architecture Search\n+ #### Evolutionary Algorithms\n  - 2019 | Evolutionary Neural AutoML for Deep Learning | GECCO | [`PDF`](https://dl.acm.org/doi/pdf/10.1145/3321707.3321721)\n  - 2017 | Large-Scale Evolution of Image Classifiers | PMLR | [`PDF`](https://arxiv.org/abs/1703.01041)\n\n+ #### Local Search\n  - 2017 | Simple and Efficient Architecture Search for Convolutional Neural Networks | ICLR | [`PDF`](https://arxiv.org/pdf/1711.04528.pdf)\n  \n+ #### Meta Learning\n  - 2016 | Learning to Optimize | arXiv | [`PDF`](https://arxiv.org/pdf/1606.01885.pdf)\n  \n+ #### Reinforcement Learning\n  - 2018 | AMC: AutoML for Model Compression and Acceleration on Mobile Devices | [`PDF`](http://openaccess.thecvf.com/content_ECCV_2018/papers/Yihui_He_AMC_Automated_Model_ECCV_2018_paper.pdf)\n  - 2018 | Efficient Neural Architecture Search via Parameter Sharing |arXiv | [`PDF`](https://arxiv.org/abs/1802.03268)\n  - 2017 | Neural Architecture Search with Reinforcement Learning |ICLR | [`PDF`](https://arxiv.org/pdf/1611.01578.pdf)\n  \n+ #### Transfer Learning\n  - 2017 | Learning Transferable Architectures for Scalable Image Recognition |  arXiv | [`PDF`](https://arxiv.org/abs/1707.07012)\n  \n+ #### Network Morphism\n  - 2018 | Efficient Neural Architecture Search with Network Morphism | arXiv | [`PDF`](https://arxiv.org/abs/1806.10282)\n  \n+ #### Continuous Optimization\n  - 2018 | Neural Architecture Optimization | arXiv | [`PDF`](https://arxiv.org/abs/1808.07233)\n  - 2019 | DARTS: Differentiable Architecture Search ICLR | [`PDF`](https://arxiv.org/abs/1806.09055)\n\n### Hyperparameter Optimization\n+ #### Bayesian Optimization\n  - 2019 | Bayesian Optimization with Unknown Search Space  | NeurIPS | [`PDF`](http://papers.nips.cc/paper/9350-bayesian-optimization-with-unknown-search-space.pdf)\n  - 2019 | Constrained Bayesian optimization with noisy experiments | [`PDF`](https://projecteuclid.org/download/pdfview_1/euclid.ba/1533866666)\n  \n  - 2018 | Scalable hyperparameter transfer learning | NeurIPS | [`PDF`](http://papers.nips.cc/paper/7917-scalable-hyperparameter-transfer-learning.pdf)\n\n  - 2013 | Auto-WEKA: Combined Selection and Hyperparameter Optimization of Classification Algorithms | [`PDF`](http://www.cs.ubc.ca/labs/beta/Projects/autoweka/papers/autoweka.pdf)\n \n+ #### Evolutionary Algorithms\n  - 2018 | Autostacker: A Compositional Evolutionary Learning System  | arXiv | [`PDF`](https://arxiv.org/pdf/1803.00684.pdf)\n  - 2017 | Large-Scale Evolution of Image Classifiers | Esteban Real, et al. | PMLR | [`PDF`](https://arxiv.org/pdf/1703.01041.pdf)\n\n+ ####  Lipschitz Functions\n  - 2017 | Global Optimization of Lipschitz functions | arXiv | [`PDF`](https://arxiv.org/pdf/1703.02628.pdf)\n\n+ #### Local Search\n \n+ #### Meta Learning\n  - 2008 | Cross-Disciplinary Perspectives on Meta-Learning for Algorithm Selection | [`PDF`](https://dl.acm.org/citation.cfm?id=1456656)\n  - 2019 | SMARTML: A Meta Learning-Based Framework for Automated Selection and Hyperparameter Tuning for Machine Learning Algorithms | [`PDF`](http://openproceedings.org/2019/conf/edbt/EDBT19_paper_235.pdf)\n\n+ #### Particle Swarm Optimization\n \n  - 2008 | Particle Swarm Optimization for Parameter Determination and Feature Selection of Support Vector Machines  | Expert Systems with Applications | [`PDF`](http://www.sciencedirect.com/science/article/pii/S0957417407003752)\n+ #### Random Search\n  - 2016 | Hyperband: A Novel Bandit-Based Approach to Hyperparameter Optimization | arXiv |  [`PDF`](https://arxiv.org/pdf/1603.06560.pdf)\n \n+ #### Transfer Learning\n  - 2016 | Efficient Transfer Learning Method for Automatic Hyperparameter Tuning | [`PDF`](https://pdfs.semanticscholar.org/75f2/6734972ebaffc6b43d45abd3048ef75f15a5.pdf)\n  \n\n# Reference Blog\n- AutoML\n   - Methods, Systems, Challenges | [`Download`](https://www.automl.org/book/) |\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FDeepHiveMind%2FAutoML_AutoKeras_HPO","html_url":"https://awesome.ecosyste.ms/projects/github.com%2FDeepHiveMind%2FAutoML_AutoKeras_HPO","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FDeepHiveMind%2FAutoML_AutoKeras_HPO/lists"}