{"id":13577983,"url":"https://github.com/SimonBlanke/Gradient-Free-Optimizers","last_synced_at":"2025-04-05T15:31:59.982Z","repository":{"id":39372352,"uuid":"252927263","full_name":"SimonBlanke/Gradient-Free-Optimizers","owner":"SimonBlanke","description":"Simple and reliable optimization with local, global, population-based and sequential techniques in numerical discrete search spaces.","archived":false,"fork":false,"pushed_at":"2025-03-31T16:47:29.000Z","size":40138,"stargazers_count":1222,"open_issues_count":15,"forks_count":87,"subscribers_count":21,"default_branch":"master","last_synced_at":"2025-04-03T11:45:19.391Z","etag":null,"topics":["bayesian-optimization","blackbox-optimization","constrained-optimization","evolution-strategies","gradient-free-optimization","hill-climbing","hyperactive","hyperparameter-optimization","machine-learning","meta-heuristic","nelder-mead","optimization","particle-swarm-optimization","random-search","simulated-annealing","tree-of-parzen-estimator"],"latest_commit_sha":null,"homepage":"https://simonblanke.github.io/gradient-free-optimizers-documentation","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/SimonBlanke.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":"CODE_OF_CONDUCT.md","threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2020-04-04T06:35:55.000Z","updated_at":"2025-03-31T16:47:33.000Z","dependencies_parsed_at":"2023-02-03T20:16:26.489Z","dependency_job_id":"73918417-08e2-4d2a-8388-caaec1c63be3","html_url":"https://github.com/SimonBlanke/Gradient-Free-Optimizers","commit_stats":{"total_commits":879,"total_committers":7,"mean_commits":"125.57142857142857","dds":"0.014789533560864654","last_synced_commit":"89eb9eabaea7e8afe8ef0244366b9b6d3c7a0e84"},"previous_names":[],"tags_count":7,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/SimonBlanke%2FGradient-Free-Optimizers","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/SimonBlanke%2FGradient-Free-Optimizers/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/SimonBlanke%2FGradient-Free-Optimizers/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/SimonBlanke%2FGradient-Free-Optimizers/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/SimonBlanke","download_url":"https://codeload.github.com/SimonBlanke/Gradient-Free-Optimizers/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":247359112,"owners_count":20926369,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["bayesian-optimization","blackbox-optimization","constrained-optimization","evolution-strategies","gradient-free-optimization","hill-climbing","hyperactive","hyperparameter-optimization","machine-learning","meta-heuristic","nelder-mead","optimization","particle-swarm-optimization","random-search","simulated-annealing","tree-of-parzen-estimator"],"created_at":"2024-08-01T15:01:26.022Z","updated_at":"2025-04-05T15:31:59.975Z","avatar_url":"https://github.com/SimonBlanke.png","language":"Python","funding_links":[],"categories":["Python","其他_机器学习与深度学习"],"sub_categories":[],"readme":"\u003cp align=\"center\"\u003e\n  \u003cbr\u003e\n  \u003ca href=\"https://github.com/SimonBlanke/Gradient-Free-Optimizers\"\u003e\u003cimg src=\"./docs/images/gradient_logo_ink.png\" height=\"280\"\u003e\u003c/a\u003e\n  \u003cbr\u003e\n\u003c/p\u003e\n\n\u003cbr\u003e\n\n---\n\n\n\n\u003ch2 align=\"center\"\u003e\n  Simple and reliable optimization with local, global, population-based and sequential techniques in numerical discrete search spaces.\n\u003c/h2\u003e\n\n\u003cbr\u003e\n\n\u003ctable\u003e\n  \u003ctbody\u003e\n    \u003ctr align=\"left\" valign=\"center\"\u003e\n      \u003ctd\u003e\n        \u003cstrong\u003eMaster status:\u003c/strong\u003e\n      \u003c/td\u003e\n      \u003ctd\u003e\n        \u003ca href=\"https://github.com/SimonBlanke/Gradient-Free-Optimizers/actions\"\u003e\n          \u003cimg src=\"https://github.com/SimonBlanke/Gradient-Free-Optimizers/actions/workflows/tests_ubuntu.yml/badge.svg?branch=master\" alt=\"img not loaded: try F5 :)\"\u003e\n        \u003c/a\u003e\n        \u003ca href=\"https://github.com/SimonBlanke/Gradient-Free-Optimizers/actions\"\u003e\n          \u003cimg src=\"https://github.com/SimonBlanke/Gradient-Free-Optimizers/actions/workflows/tests_windows.yml/badge.svg?branch=master\" alt=\"img not loaded: try F5 :)\"\u003e\n        \u003c/a\u003e\n        \u003ca href=\"https://github.com/SimonBlanke/Gradient-Free-Optimizers/actions\"\u003e\n          \u003cimg src=\"https://github.com/SimonBlanke/Gradient-Free-Optimizers/actions/workflows/tests_macos.yml/badge.svg?branch=master\" alt=\"img not loaded: try F5 :)\"\u003e\n        \u003c/a\u003e\n        \u003ca href=\"https://app.codecov.io/gh/SimonBlanke/Gradient-Free-Optimizers\"\u003e\n          \u003cimg src=\"https://img.shields.io/codecov/c/github/SimonBlanke/Gradient-Free-Optimizers/master\" alt=\"img not loaded: try F5 :)\"\u003e\n        \u003c/a\u003e\n      \u003c/td\u003e\n    \u003c/tr\u003e\n    \u003ctr align=\"left\" valign=\"center\"\u003e\n      \u003ctd\u003e\n        \u003cstrong\u003eDev status:\u003c/strong\u003e\n      \u003c/td\u003e\n      \u003ctd\u003e\n        \u003ca href=\"https://github.com/SimonBlanke/Gradient-Free-Optimizers/actions\"\u003e\n          \u003cimg src=\"https://github.com/SimonBlanke/Gradient-Free-Optimizers/actions/workflows/tests_ubuntu.yml/badge.svg?branch=dev\" alt=\"img not loaded: try F5 :)\"\u003e\n        \u003c/a\u003e\n        \u003ca href=\"https://github.com/SimonBlanke/Gradient-Free-Optimizers/actions\"\u003e\n          \u003cimg src=\"https://github.com/SimonBlanke/Gradient-Free-Optimizers/actions/workflows/tests_windows.yml/badge.svg?branch=dev\" alt=\"img not loaded: try F5 :)\"\u003e\n        \u003c/a\u003e\n        \u003ca href=\"https://github.com/SimonBlanke/Gradient-Free-Optimizers/actions\"\u003e\n          \u003cimg src=\"https://github.com/SimonBlanke/Gradient-Free-Optimizers/actions/workflows/tests_macos.yml/badge.svg?branch=dev\" alt=\"img not loaded: try F5 :)\"\u003e\n        \u003c/a\u003e\n        \u003ca href=\"https://app.codecov.io/gh/SimonBlanke/Gradient-Free-Optimizers\"\u003e\n          \u003cimg src=\"https://img.shields.io/codecov/c/github/SimonBlanke/Gradient-Free-Optimizers/dev\" alt=\"img not loaded: try F5 :)\"\u003e\n        \u003c/a\u003e\n      \u003c/td\u003e\n    \u003c/tr\u003e\n    \u003ctr align=\"left\" valign=\"center\"\u003e\n      \u003ctd\u003e\n         \u003cstrong\u003eCode quality:\u003c/strong\u003e\n      \u003c/td\u003e\n      \u003ctd\u003e\n        \u003ca href=\"https://codeclimate.com/github/SimonBlanke/Gradient-Free-Optimizers\"\u003e\n        \u003cimg src=\"https://img.shields.io/codeclimate/maintainability/SimonBlanke/Gradient-Free-Optimizers?style=flat-square\u0026logo=code-climate\" alt=\"img not loaded: try F5 :)\"\u003e\n        \u003c/a\u003e\n        \u003ca href=\"https://scrutinizer-ci.com/g/SimonBlanke/Gradient-Free-Optimizers/\"\u003e\n        \u003cimg src=\"https://img.shields.io/scrutinizer/quality/g/SimonBlanke/Gradient-Free-Optimizers?style=flat-square\u0026logo=scrutinizer-ci\" alt=\"img not loaded: try F5 :)\"\u003e\n        \u003c/a\u003e\n      \u003c/td\u003e\n    \u003c/tr\u003e\n    \u003ctr align=\"left\" valign=\"center\"\u003e\n      \u003ctd\u003e\n        \u003cstrong\u003eLatest versions:\u003c/strong\u003e\n      \u003c/td\u003e\n      \u003ctd\u003e\n        \u003ca href=\"https://pypi.org/project/gradient_free_optimizers/\"\u003e\n          \u003cimg src=\"https://img.shields.io/pypi/v/Gradient-Free-Optimizers?style=flat-square\u0026logo=PyPi\u0026logoColor=white\u0026color=blue\" alt=\"img not loaded: try F5 :)\"\u003e\n        \u003c/a\u003e\n      \u003c/td\u003e\n    \u003c/tr\u003e\n  \u003c/tbody\u003e\n\u003c/table\u003e\n\n\u003cbr\u003e\n\n\n\n\n\n\n## Introduction\n\nGradient-Free-Optimizers provides a collection of easy to use optimization techniques, \nwhose objective function only requires an arbitrary score that gets maximized. \nThis makes gradient-free methods capable of solving various optimization problems, including: \n- Optimizing arbitrary mathematical functions.\n- Fitting multiple gauss-distributions to data.\n- Hyperparameter-optimization of machine-learning methods.\n\nGradient-Free-Optimizers is the optimization backend of \u003ca href=\"https://github.com/SimonBlanke/Hyperactive\"\u003eHyperactive\u003c/a\u003e  (in v3.0.0 and higher) but it can also be used by itself as a leaner and simpler optimization toolkit. \n\n\n\u003cbr\u003e\n\n---\n\n\u003cdiv align=\"center\"\u003e\u003ca name=\"menu\"\u003e\u003c/a\u003e\n  \u003ch3\u003e\n    \u003ca href=\"https://github.com/SimonBlanke/Gradient-Free-Optimizers#optimization-algorithms\"\u003eOptimization algorithms\u003c/a\u003e •\n    \u003ca href=\"https://github.com/SimonBlanke/Gradient-Free-Optimizers#installation\"\u003eInstallation\u003c/a\u003e •\n    \u003ca href=\"https://github.com/SimonBlanke/Gradient-Free-Optimizers#examples\"\u003eExamples\u003c/a\u003e •\n    \u003ca href=\"https://simonblanke.github.io/gradient-free-optimizers-documentation\"\u003eAPI reference\u003c/a\u003e •\n  \u003c/h3\u003e\n\u003c/div\u003e\n\n---\n\n\n\u003cbr\u003e\n\n## Main features\n\n- Easy to use:\n  \u003cdetails\u003e\n  \u003csummary\u003e\u003cb\u003e Simple API-design\u003c/b\u003e\u003c/summary\u003e\n\n  \u003cbr\u003e\n\n  You can optimize anything that can be defined in a python function. For example a simple parabola function:\n  ```python\n  def objective_function(para):\n      score = para[\"x1\"] * para[\"x1\"]\n      return -score\n  ```\n\n  Define where to search via numpy ranges:\n  ```python\n  search_space = {\n      \"x\": np.arange(0, 5, 0.1),\n  }\n  ```\n\n  That`s all the information the algorithm needs to search for the maximum in the objective function:\n  ```python\n  from gradient_free_optimizers import RandomSearchOptimizer\n\n  opt = RandomSearchOptimizer(search_space)\n  opt.search(objective_function, n_iter=100000)\n  ```\n\n\n  \u003c/details\u003e\n\n\n  \u003cdetails\u003e\n  \u003csummary\u003e\u003cb\u003e Receive prepared information about ongoing and finished optimization runs\u003c/b\u003e\u003c/summary\u003e\n\n  \u003cbr\u003e\n\n  During the optimization you will receive ongoing information in a progress bar:\n    - current best score\n    - the position in the search space of the current best score\n    - the iteration when the current best score was found\n    - other information about the progress native to tqdm\n\n  \u003c/details\u003e\n\n\n- High performance:\n  \u003cdetails\u003e\n  \u003csummary\u003e\u003cb\u003e Modern optimization techniques\u003c/b\u003e\u003c/summary\u003e\n\n  \u003cbr\u003e\n\n  Gradient-Free-Optimizers provides not just meta-heuristic optimization methods but also sequential model based optimizers like bayesian optimization, which delivers good results for expensive objetive functions like deep-learning models.\n\n  \u003c/details\u003e\n\n\n  \u003cdetails\u003e\n  \u003csummary\u003e\u003cb\u003e Lightweight backend\u003c/b\u003e\u003c/summary\u003e\n\n  \u003cbr\u003e\n\n  Even for the very simple parabola function the optimization time is about 60% of the entire iteration time when optimizing with random search.  This shows, that (despite all its features) Gradient-Free-Optimizers has an efficient optimization backend without any unnecessary slowdown.\n\n  \u003c/details\u003e\n\n\n  \u003cdetails\u003e\n  \u003csummary\u003e\u003cb\u003e Save time with memory dictionary\u003c/b\u003e\u003c/summary\u003e\n\n  \u003cbr\u003e\n\n  Per default Gradient-Free-Optimizers will look for the current position in a memory dictionary before evaluating the objective function. \n  \n    - If the position is not in the dictionary the objective function will be evaluated and the position and score is saved in the dictionary. \n    \n    - If a position is already saved in the dictionary Gradient-Free-Optimizers will just extract the score from it instead of evaluating the objective function. This avoids reevaluating computationally expensive objective functions (machine- or deep-learning) and therefore saves time.\n\n\n  \u003c/details\u003e\n\n\n- High reliability:\n  \u003cdetails\u003e\n  \u003csummary\u003e\u003cb\u003e Extensive testing\u003c/b\u003e\u003c/summary\u003e\n\n  \u003cbr\u003e\n\n  Gradient-Free-Optimizers is extensivly tested with more than 400 tests in 2500 lines of test code. This includes the testing of:\n    - Each optimization algorithm \n    - Each optimization parameter\n    - All attributes that are part of the public api\n\n  \u003c/details\u003e\n\n\n  \u003cdetails\u003e\n  \u003csummary\u003e\u003cb\u003e Performance test for each optimizer\u003c/b\u003e\u003c/summary\u003e\n\n  \u003cbr\u003e\n\n  Each optimization algorithm must perform above a certain threshold to be included. Poorly performing algorithms are reworked or scraped.\n\n  \u003c/details\u003e\n\n\n\u003cbr\u003e\n\n## Optimization algorithms:\n\nGradient-Free-Optimizers supports a variety of optimization algorithms, which can make choosing the right algorithm a tedious endeavor. The gifs in this section give a visual representation how the different optimization algorithms explore the search space and exploit the collected information about the search space for a convex and non-convex objective function. More detailed explanations of all optimization algorithms can be found in the [official documentation](https://simonblanke.github.io/gradient-free-optimizers-documentation).\n\n\n\n\u003cbr\u003e\n\n### Local Optimization\n\n\u003cdetails\u003e\n\u003csummary\u003e\u003cb\u003eHill Climbing\u003c/b\u003e\u003c/summary\u003e\n\n\u003cbr\u003e\n\nEvaluates the score of n neighbours in an epsilon environment and moves to the best one.\n\n\u003cbr\u003e\n\n\u003ctable style=\"width:100%\"\u003e\n  \u003ctr\u003e\n    \u003cth\u003e \u003cb\u003eConvex Function\u003c/b\u003e \u003c/th\u003e \n    \u003cth\u003e \u003cb\u003eNon-convex Function\u003c/b\u003e \u003c/th\u003e\n  \u003c/tr\u003e\n  \u003ctr\u003e\n    \u003ctd\u003e \u003cimg src=\"./docs/gifs/hill_climbing_sphere_function_.gif\" width=\"100%\"\u003e \u003c/td\u003e\n    \u003ctd\u003e \u003cimg src=\"./docs/gifs/hill_climbing_ackley_function_.gif\" width=\"100%\"\u003e \u003c/td\u003e\n  \u003c/tr\u003e\n\u003c/table\u003e\n\n\u003c/details\u003e\n\n\n\u003cdetails\u003e\n\u003csummary\u003e\u003cb\u003eStochastic Hill Climbing\u003c/b\u003e\u003c/summary\u003e\n\n\u003cbr\u003e\n\nAdds a probability to the hill climbing to move to a worse position in the search-space to escape local optima.\n\n\u003cbr\u003e\n\n\u003ctable style=\"width:100%\"\u003e\n  \u003ctr\u003e\n    \u003cth\u003e \u003cb\u003eConvex Function\u003c/b\u003e \u003c/th\u003e \n    \u003cth\u003e \u003cb\u003eNon-convex Function\u003c/b\u003e \u003c/th\u003e\n  \u003c/tr\u003e\n  \u003ctr\u003e\n    \u003ctd\u003e \u003cimg src=\"./docs/gifs/stochastic_hill_climbing_sphere_function_.gif\" width=\"100%\"\u003e \u003c/td\u003e\n    \u003ctd\u003e \u003cimg src=\"./docs/gifs/stochastic_hill_climbing_ackley_function_.gif\" width=\"100%\"\u003e \u003c/td\u003e\n  \u003c/tr\u003e\n\u003c/table\u003e\n\n\u003c/details\u003e\n\n\n\u003cdetails\u003e\n\u003csummary\u003e\u003cb\u003eRepulsing Hill Climbing\u003c/b\u003e\u003c/summary\u003e\n\n\u003cbr\u003e\n\nHill climbing algorithm with the addition of increasing epsilon by a factor if no better neighbour was found.\n\n\u003cbr\u003e\n\n\u003ctable style=\"width:100%\"\u003e\n  \u003ctr\u003e\n    \u003cth\u003e \u003cb\u003eConvex Function\u003c/b\u003e \u003c/th\u003e \n    \u003cth\u003e \u003cb\u003eNon-convex Function\u003c/b\u003e \u003c/th\u003e\n  \u003c/tr\u003e\n  \u003ctr\u003e\n    \u003ctd\u003e \u003cimg src=\"./docs/gifs/repulsing_hill_climbing_sphere_function_.gif\" width=\"100%\"\u003e \u003c/td\u003e\n    \u003ctd\u003e \u003cimg src=\"./docs/gifs/repulsing_hill_climbing_ackley_function_.gif\" width=\"100%\"\u003e \u003c/td\u003e\n  \u003c/tr\u003e\n\u003c/table\u003e\n\n\u003c/details\u003e\n\n\n\u003cdetails\u003e\n\u003csummary\u003e\u003cb\u003eSimulated Annealing\u003c/b\u003e\u003c/summary\u003e\n\n\u003cbr\u003e\n\nAdds a probability to the hill climbing to move to a worse position in the search-space to escape local optima with decreasing probability over time.\n\n\u003cbr\u003e\n\n\u003ctable style=\"width:100%\"\u003e\n  \u003ctr\u003e\n    \u003cth\u003e \u003cb\u003eConvex Function\u003c/b\u003e \u003c/th\u003e \n    \u003cth\u003e \u003cb\u003eNon-convex Function\u003c/b\u003e \u003c/th\u003e\n  \u003c/tr\u003e\n  \u003ctr\u003e\n    \u003ctd\u003e \u003cimg src=\"./docs/gifs/simulated_annealing_sphere_function_.gif\" width=\"100%\"\u003e \u003c/td\u003e\n    \u003ctd\u003e \u003cimg src=\"./docs/gifs/simulated_annealing_ackley_function_.gif\" width=\"100%\"\u003e \u003c/td\u003e\n  \u003c/tr\u003e\n\u003c/table\u003e\n\n\u003c/details\u003e\n\n\n\u003cdetails\u003e\n\u003csummary\u003e\u003cb\u003eDownhill Simplex Optimization\u003c/b\u003e\u003c/summary\u003e\n\n\u003cbr\u003e\n\nConstructs a simplex from multiple positions that moves through the search-space by reflecting, expanding, contracting or shrinking.\n\n\u003cbr\u003e\n\n\u003ctable style=\"width:100%\"\u003e\n  \u003ctr\u003e\n    \u003cth\u003e \u003cb\u003eConvex Function\u003c/b\u003e \u003c/th\u003e \n    \u003cth\u003e \u003cb\u003eNon-convex Function\u003c/b\u003e \u003c/th\u003e\n  \u003c/tr\u003e\n  \u003ctr\u003e\n    \u003ctd\u003e \u003cimg src=\"./docs/gifs/downhill_simplex_sphere_function_.gif\" width=\"100%\"\u003e \u003c/td\u003e\n    \u003ctd\u003e \u003cimg src=\"./docs/gifs/downhill_simplex_ackley_function_.gif\" width=\"100%\"\u003e \u003c/td\u003e\n  \u003c/tr\u003e\n\u003c/table\u003e\n\n\u003c/details\u003e\n\n\u003cbr\u003e\n\n### Global Optimization\n\n\u003cdetails\u003e\n\u003csummary\u003e\u003cb\u003eRandom Search\u003c/b\u003e\u003c/summary\u003e\n\n\u003cbr\u003e\n\nMoves to random positions in each iteration.\n\n\u003cbr\u003e\n\n\u003ctable style=\"width:100%\"\u003e\n  \u003ctr\u003e\n    \u003cth\u003e \u003cb\u003eConvex Function\u003c/b\u003e \u003c/th\u003e \n    \u003cth\u003e \u003cb\u003eNon-convex Function\u003c/b\u003e \u003c/th\u003e\n  \u003c/tr\u003e\n  \u003ctr\u003e\n    \u003ctd\u003e \u003cimg src=\"./docs/gifs/random_search_sphere_function_.gif\" width=\"100%\"\u003e \u003c/td\u003e\n    \u003ctd\u003e \u003cimg src=\"./docs/gifs/random_search_ackley_function_.gif\" width=\"100%\"\u003e \u003c/td\u003e\n  \u003c/tr\u003e\n\u003c/table\u003e\n\n\u003c/details\u003e\n\n\n\u003cdetails\u003e\n\u003csummary\u003e\u003cb\u003eGrid Search\u003c/b\u003e\u003c/summary\u003e\n\n\u003cbr\u003e\n\nGrid-search that moves through search-space diagonal (with step-size=1) starting from a corner.\n\n\u003cbr\u003e\n\n\u003ctable style=\"width:100%\"\u003e\n  \u003ctr\u003e\n    \u003cth\u003e \u003cb\u003eConvex Function\u003c/b\u003e \u003c/th\u003e \n    \u003cth\u003e \u003cb\u003eNon-convex Function\u003c/b\u003e \u003c/th\u003e\n  \u003c/tr\u003e\n  \u003ctr\u003e\n    \u003ctd\u003e \u003cimg src=\"./docs/gifs/grid_search_sphere_function_.gif\" width=\"100%\"\u003e \u003c/td\u003e\n    \u003ctd\u003e \u003cimg src=\"./docs/gifs/grid_search_ackley_function_.gif\" width=\"100%\"\u003e \u003c/td\u003e\n  \u003c/tr\u003e\n\u003c/table\u003e\n\n\u003c/details\u003e\n\n\n\u003cdetails\u003e\n\u003csummary\u003e\u003cb\u003eRandom Restart Hill Climbing\u003c/b\u003e\u003c/summary\u003e\n\n\u003cbr\u003e\n\nHill climbingm, that moves to a random position after n iterations.\n\n\u003cbr\u003e\n\n\u003ctable style=\"width:100%\"\u003e\n  \u003ctr\u003e\n    \u003cth\u003e \u003cb\u003eConvex Function\u003c/b\u003e \u003c/th\u003e \n    \u003cth\u003e \u003cb\u003eNon-convex Function\u003c/b\u003e \u003c/th\u003e\n  \u003c/tr\u003e\n  \u003ctr\u003e\n    \u003ctd\u003e \u003cimg src=\"./docs/gifs/random_restart_hill_climbing_sphere_function_.gif\" width=\"100%\"\u003e \u003c/td\u003e\n    \u003ctd\u003e \u003cimg src=\"./docs/gifs/random_restart_hill_climbing_ackley_function_.gif\" width=\"100%\"\u003e \u003c/td\u003e\n  \u003c/tr\u003e\n\u003c/table\u003e\n\n\u003c/details\u003e\n\n\n\u003cdetails\u003e\n\u003csummary\u003e\u003cb\u003eRandom Annealing\u003c/b\u003e\u003c/summary\u003e\n\n\u003cbr\u003e\n\nHill Climbing, that has large epsilon at the start of the search decreasing over time.\n\n\u003cbr\u003e\n\n\u003ctable style=\"width:100%\"\u003e\n  \u003ctr\u003e\n    \u003cth\u003e \u003cb\u003eConvex Function\u003c/b\u003e \u003c/th\u003e \n    \u003cth\u003e \u003cb\u003eNon-convex Function\u003c/b\u003e \u003c/th\u003e\n  \u003c/tr\u003e\n  \u003ctr\u003e\n    \u003ctd\u003e \u003cimg src=\"./docs/gifs/random_annealing_sphere_function_.gif\" width=\"100%\"\u003e \u003c/td\u003e\n    \u003ctd\u003e \u003cimg src=\"./docs/gifs/random_annealing_ackley_function_.gif\" width=\"100%\"\u003e \u003c/td\u003e\n  \u003c/tr\u003e\n\u003c/table\u003e\n\n\u003c/details\u003e\n\n\n\u003cdetails\u003e\n\u003csummary\u003e\u003cb\u003ePattern Search\u003c/b\u003e\u003c/summary\u003e\n\n\u003cbr\u003e\n\nCreates cross-shaped collection of positions that move through search-space by moving as a whole towards optima or shrinking the cross.\n\n\u003cbr\u003e\n\n\u003ctable style=\"width:100%\"\u003e\n  \u003ctr\u003e\n    \u003cth\u003e \u003cb\u003eConvex Function\u003c/b\u003e \u003c/th\u003e \n    \u003cth\u003e \u003cb\u003eNon-convex Function\u003c/b\u003e \u003c/th\u003e\n  \u003c/tr\u003e\n  \u003ctr\u003e\n    \u003ctd\u003e \u003cimg src=\"./docs/gifs/pattern_search_sphere_function_.gif\" width=\"100%\"\u003e \u003c/td\u003e\n    \u003ctd\u003e \u003cimg src=\"./docs/gifs/pattern_search_ackley_function_.gif\" width=\"100%\"\u003e \u003c/td\u003e\n  \u003c/tr\u003e\n\u003c/table\u003e\n\n\u003c/details\u003e\n\n\n\u003cdetails\u003e\n\u003csummary\u003e\u003cb\u003ePowell's Method\u003c/b\u003e\u003c/summary\u003e\n\n\u003cbr\u003e\n\nOptimizes each search-space dimension at a time with a hill-climbing algorithm.\n\n\u003cbr\u003e\n\n\u003ctable style=\"width:100%\"\u003e\n  \u003ctr\u003e\n    \u003cth\u003e \u003cb\u003eConvex Function\u003c/b\u003e \u003c/th\u003e \n    \u003cth\u003e \u003cb\u003eNon-convex Function\u003c/b\u003e \u003c/th\u003e\n  \u003c/tr\u003e\n  \u003ctr\u003e\n    \u003ctd\u003e \u003cimg src=\"./docs/gifs/powells_method_sphere_function_.gif\" width=\"100%\"\u003e \u003c/td\u003e\n    \u003ctd\u003e \u003cimg src=\"./docs/gifs/powells_method_ackley_function_.gif\" width=\"100%\"\u003e \u003c/td\u003e\n  \u003c/tr\u003e\n\u003c/table\u003e\n\n\u003c/details\u003e\n\n\n\u003cbr\u003e\n\n\n\n\n### Population-Based Optimization\n\n\u003cdetails\u003e\n\u003csummary\u003e\u003cb\u003eParallel Tempering\u003c/b\u003e\u003c/summary\u003e\n\n\u003cbr\u003e\n\nPopulation of n simulated annealers, which occasionally swap transition probabilities.\n\n\u003cbr\u003e\n\n\u003ctable style=\"width:100%\"\u003e\n  \u003ctr\u003e\n    \u003cth\u003e \u003cb\u003eConvex Function\u003c/b\u003e \u003c/th\u003e \n    \u003cth\u003e \u003cb\u003eNon-convex Function\u003c/b\u003e \u003c/th\u003e\n  \u003c/tr\u003e\n  \u003ctr\u003e\n    \u003ctd\u003e \u003cimg src=\"./docs/gifs/parallel_tempering_sphere_function_.gif\" width=\"100%\"\u003e \u003c/td\u003e\n    \u003ctd\u003e \u003cimg src=\"./docs/gifs/parallel_tempering_ackley_function_.gif\" width=\"100%\"\u003e \u003c/td\u003e\n  \u003c/tr\u003e\n\u003c/table\u003e\n\n\u003c/details\u003e\n\n\n\u003cdetails\u003e\n\u003csummary\u003e\u003cb\u003eParticle Swarm Optimization\u003c/b\u003e\u003c/summary\u003e\n\n\u003cbr\u003e\n\nPopulation of n particles attracting each other and moving towards the best particle.\n\n\u003cbr\u003e\n\n\u003ctable style=\"width:100%\"\u003e\n  \u003ctr\u003e\n    \u003cth\u003e \u003cb\u003eConvex Function\u003c/b\u003e \u003c/th\u003e \n    \u003cth\u003e \u003cb\u003eNon-convex Function\u003c/b\u003e \u003c/th\u003e\n  \u003c/tr\u003e\n  \u003ctr\u003e\n    \u003ctd\u003e \u003cimg src=\"./docs/gifs/particle_swarm_optimization_sphere_function_.gif\" width=\"100%\"\u003e \u003c/td\u003e\n    \u003ctd\u003e \u003cimg src=\"./docs/gifs/particle_swarm_optimization_ackley_function_.gif\" width=\"100%\"\u003e \u003c/td\u003e\n  \u003c/tr\u003e\n\u003c/table\u003e\n\n\u003c/details\u003e\n\n\n\u003cdetails\u003e\n\u003csummary\u003e\u003cb\u003eSpiral Optimization\u003c/b\u003e\u003c/summary\u003e\n\n\u003cbr\u003e\n\nPopulation of n particles moving in a spiral pattern around the best position.\n\n\n\u003cbr\u003e\n\n\u003ctable style=\"width:100%\"\u003e\n  \u003ctr\u003e\n    \u003cth\u003e \u003cb\u003eConvex Function\u003c/b\u003e \u003c/th\u003e \n    \u003cth\u003e \u003cb\u003eNon-convex Function\u003c/b\u003e \u003c/th\u003e\n  \u003c/tr\u003e\n  \u003ctr\u003e\n    \u003ctd\u003e \u003cimg src=\"./docs/gifs/spiral_optimization_sphere_function_.gif\" width=\"100%\"\u003e \u003c/td\u003e\n    \u003ctd\u003e \u003cimg src=\"./docs/gifs/spiral_optimization_ackley_function_.gif\" width=\"100%\"\u003e \u003c/td\u003e\n  \u003c/tr\u003e\n\u003c/table\u003e\n\n\u003c/details\u003e\n\n\n\n\u003cdetails\u003e\n\u003csummary\u003e\u003cb\u003eGenetic Algorithm\u003c/b\u003e\u003c/summary\u003e\n\n\u003cbr\u003e\n\nEvolutionary algorithm selecting the best individuals in the population, mixing their parameters to get new solutions.\n\n\u003cbr\u003e\n\n\u003ctable style=\"width:100%\"\u003e\n  \u003ctr\u003e\n    \u003cth\u003e \u003cb\u003eConvex Function\u003c/b\u003e \u003c/th\u003e \n    \u003cth\u003e \u003cb\u003eNon-convex Function\u003c/b\u003e \u003c/th\u003e\n  \u003c/tr\u003e\n  \u003ctr\u003e\n    \u003ctd\u003e \u003cimg src=\"./docs/gifs/genetic_algorithm_sphere_function_.gif\" width=\"100%\"\u003e \u003c/td\u003e\n    \u003ctd\u003e \u003cimg src=\"./docs/gifs/genetic_algorithm_ackley_function_.gif\" width=\"100%\"\u003e \u003c/td\u003e\n  \u003c/tr\u003e\n\u003c/table\u003e\n\n\u003c/details\u003e\n\n\n\u003cdetails\u003e\n\u003csummary\u003e\u003cb\u003eEvolution Strategy\u003c/b\u003e\u003c/summary\u003e\n\n\u003cbr\u003e\n\nPopulation of n hill climbers occasionally mixing positional information and removing worst positions from population.\n\n\u003cbr\u003e\n\n\u003ctable style=\"width:100%\"\u003e\n  \u003ctr\u003e\n    \u003cth\u003e \u003cb\u003eConvex Function\u003c/b\u003e \u003c/th\u003e \n    \u003cth\u003e \u003cb\u003eNon-convex Function\u003c/b\u003e \u003c/th\u003e\n  \u003c/tr\u003e\n  \u003ctr\u003e\n    \u003ctd\u003e \u003cimg src=\"./docs/gifs/evolution_strategy_sphere_function_.gif\" width=\"100%\"\u003e \u003c/td\u003e\n    \u003ctd\u003e \u003cimg src=\"./docs/gifs/evolution_strategy_ackley_function_.gif\" width=\"100%\"\u003e \u003c/td\u003e\n  \u003c/tr\u003e\n\u003c/table\u003e\n\n\u003c/details\u003e\n\n\n\u003cdetails\u003e\n\u003csummary\u003e\u003cb\u003eDifferential Evolution\u003c/b\u003e\u003c/summary\u003e\n\n\u003cbr\u003e\n\nImproves a population of candidate solutions by creating trial vectors through the differential mutation of three randomly selected individuals.\n\n\u003cbr\u003e\n\n\u003ctable style=\"width:100%\"\u003e\n  \u003ctr\u003e\n    \u003cth\u003e \u003cb\u003eConvex Function\u003c/b\u003e \u003c/th\u003e \n    \u003cth\u003e \u003cb\u003eNon-convex Function\u003c/b\u003e \u003c/th\u003e\n  \u003c/tr\u003e\n  \u003ctr\u003e\n    \u003ctd\u003e \u003cimg src=\"./docs/gifs/differential_evolution_sphere_function_.gif\" width=\"100%\"\u003e \u003c/td\u003e\n    \u003ctd\u003e \u003cimg src=\"./docs/gifs/differential_evolution_ackley_function_.gif\" width=\"100%\"\u003e \u003c/td\u003e\n  \u003c/tr\u003e\n\u003c/table\u003e\n\n\u003c/details\u003e\n\n\n\u003cbr\u003e\n\n\n### Sequential Model-Based Optimization\n\n\u003cdetails\u003e\n\u003csummary\u003e\u003cb\u003eBayesian Optimization\u003c/b\u003e\u003c/summary\u003e\n\n\u003cbr\u003e\n\nGaussian process fitting to explored positions and predicting promising new positions.\n\n\u003cbr\u003e\n\n\u003ctable style=\"width:100%\"\u003e\n  \u003ctr\u003e\n    \u003cth\u003e \u003cb\u003eConvex Function\u003c/b\u003e \u003c/th\u003e \n    \u003cth\u003e \u003cb\u003eNon-convex Function\u003c/b\u003e \u003c/th\u003e\n  \u003c/tr\u003e\n  \u003ctr\u003e\n    \u003ctd\u003e \u003cimg src=\"./docs/gifs/bayesian_optimization_sphere_function_.gif\" width=\"100%\"\u003e \u003c/td\u003e\n    \u003ctd\u003e \u003cimg src=\"./docs/gifs/bayesian_optimization_ackley_function_.gif\" width=\"100%\"\u003e \u003c/td\u003e\n  \u003c/tr\u003e\n\u003c/table\u003e\n\n\u003c/details\u003e\n\n\n\u003cdetails\u003e\n\u003csummary\u003e\u003cb\u003eLipschitz Optimization\u003c/b\u003e\u003c/summary\u003e\n\n\u003cbr\u003e\n\nCalculates an upper bound from the distances of the previously explored positions to find new promising positions.\n\n\u003cbr\u003e\n\n\u003ctable style=\"width:100%\"\u003e\n  \u003ctr\u003e\n    \u003cth\u003e \u003cb\u003eConvex Function\u003c/b\u003e \u003c/th\u003e \n    \u003cth\u003e \u003cb\u003eNon-convex Function\u003c/b\u003e \u003c/th\u003e\n  \u003c/tr\u003e\n  \u003ctr\u003e\n    \u003ctd\u003e \u003cimg src=\"./docs/gifs/lipschitz_optimizer_sphere_function_.gif\" width=\"100%\"\u003e \u003c/td\u003e\n    \u003ctd\u003e \u003cimg src=\"./docs/gifs/lipschitz_optimizer_ackley_function_.gif\" width=\"100%\"\u003e \u003c/td\u003e\n  \u003c/tr\u003e\n\u003c/table\u003e\n\n\u003c/details\u003e\n\n\n\u003cdetails\u003e\n\u003csummary\u003e\u003cb\u003eDIRECT algorithm\u003c/b\u003e\u003c/summary\u003e\n\n\u003cbr\u003e\n\nSeparates search space into subspaces. It evaluates the center position of each subspace to decide which subspace to sepate further.\n\n\u003cbr\u003e\n\n\u003ctable style=\"width:100%\"\u003e\n  \u003ctr\u003e\n    \u003cth\u003e \u003cb\u003eConvex Function\u003c/b\u003e \u003c/th\u003e \n    \u003cth\u003e \u003cb\u003eNon-convex Function\u003c/b\u003e \u003c/th\u003e\n  \u003c/tr\u003e\n  \u003ctr\u003e\n    \u003ctd\u003e \u003cimg src=\"./docs/gifs/direct_algorithm_sphere_function_.gif\" width=\"100%\"\u003e \u003c/td\u003e\n    \u003ctd\u003e \u003cimg src=\"./docs/gifs/direct_algorithm_ackley_function_.gif\" width=\"100%\"\u003e \u003c/td\u003e\n  \u003c/tr\u003e\n\u003c/table\u003e\n\n\u003c/details\u003e\n\n\n\u003cdetails\u003e\n\u003csummary\u003e\u003cb\u003eTree of Parzen Estimators\u003c/b\u003e\u003c/summary\u003e\n\n\u003cbr\u003e\n\nKernel density estimators fitting to good and bad explored positions and predicting promising new positions.\n\n\u003cbr\u003e\n\n\u003ctable style=\"width:100%\"\u003e\n  \u003ctr\u003e\n    \u003cth\u003e \u003cb\u003eConvex Function\u003c/b\u003e \u003c/th\u003e \n    \u003cth\u003e \u003cb\u003eNon-convex Function\u003c/b\u003e \u003c/th\u003e\n  \u003c/tr\u003e\n  \u003ctr\u003e\n    \u003ctd\u003e \u003cimg src=\"./docs/gifs/tree_structured_parzen_estimators_sphere_function_.gif\" width=\"100%\"\u003e \u003c/td\u003e\n    \u003ctd\u003e \u003cimg src=\"./docs/gifs/tree_structured_parzen_estimators_ackley_function_.gif\" width=\"100%\"\u003e \u003c/td\u003e\n  \u003c/tr\u003e\n\u003c/table\u003e\n\n\u003c/details\u003e\n\n\n\u003cdetails\u003e\n\u003csummary\u003e\u003cb\u003eForest Optimizer\u003c/b\u003e\u003c/summary\u003e\n\n\u003cbr\u003e\n\nEnsemble of decision trees fitting to explored positions and predicting promising new positions.\n\n\u003cbr\u003e\n\n\u003ctable style=\"width:100%\"\u003e\n  \u003ctr\u003e\n    \u003cth\u003e \u003cb\u003eConvex Function\u003c/b\u003e \u003c/th\u003e \n    \u003cth\u003e \u003cb\u003eNon-convex Function\u003c/b\u003e \u003c/th\u003e\n  \u003c/tr\u003e\n  \u003ctr\u003e\n    \u003ctd\u003e \u003cimg src=\"./docs/gifs/forest_optimization_sphere_function_.gif\" width=\"100%\"\u003e \u003c/td\u003e\n    \u003ctd\u003e \u003cimg src=\"./docs/gifs/forest_optimization_ackley_function_.gif\" width=\"100%\"\u003e \u003c/td\u003e\n  \u003c/tr\u003e\n\u003c/table\u003e\n\n\u003c/details\u003e\n\n\n\n\u003cbr\u003e\n\n## Sideprojects and Tools\n\nThe following packages are designed to support Gradient-Free-Optimizers and expand its use cases. \n\n| Package                                                                       | Description                                                                          |\n|-------------------------------------------------------------------------------|--------------------------------------------------------------------------------------|\n| [Search-Data-Collector](https://github.com/SimonBlanke/search-data-collector) | Simple tool to save search-data during or after the optimization run into csv-files. |\n| [Search-Data-Explorer](https://github.com/SimonBlanke/search-data-explorer)   | Visualize search-data with plotly inside a streamlit dashboard.\n\nIf you want news about Gradient-Free-Optimizers and related projects you can follow me on [twitter](https://twitter.com/blanke_simon).\n\n\n\u003cbr\u003e\n\n## Installation\n\n[![PyPI version](https://badge.fury.io/py/gradient-free-optimizers.svg)](https://badge.fury.io/py/gradient-free-optimizers)\n\nThe most recent version of Gradient-Free-Optimizers is available on PyPi:\n\n```console\npip install gradient-free-optimizers\n```\n\n\u003cbr\u003e\n\n\n## Examples\n\n\u003cdetails\u003e\n\u003csummary\u003e\u003cb\u003eConvex function\u003c/b\u003e\u003c/summary\u003e\n\n```python\nimport numpy as np\nfrom gradient_free_optimizers import RandomSearchOptimizer\n\n\ndef parabola_function(para):\n    loss = para[\"x\"] * para[\"x\"]\n    return -loss\n\n\nsearch_space = {\"x\": np.arange(-10, 10, 0.1)}\n\nopt = RandomSearchOptimizer(search_space)\nopt.search(parabola_function, n_iter=100000)\n```\n\n\u003c/details\u003e\n\n\n\u003cdetails\u003e\n\u003csummary\u003e\u003cb\u003eNon-convex function\u003c/b\u003e\u003c/summary\u003e\n\n```python\nimport numpy as np\nfrom gradient_free_optimizers import RandomSearchOptimizer\n\n\ndef ackley_function(pos_new):\n    x = pos_new[\"x1\"]\n    y = pos_new[\"x2\"]\n\n    a1 = -20 * np.exp(-0.2 * np.sqrt(0.5 * (x * x + y * y)))\n    a2 = -np.exp(0.5 * (np.cos(2 * np.pi * x) + np.cos(2 * np.pi * y)))\n    score = a1 + a2 + 20\n    return -score\n\n\nsearch_space = {\n    \"x1\": np.arange(-100, 101, 0.1),\n    \"x2\": np.arange(-100, 101, 0.1),\n}\n\nopt = RandomSearchOptimizer(search_space)\nopt.search(ackley_function, n_iter=30000)\n```\n\n\u003c/details\u003e\n\n\n\u003cdetails\u003e\n\u003csummary\u003e\u003cb\u003eMachine learning example\u003c/b\u003e\u003c/summary\u003e\n\n```python\nimport numpy as np\nfrom sklearn.model_selection import cross_val_score\nfrom sklearn.ensemble import GradientBoostingClassifier\nfrom sklearn.datasets import load_wine\n\nfrom gradient_free_optimizers import HillClimbingOptimizer\n\n\ndata = load_wine()\nX, y = data.data, data.target\n\n\ndef model(para):\n    gbc = GradientBoostingClassifier(\n        n_estimators=para[\"n_estimators\"],\n        max_depth=para[\"max_depth\"],\n        min_samples_split=para[\"min_samples_split\"],\n        min_samples_leaf=para[\"min_samples_leaf\"],\n    )\n    scores = cross_val_score(gbc, X, y, cv=3)\n\n    return scores.mean()\n\n\nsearch_space = {\n    \"n_estimators\": np.arange(20, 120, 1),\n    \"max_depth\": np.arange(2, 12, 1),\n    \"min_samples_split\": np.arange(2, 12, 1),\n    \"min_samples_leaf\": np.arange(1, 12, 1),\n}\n\nopt = HillClimbingOptimizer(search_space)\nopt.search(model, n_iter=50)\n```\n\n\u003c/details\u003e\n\n\n\u003cdetails\u003e\n\u003csummary\u003e\u003cb\u003eConstrained  Optimization example\u003c/b\u003e\u003c/summary\u003e\n\n```python\nimport numpy as np\nfrom gradient_free_optimizers import RandomSearchOptimizer\n\n\ndef convex_function(pos_new):\n    score = -(pos_new[\"x1\"] * pos_new[\"x1\"] + pos_new[\"x2\"] * pos_new[\"x2\"])\n    return score\n\n\nsearch_space = {\n    \"x1\": np.arange(-100, 101, 0.1),\n    \"x2\": np.arange(-100, 101, 0.1),\n}\n\n\ndef constraint_1(para):\n    # only values in 'x1' higher than -5 are valid\n    return para[\"x1\"] \u003e -5\n\n\n# put one or more constraints inside a list\nconstraints_list = [constraint_1]\n\n\n# pass list of constraints to the optimizer\nopt = RandomSearchOptimizer(search_space, constraints=constraints_list)\nopt.search(convex_function, n_iter=50)\n\nsearch_data = opt.search_data\n\n# the search-data does not contain any samples where x1 is equal or below -5\nprint(\"\\n search_data \\n\", search_data, \"\\n\")\n```\n\n\u003c/details\u003e\n\n\n\n\u003cbr\u003e\n\n## Gradient Free Optimizers \u003c=\u003e Hyperactive\n\nGradient-Free-Optimizers was created as the optimization backend of the [Hyperactive package](https://github.com/SimonBlanke/Hyperactive). Therefore the algorithms are exactly the same in both packages and deliver the same results. \nHowever you can still use Gradient-Free-Optimizers as a standalone package.\nThe separation of Gradient-Free-Optimizers from Hyperactive enables multiple advantages:\n  - Even easier to use than Hyperactive\n  - Separate and more thorough testing\n  - Other developers can easily use GFOs as an optimizaton backend if desired\n  - Better isolation from the complex information flow in Hyperactive. GFOs only uses positions and scores in a N-dimensional search-space. It returns only the new position after each iteration.\n  - a smaller and cleaner code base, if you want to explore my implementation of these optimization techniques.\n\nWhile Gradient-Free-Optimizers is relatively simple, Hyperactive is a more complex project with additional features to make optimization of computationally expensive models (like engineering simulation or machine-/deep-learning models) more convenient.\n\n\n\u003cbr\u003e\n\n## Citation\n\n    @Misc{gfo2020,\n      author =   {{Simon Blanke}},\n      title =    {{Gradient-Free-Optimizers}: Simple and reliable optimization with local, global, population-based and sequential techniques in numerical search spaces.},\n      howpublished = {\\url{https://github.com/SimonBlanke}},\n      year = {since 2020}\n    }\n\n\n\u003cbr\u003e\n\n## License\n\nGradient-Free-Optimizers is licensed under the following License:\n\n[![LICENSE](https://img.shields.io/github/license/SimonBlanke/Gradient-Free-Optimizers?style=for-the-badge)](https://github.com/SimonBlanke/Gradient-Free-Optimizers/blob/master/LICENSE)\n\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FSimonBlanke%2FGradient-Free-Optimizers","html_url":"https://awesome.ecosyste.ms/projects/github.com%2FSimonBlanke%2FGradient-Free-Optimizers","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FSimonBlanke%2FGradient-Free-Optimizers/lists"}