{"id":13436302,"url":"https://github.com/JasperSnoek/spearmint","last_synced_at":"2025-03-18T21:30:18.233Z","repository":{"id":6441947,"uuid":"7681175","full_name":"JasperSnoek/spearmint","owner":"JasperSnoek","description":"Spearmint is a package to perform Bayesian optimization according to the algorithms outlined in the paper:  Practical Bayesian Optimization of Machine Learning Algorithms. Jasper Snoek, Hugo Larochelle and Ryan P. Adams.  Advances in Neural Information Processing Systems, 2012 ","archived":false,"fork":false,"pushed_at":"2017-07-21T14:38:48.000Z","size":565,"stargazers_count":1394,"open_issues_count":46,"forks_count":335,"subscribers_count":103,"default_branch":"master","last_synced_at":"2025-03-18T11:13:54.388Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"http://people.seas.harvard.edu/~jsnoek/software.html","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/JasperSnoek.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2013-01-18T05:44:37.000Z","updated_at":"2025-03-11T11:49:42.000Z","dependencies_parsed_at":"2022-08-26T04:01:26.212Z","dependency_job_id":null,"html_url":"https://github.com/JasperSnoek/spearmint","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/JasperSnoek%2Fspearmint","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/JasperSnoek%2Fspearmint/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/JasperSnoek%2Fspearmint/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/JasperSnoek%2Fspearmint/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/JasperSnoek","download_url":"https://codeload.github.com/JasperSnoek/spearmint/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":244310240,"owners_count":20432500,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-07-31T03:00:46.420Z","updated_at":"2025-03-18T21:30:18.194Z","avatar_url":"https://github.com/JasperSnoek.png","language":"Python","funding_links":[],"categories":["Uncategorized","Python","参数优化"],"sub_categories":["Uncategorized","General-Purpose Machine Learning"],"readme":"New code repository!\nAs you might have noticed, the development of this repository has been limited to maintenance and bug fixes for some time now.  The reason is that there has been a collaborative effort to overhaul Spearmint into a new codebase.  This includes both algorithmic/theoretical and engineering improvements.  Check it out at https://github.com/HIPS/Spearmint.  Note that the new repository is under a non-commercial license with a contributor license agreement.  If you prefer not to agree to the license, you can freely use code here (though it is a bit older).\n\nSpearmint\n---------\n\nSpearmint is a package to perform Bayesian optimization according to the\nalgorithms outlined in the paper:  \n\n**Practical Bayesian Optimization of Machine Learning Algorithms**  \nJasper Snoek, Hugo Larochelle and Ryan P. Adams  \n*Advances in Neural Information Processing Systems*, 2012  \n\nThis code is designed to automatically run experiments (thus the code\nname 'spearmint') in a manner that iteratively adjusts a number of\nparameters so as to minimize some objective in as few runs as\npossible.\n\nSpearmint is the result of a collaboration primarily between machine learning researchers at [Harvard University](https://hips.seas.harvard.edu/) and the [University of Toronto](http://learning.cs.toronto.edu/).\n\nDependencies\n------------\nThis package requires:\n\n* Python 2.7\n\n* [Numpy](http://www.numpy.org/) version 1.6.1+\nOn Ubuntu linux you can install this package using the command:\n\n\t\tapt-get install python-numpy\n\n* [Scipy](http://www.scipy.org/) version 0.9.0+\nOn Ubuntu linux you can install this package using the command:\n\n\t\tapt-get install python-scipy\n\n* [Google Protocol Buffers](https://developers.google.com/protocol-buffers/) (for the fully automated code).\nNote that you should be able to install protocol-buffers from source without requiring administrator privileges.  Otherwise, on Ubuntu linux you can install this package using the command:\n\n\t\tapt-get install python-protobuf\n\t\t\t\t\t\t\n\tand on Mac (if you use homebrew) with:\n\n\t\tbrew install protobuf\n\n\tThen from within the spearmint sub-directory run the command:\n\n\t\tbin/make_protobufs\n\nThis package has been tested on Ubuntu linux (versions 11.0+) and\nMac-OSX.\n\nThe code consists of several parts.  It is designed to be modular to\nallow swapping out various 'driver' and 'chooser' modules.  The\n'chooser' modules are implementations of acquisition functions such as\nexpected improvement, UCB or random.  The drivers determine how\nexperiments are distributed and run on the system.  As the code is\ndesigned to run experiments in parallel (spawning a new experiment as\nsoon a result comes in), this requires some engineering.  The current\nimplementations of these are in the 'spearmint' and 'spearmint-lite' subdirectories:\n\n**Spearmint** is designed to automatically manage the launching and associated bookkeeping \nof experiments in either a single machine or cluster environment.  This requires that you provide\na 'wrapper' in a supported language (currently Python or Matlab) and a configuration file detailing parameters\nto be tuned and their respective bounds. The wrapper must accept parameter values and then return simply a value which you wish to minimize with respect to the parameters. Spearmint will then iteratively call the wrapper with different \nparameter settings in an order that seeks to find the minimum value in as few evaluations (or cost) as possible.\n\n**Spearmint-lite** is the 'bare-bones' stripped version of the code.\nThis version is simply driven by a flat-file and does not\nautomatically run experiments.  Instead, it proposes new experiments\n(potentially multiple at a time) and requires that the user fill in\nthe result.  This is well suited to the case where writing a wrapper\naround the code doesn't make sense (e.g. if the experiments don't\ninvolve code at all) or if the user desires full control of the\nprocess.  Also, the dependency on Google protocol buffers is replaced\nwith JSON.\n\nRunning the automated code: Spearmint\n--------------------------------------------------------\n\nThe simplest way to get to know the code is probably to look at an\nexample.  In order to start a new experiment, you must create a directory \nthat includes a wrapper script and config file.\nWe have created one simple example for you that optimizes the\n'Braninhoo' benchmark in the subdirectory **examples/braninpy**.  Take a look at\n**config.pb**.  It contains the specifications of the algorithm in\nprotocol buffer format.  In order to specify your optimization, you\nhave to fill in the variables 'language' (e.g. PYTHON or MATLAB) and\n'name' (the name of the wrapper function you want to optimize).\n\nFollowed by these is a list of 'variables', which specifies the name,\ntype and size of the variables you wish to optimize over.  Each\nvariable must be either a FLOAT, INT or ENUM type, corresponding to\ncontinuous real valued parameters, integer sequences and categorical\nvariables respectively.  MAX and MIN specify the bounds of the\nvariables over which to optimize and SIZE is the number of variables\nof this type with these bounds.  Spearmint will call your wrapper\nfunction with a dictionary type (in python) containing each of your\nvariables in a vector of size 'size', which you can access using the\nname specified.  \n\nNow take a look at branin.py (the wrapper which was\nspecified in the 'name' variable at the top of config.pb).  You will\nsee the file has a function 'main(job_id, params)'.  Your wrapper must\ninclude this function, which spearmint will call passing in a job_id\n(which is probably not interesting to you) and a dictionary, 'params',\ncontaining the parameter vectors of the next experiment spearmint\nwants to run.  The main function should take as input these parameters\nand return a single real valued number representing the observed\nfunction value (that is being optimized) at these inputs.\n\nTo install spearmint, go into the spearmint subdirectory and type (most likely preceded with 'sudo'):\n\n\tpython setup.py install\n\nYou should add the spearmint subdirectory to your PYTHONPATH directory.  Note that you can often avoid the above step, but may have to add all the relevant modules to your PYTHONPATH and call spearmint directory from the directory using main.py.\n\nTo run spearmint, go into the **/bin** subdirectory and type:\n\n\t./spearmint ../examples/braninpy/config.pb --driver=local --method=GPEIOptChooser --method-args=noiseless=1\n\nor alternatively in the **/spearmint** subdirectory:\n\n\tpython main.py --driver=local --method=GPEIOptChooser --method-args=noiseless=1 ../examples/braninpy/config.pb\n\nThis will run spearmint according to the GP-EI MCMC strategy.  The code will sequentially spawn\nprocesses that call the wrapper function and it will poll for results.\nYou will see that the code prints out the current best (i.e. lowest)\nobservation seen thus far and sequences of numbers corresponding to GP\nhyperparameter samples and candidates it is optimizing over.  The\n'method' argument specifies the chooser module (acquisition function)\nto use and 'method-args' specifies chooser specific arguments.  In\nthis case, as braninhoo is an analytic function we tell the GP\nhyperparameter sampling routine to not try to estimate noise.  This parameter is very important to specify correctly for the optimization to proceed properly.  If your algorithm is entirely deterministic (e.g. analytic) then specifying that it is noiseless will speed up the optimization considerably.  If your algorithm is not deterministic, as we expect for most machine learning algorithms and indeed most expensive experiments, then you should leave this out or set noiseless=0.\n\nIf you let it run for a while you will see that the current-best\ndecreases, eventually reaching the minimum at ~0.39. You can kill the\nprocess (ctrl-c) at any time and you can restart from where it left off simply\nby rerunning the spearmint command.\n\nIf you go back in to the braninpy directory you will see a number of\nnew files that spearmint uses to do bookkeeping.  Of particular\ninterest are **trace.csv** and the **output** directory.  **trace.csv**\ncontains a record of the experiments run so far and the best result\n(and which experiment it came from) as a series over time.  Each line\nof trace.csv contains the following in csv format: a timestamp, the\nbest value observed up to that timestamp, the job-id of the best value\nobserved, the number of potential candidates left, the number of\npending (currently running) experiments, the number of experiments\ncompleted thus far.  The output directory contains a text file for\neach job-id, containing the output of that job.  So if you want to\nsee, e.g. what the output (i.e. standard out and standard error) was\nfor the best job (as obtained from trace.csv) you can look up\njob-id.txt in the output directory. \n\n If you are debugging your code,\nor the code is crashing for some reason, it's a good idea to look at\nthese files. Finally for ease of use, spearmint also prints out at\neach iteration a file called 'best_job_and_result.txt' that contains the\nbest result observed so far, the job-id it came from and a dump of \nthe names and values of all of the parameters corresponding to that result.\n\nA script, bin/cleanup, is provided to completely restart an experiment\nand delete all the results and intermediate files.  Simply run\n`bin/cleanup \u003cexperiment_dir\u003e`\n\nMatlab code can also be optimized using this package. To do so, you\nmust specify in config.pb \"type: Matlab\" and use a matlab wrapper with\nthe \"name\" specified in config.pb.  The matlab wrapper must have a\nfunction with the same name as the file name of the following form:\n\"function result = braninhoo(job_id, params)\" Above we assume the file\nname is \"braninhoo.m\".  Spearmint will pass in to this wrapper a\njob_id and a matlab struct 'params' where the fields are given by the\nvariables specified in config.pb.  See the subdirectory \"braninhoo\"\nfor a matlab example matching that of python 'braninpy' described\nabove.\n\nTo run multiple jobs in parallel, pass to spearmint the argument:\n`--max-concurrent=\u003c#jobs\u003e`\n\nSpearmint is designed to be run in parallel either using multiple processors on a single machine or in a cluster environment.  These different environments, however, involve different queuing and fault-checking code and are thus coded as 'driver' modules.  Currently two drivers are available, but one can easily create a driver for a different environment by creating a new driver module (see the driver subdirectory for examples).\n\nUsing the `--driver=sge` flag, Spearmint can run on a system with Sun Grid Engine and it uses SGE to distribute experiments on a multi-node cluster in parallel using a queueing system in a fault-tolerant way.  It is particularly\nwell suited to the Amazon EC2 system.  Using [StarCluster](http://star.mit.edu/cluster/) will allow you to set up a large cluster and start distributing experiments within minutes.\n\nUsing the `--driver=local` flag will run Spearmint on a single machine with potentially many cores.  This driver simply spawns a new process on the current machine to run a new experiment.  This does not allow you to distribute across multiple machines, however.\n\n### Web Status Page\nRunning spearmint with the flag `-w` will spawn a local web server that will display useful info and statistics about your optimization run in a much more interpretable and intuitive interface than the command line.  Spearmint will provide a link to the status page that you can simply paste into a browser window.\n\n\nRunning the basic code: Spearmint-lite \n---------------------------------------\n\nSpearmint-lite is designed to be simple.  To run an experiment in\nspearmint-lite, create a subdirectory as explained above.  Again, the\nbraninpy directory is provided as a demonstration.  In this case, the\nexperiment specification, which must be provided in config.json, is in\nJSON format.  You must specify your problem as a sequence of JSON\nobjects.  As in the protocol buffer format above, each object must\nhave a name, a type (float, int or enum), a 'min', a 'max' and a\n'size'. Nothing else needs to be specified.  \n\nGo back to the top-level directory and run: \n\n\tpython spearmint-lite.py braninpy\n\nSpearmint-lite will run one iteration of Bayesian\noptimization and write out to a file named results.dat in the braninpy\nsubdirectory.  results.dat will contain a white-space delimited line\nfor each experiment, of the format: \n`\u003cresult\u003e \u003ctime-taken\u003e \u003clist of parameters in the same order as config.json\u003e`\n\nSpearmint will propose new experiments and append them to results.dat each \ntime it is run. Each proposed experiment will have a 'pending' result and \ntime-taken, indicated by the letter P. The user must then run the experiment \nand fill in these values. Note that the time can safely be set to an arbitrary\nvalue if the chooser module does not use it (only GPEIperSecChooser currently \ndoes). Spearmint will condition on the pending experiments when proposing new \nones, so any number of experiments can be conducted in parallel.\n\nA script, **cleanup.sh**, is provided to completely clean up all the intermediate\nfiles and results in an experimental directory and restart the\nexperiment from scratch.\n\nChooser modules:\n--------------- \n\nThe chooser modules implement functions that tell spearmint which next\njob to run.  Some correspond to 'acquisition functions' in the\nBayesian optimization literature.  Spearmint takes as an argument\n`--method=ChooserModule` which allows one to easy swap out acquisition\nfunctions. Choosers may optionally include parameters that can be\npassed to spearmint using the argument\n`--method-args=argument1,argument2,etc`.  These include, for example, the\nnumber of GP hyperparameter samples to use. See the comments in\nchooser files for chooser dependent arguments.  Below are described\nthe choosers provided in this package:\n\n* **SequentialChooser:** Chooses the next experiment using a high\ndiscrepancy Sobol sequence.  Experiments are taken sequentially from a\ncoarse-to-fine grid.\n\n* **RandomChooser**: Experiments are sampled randomly from the unit hypercube.\n\n* **GPEIOptChooser:** The GP EI MCMC algorithm from the paper. Jobs \nare first sampled densely from a dense grid on the unit hypercube\nand then the best candidates are optimized 'fine-tuned' according\nto EI.\n\n* **GPEIperSecChooser:** The GP EI per Second algorithm from the paper.\nSimilar to GPEIOptChooser except points are optimized and evaluated\nbased on the EI per Second criterion, where each job is weighted\nby the expected running time of the experiment.\n\n* **GPEIChooser:** Points are densely sampled on the unit hypercube and the\nbest is returned according to the EI criterion.  This is considerably\nfaster than the GPEIOptChooser and works well in low-dimensional cases\nwhere the grid covers the search space densely.\n\n**IMPORTANT!**\n\nWhen estimating noise, the Gaussian process prior over noise assumes\nthat the noise level is 'low' (e.g. between -1 to 1). If this is not\nthe case, make sure to rescale your function to make this true\n(e.g. to approximately be between -1 and 1).  Otherwise, the algorithm\nwill find a bad high-noise mode that will result in bad performance.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FJasperSnoek%2Fspearmint","html_url":"https://awesome.ecosyste.ms/projects/github.com%2FJasperSnoek%2Fspearmint","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FJasperSnoek%2Fspearmint/lists"}