{"id":13751644,"url":"https://github.com/MLBazaar/BTB","last_synced_at":"2025-05-09T18:31:55.530Z","repository":{"id":44715601,"uuid":"109332441","full_name":"MLBazaar/BTB","owner":"MLBazaar","description":"A simple, extensible library for developing AutoML systems","archived":false,"fork":false,"pushed_at":"2023-07-28T16:53:13.000Z","size":17131,"stargazers_count":175,"open_issues_count":15,"forks_count":41,"subscribers_count":22,"default_branch":"main","last_synced_at":"2025-04-17T17:22:59.317Z","etag":null,"topics":["automl","gaussian-processes","hyperparameter-optimization","multi-armed-bandit"],"latest_commit_sha":null,"homepage":"https://mlbazaar.github.io/BTB/","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/MLBazaar.png","metadata":{"files":{"readme":"README.md","changelog":"HISTORY.md","contributing":"CONTRIBUTING.rst","funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":"AUTHORS.rst"}},"created_at":"2017-11-03T00:18:41.000Z","updated_at":"2025-04-10T11:51:40.000Z","dependencies_parsed_at":"2024-01-15T03:41:45.286Z","dependency_job_id":null,"html_url":"https://github.com/MLBazaar/BTB","commit_stats":{"total_commits":709,"total_committers":12,"mean_commits":"59.083333333333336","dds":0.4400564174894217,"last_synced_commit":"5053ed705cf2542e320e8a7605642f5b01db8272"},"previous_names":[],"tags_count":24,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/MLBazaar%2FBTB","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/MLBazaar%2FBTB/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/MLBazaar%2FBTB/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/MLBazaar%2FBTB/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/MLBazaar","download_url":"https://codeload.github.com/MLBazaar/BTB/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":253303027,"owners_count":21886873,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["automl","gaussian-processes","hyperparameter-optimization","multi-armed-bandit"],"created_at":"2024-08-03T09:00:50.871Z","updated_at":"2025-05-09T18:31:50.500Z","avatar_url":"https://github.com/MLBazaar.png","language":"Python","funding_links":[],"categories":["参数优化"],"sub_categories":[],"readme":"\u003cp align=\"left\"\u003e\n\u003cimg width=\"15%\" src=\"https://dai.lids.mit.edu/wp-content/uploads/2018/06/Logo_DAI_highres.png\" alt=\"BTB\" /\u003e\n\u003ci\u003eAn open source project from Data to AI Lab at MIT.\u003c/i\u003e\n\u003c/p\u003e\n\n![](https://raw.githubusercontent.com/MLBazaar/BTB/master/docs/images/BTB-Icon-small.png)\n\nA simple, extensible backend for developing auto-tuning systems.\n\n[![Development Status](https://img.shields.io/badge/Development%20Status-2%20--%20Pre--Alpha-yellow)](https://pypi.org/search/?c=Development+Status+%3A%3A+2+-+Pre-Alpha)\n[![PyPi Shield](https://img.shields.io/pypi/v/baytune.svg)](https://pypi.python.org/pypi/baytune)\n[![Travis CI Shield](https://travis-ci.com/MLBazaar/BTB.svg?branch=master)](https://travis-ci.com/MLBazaar/BTB)\n[![Coverage Status](https://codecov.io/gh/MLBazaar/BTB/branch/master/graph/badge.svg)](https://codecov.io/gh/MLBazaar/BTB)\n[![Downloads](https://pepy.tech/badge/baytune)](https://pepy.tech/project/baytune)\n[![Binder](https://mybinder.org/badge_logo.svg)](https://mybinder.org/v2/gh/MLBazaar/BTB/master?filepath=tutorials)\n\n* License: [MIT](https://github.com/MLBazaar/BTB/blob/master/LICENSE)\n* Development Status: [Pre-Alpha](https://pypi.org/search/?c=Development+Status+%3A%3A+2+-+Pre-Alpha)\n* Documentation: https://mlbazaar.github.io/BTB\n* Homepage: https://github.com/MLBazaar/BTB\n\n# Overview\n\nBTB (\"Bayesian Tuning and Bandits\") is a simple, extensible backend for developing auto-tuning\nsystems such as AutoML systems. It provides an easy-to-use interface for *tuning* models and\n*selecting* between models.\n\nIt is currently being used in several AutoML systems:\n\n- [ATM](https://github.com/HDI-Project/ATM), a distributed, multi-tenant AutoML system for\nclassifier tuning\n- [MIT's system](https://github.com/HDI-Project/mit-d3m-ta2/) for the DARPA\n[Data-driven discovery of models](https://www.darpa.mil/program/data-driven-discovery-of-models) (D3M) program\n- [AutoBazaar](https://github.com/MLBazaar/AutoBazaar), a flexible, general-purpose\nAutoML system\n\n## Try it out now!\n\nIf you want to quickly discover **BTB**, simply click the button below and follow the tutorials!\n\n[![Binder](https://mybinder.org/badge_logo.svg)](https://mybinder.org/v2/gh/MLBazaar/BTB/master?filepath=tutorials)\n\n# Install\n\n## Requirements\n\n**BTB** has been developed and tested on [Python 3.6, 3.7 and 3.8](https://www.python.org/downloads/)\n\nAlso, although it is not strictly required, the usage of a\n[virtualenv](https://virtualenv.pypa.io/en/latest/) is highly recommended in order to avoid\ninterfering with other software installed in the system where **BTB** is run.\n\n## Install with pip\n\nThe easiest and recommended way to install **BTB** is using [pip](\nhttps://pip.pypa.io/en/stable/):\n\n```bash\npip install baytune\n```\n\nThis will pull and install the latest stable release from [PyPi](https://pypi.org/).\n\nIf you want to install from source or contribute to the project please read the\n[Contributing Guide](https://mlbazaar.github.io/BTB/contributing.html#get-started).\n\n# Quickstart\n\nIn this short tutorial we will guide you through the necessary steps to get started using BTB\nto `select` between models and `tune` a model to solve a Machine Learning problem.\n\nIn particular, in this example we will be using ``BTBSession`` to perform solve the [Wine](\nhttps://archive.ics.uci.edu/ml/machine-learning-databases/wine/wine.data) classification problem\nby selecting between the `DecisionTreeClassifier` and the `SGDClassifier` models from\n[scikit-learn](https://scikit-learn.org/) while also searching for their best `hyperparameter`\nconfiguration.\n\n## Prepare a scoring function\n\nThe first step in order to use the `BTBSession` class is to develop a `scoring` function.\n\nThis is a Python function that, given a model name and a `hyperparameter` configuration,\nevaluates the performance of the model on your data and returns a score.\n\n```python3\nfrom sklearn.datasets import load_wine\nfrom sklearn.linear_model import SGDClassifier\nfrom sklearn.metrics import f1_score, make_scorer\nfrom sklearn.model_selection import cross_val_score\nfrom sklearn.tree import DecisionTreeClassifier\n\n\ndataset = load_wine()\nmodels = {\n    'DTC': DecisionTreeClassifier,\n    'SGDC': SGDClassifier,\n}\n\ndef scoring_function(model_name, hyperparameter_values):\n    model_class = models[model_name]\n    model_instance = model_class(**hyperparameter_values)\n    scores = cross_val_score(\n        estimator=model_instance,\n        X=dataset.data,\n        y=dataset.target,\n        scoring=make_scorer(f1_score, average='macro')\n    )\n    return scores.mean()\n```\n\n## Define the tunable hyperparameters\n\nThe second step is to define the `hyperparameters` that we want to `tune` for each model as\n`Tunables`.\n\n```python3\nfrom btb.tuning import Tunable\nfrom btb.tuning import hyperparams as hp\n\ntunables = {\n    'DTC': Tunable({\n        'max_depth': hp.IntHyperParam(min=3, max=200),\n        'min_samples_split': hp.FloatHyperParam(min=0.01, max=1)\n    }),\n    'SGDC': Tunable({\n        'max_iter': hp.IntHyperParam(min=1, max=5000, default=1000),\n        'tol': hp.FloatHyperParam(min=1e-3, max=1, default=1e-3),\n    })\n}\n```\n\n## Start the searching process\n\nOnce you have defined a `scoring` function and the tunable `hyperparameters` specification of your\nmodels, you can start the searching for the best model and `hyperparameter` configuration by using\nthe `btb.BTBSession`.\n\nAll you need to do is create an instance passing the tunable `hyperparameters` scpecification\nand the scoring function.\n\n```python3\nfrom btb import BTBSession\n\nsession = BTBSession(\n    tunables=tunables,\n    scorer=scoring_function\n)\n```\n\nAnd then call the `run` method indicating how many tunable iterations you want the `BTBSession` to\nperform:\n\n\n```python3\nbest_proposal = session.run(20)\n```\n\nThe result will be a dictionary indicating the name of the best model that could be found\nand the `hyperparameter` configuration that was used:\n\n```\n{\n    'id': '826aedc2eff31635444e8104f0f3da43',\n    'name': 'DTC',\n    'config': {\n        'max_depth': 21,\n        'min_samples_split': 0.044010284821858835\n    },\n    'score': 0.907229308339589\n}\n ```\n\n# How does BTB perform?\n\nWe have a comprehensive [benchmarking framework](https://github.com/MLBazaar/BTB/tree/master/benchmark)\nthat we use to evaluate the performance of our `Tuners`. For every release, we perform benchmarking\nagainst 100's of challenges, comparing tuners against each other in terms of number of wins.\nWe present the latest leaderboard from latest release below:\n\n## Number of Wins on latest Version\n\n| tuner                   | with ties | without ties |\n|-------------------------|-----------|--------------|\n| `Ax.optimize`           |    220    |           32 |\n| `BTB.GCPEiTuner`        |    139    |            2 |\n| `BTB.GCPTuner`          |  **252**  |       **90** |\n| `BTB.GPEiTuner`         |    208    |           16 |\n| `BTB.GPTuner`           |    213    |           24 |\n| `BTB.UniformTuner`      |    177    |            1 |\n| `HyperOpt.tpe`          |    186    |            6 |\n| `SMAC.HB4AC`            |    180    |            4 |\n| `SMAC.SMAC4HPO_EI`      |    220    |           31 |\n| `SMAC.SMAC4HPO_LCB`     |    205    |           16 |\n| `SMAC.SMAC4HPO_PI`      |    221    |           35 |\n\n- Detailed results from which this summary emerged are available [here](https://docs.google.com/spreadsheets/d/15a-pAV_t7CCDvqDyloYmdVNFhiKJFOJ7bbgpmYIpyTs/edit?usp=sharing).\n- If you want to compare your own tuner, follow the steps in our benchmarking framework [here](https://github.com/MLBazaar/BTB/tree/master/benchmark).\n- If you have a proposal for tuner that we should include in our benchmarking get in touch\nwith us at [dailabmit@gmail.com](mailto:dailabmit@gmail.com).\n\n# More tutorials\n\n1. To just `tune` `hyperparameters` - see our `tuning` tutorial [here](\nhttps://github.com/MLBazaar/BTB/blob/master/tutorials/01_Tuning.ipynb) and\n[documentation here](https://mlbazaar.github.io/BTB/tutorials/01_Tuning.html).\n2. To see the [types of `hyperparameters`](\nhttps://mlbazaar.github.io/BTB/tutorials/01_Tuning.html#What-is-a-Hyperparameter?) we support\nsee our [documentation here](https://mlbazaar.github.io/BTB/tutorials/01_Tuning.html#What-is-a-Hyperparameter?).\n3. You can read about [our benchmarking framework here](https://mlbazaar.github.io/BTB/benchmark.html#).\n4. See our [tutorial on `selection` here](https://github.com/MLBazaar/BTB/blob/master/tutorials/02_Selection.ipynb)\nand [documentation here](https://mlbazaar.github.io/BTB/tutorials/02_Selection.html).\n\nFor more details about **BTB** and all its possibilities and features, please check the\n[project documentation site](https://mlbazaar.github.io/BTB/)!\n\nAlso do not forget to have a look at the [notebook tutorials](tutorials).\n\n# Citing BTB\n\nIf you use **BTB**, please consider citing the following [paper](\nhttps://arxiv.org/pdf/1905.08942.pdf):\n\n```\n@article{smith2019mlbazaar,\n  author = {Smith, Micah J. and Sala, Carles and Kanter, James Max and Veeramachaneni, Kalyan},\n  title = {The Machine Learning Bazaar: Harnessing the ML Ecosystem for Effective System Development},\n  journal = {arXiv e-prints},\n  year = {2019},\n  eid = {arXiv:1905.08942},\n  pages = {arxiv:1904.09535},\n  archivePrefix = {arXiv},\n  eprint = {1905.08942},\n}\n`````\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FMLBazaar%2FBTB","html_url":"https://awesome.ecosyste.ms/projects/github.com%2FMLBazaar%2FBTB","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FMLBazaar%2FBTB/lists"}