{"id":27490341,"url":"https://github.com/microsoft/mlos","last_synced_at":"2025-05-16T06:04:50.752Z","repository":{"id":37669140,"uuid":"253620591","full_name":"microsoft/MLOS","owner":"microsoft","description":"MLOS is a project to enable autotuning for systems.","archived":false,"fork":false,"pushed_at":"2025-05-16T01:00:30.000Z","size":351516,"stargazers_count":157,"open_issues_count":111,"forks_count":73,"subscribers_count":12,"default_branch":"main","last_synced_at":"2025-05-16T06:04:49.435Z","etag":null,"topics":["autotuning","benchmarking","benchmarking-framework","data-science","infrastructure","optimize-systems","performance-engineering"],"latest_commit_sha":null,"homepage":"https://microsoft.github.io/MLOS","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/microsoft.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE.txt","code_of_conduct":"CODE_OF_CONDUCT.md","threat_model":null,"audit":null,"citation":null,"codeowners":"CODEOWNERS","security":"SECURITY.md","support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2020-04-06T21:33:40.000Z","updated_at":"2025-05-14T17:34:51.000Z","dependencies_parsed_at":"2023-12-07T02:26:36.781Z","dependency_job_id":"cc6eaf8a-9322-4e52-817d-64d97a82b9f9","html_url":"https://github.com/microsoft/MLOS","commit_stats":null,"previous_names":[],"tags_count":17,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/microsoft%2FMLOS","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/microsoft%2FMLOS/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/microsoft%2FMLOS/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/microsoft%2FMLOS/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/microsoft","download_url":"https://codeload.github.com/microsoft/MLOS/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":254478187,"owners_count":22077676,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["autotuning","benchmarking","benchmarking-framework","data-science","infrastructure","optimize-systems","performance-engineering"],"created_at":"2025-04-16T20:55:07.887Z","updated_at":"2025-05-16T06:04:50.730Z","avatar_url":"https://github.com/microsoft.png","language":"Python","readme":"# MLOS\n\n[![MLOS DevContainer](https://github.com/microsoft/MLOS/actions/workflows/devcontainer.yml/badge.svg)](https://github.com/microsoft/MLOS/actions/workflows/devcontainer.yml)\n[![MLOS Linux](https://github.com/microsoft/MLOS/actions/workflows/linux.yml/badge.svg)](https://github.com/microsoft/MLOS/actions/workflows/linux.yml)\n[![MLOS MacOS](https://github.com/microsoft/MLOS/actions/workflows/macos.yml/badge.svg)](https://github.com/microsoft/MLOS/actions/workflows/macos.yml)\n[![MLOS Windows](https://github.com/microsoft/MLOS/actions/workflows/windows.yml/badge.svg)](https://github.com/microsoft/MLOS/actions/workflows/windows.yml)\n[![Code Coverage Status](https://microsoft.github.io/MLOS/_images/coverage.svg)](https://microsoft.github.io/MLOS/htmlcov/index.html)\n[![pre-commit.ci status](https://results.pre-commit.ci/badge/github/microsoft/MLOS/main.svg)](https://results.pre-commit.ci/latest/github/microsoft/MLOS/main)\n\nMLOS is a project to enable autotuning for systems.\n\n## Contents\n\n\u003c!-- mdformat-toc start --slug=github --no-anchors --maxlevel=4 --minlevel=2 --\u003e\n\n- [Contents](#contents)\n- [Overview](#overview)\n- [Organization](#organization)\n- [Contributing](#contributing)\n- [Getting Started](#getting-started)\n  - [`conda` activation](#conda-activation)\n  - [Usage Examples](#usage-examples)\n    - [`mlos-core`](#mlos-core)\n    - [`mlos-bench`](#mlos-bench)\n    - [`mlos-viz`](#mlos-viz)\n- [Installation](#installation)\n- [See Also](#see-also)\n  - [Examples](#examples)\n  - [Publications](#publications)\n\n\u003c!-- mdformat-toc end --\u003e\n\n## Overview\n\nMLOS currently focuses on an *offline* tuning approach, though we intend to add online tuning in the future.\n\nTo accomplish this, the general flow involves\n\n- Running a workload (i.e., benchmark) against a system (e.g., a database, web server, or key-value store).\n- Retrieving the results of that benchmark, and perhaps some other metrics from the system.\n- Feed that data to an optimizer (e.g., using [Bayesian Optimization](https://en.wikipedia.org/wiki/Bayesian_optimization) or other techniques).\n- Obtain a new suggested config to try from the optimizer.\n- Apply that configuration to the target system.\n- Repeat until either the exploration budget is consumed or the configurations' performance appear to have converged.\n\n\u003c!-- markdownlint-disable MD033 --\u003e\n\n\u003cimg src=\"./doc/source/_static/llamatune-loop.png\" style=\"width:700px\" alt=\"optimization loop\" /\u003e\n\n\u003c!-- markdownlint-enable MD033 --\u003e\n\n\u003e Source: [LlamaTune: VLDB 2022](https://arxiv.org/abs/2203.05128)\n\nFor a brief overview of some of the features and capabilities of MLOS, please see the following video:\n\n[![demo video](https://aka.ms/MLOS/demo-video-preview-image)](https://aka.ms/MLOS/demo-video)\n\n## Organization\n\nTo do this this repo provides three Python modules, which can be used independently or in combination:\n\n- [`mlos-bench`](./mlos_bench/) provides a framework to help automate running benchmarks as described above.\n\n- [`mlos-viz`](./mlos_viz/) provides some simple APIs to help automate visualizing the results of benchmark experiments and their trials.\n\n  It provides a simple `plot(experiment_data)` API, where `experiment_data` is obtained from the `mlos_bench.storage` module.\n\n- [`mlos-core`](./mlos_core/) provides an abstraction around existing optimization frameworks (e.g., [FLAML](https://github.com/microsoft/FLAML), [SMAC](https://github.com/automl/SMAC3), etc.)\n\n  It is intended to provide a simple, easy to consume (e.g. via `pip`), with low dependencies abstraction to\n\n  - describe a space of context, parameters, their ranges, constraints, etc. and result objectives\n  - an \"optimizer\" service [abstraction](https://microsoft.github.io/MLOS/source_tree_docs/index.html) (e.g. [`register()`](https://microsoft.github.io/MLOS/autoapi/mlos_bench/optimizers/base_optimizer/index.html#mlos_bench.optimizers.base_optimizer.Optimizer.register) and [`suggest()`](https://microsoft.github.io/MLOS/autoapi/mlos_bench/optimizers/base_optimizer/index.html#mlos_bench.optimizers.base_optimizer.Optimizer.suggest)) so we can easily swap out different implementations methods of searching (e.g. random, BO, LLM, etc.)\n  - provide some helpers for [automating optimization experiment](https://microsoft.github.io/MLOS/source_tree_docs/mlos_bench/index.html) runner loops and data collection\n\nFor these design requirements we intend to reuse as much from existing OSS libraries as possible and layer policies and optimizations specifically geared towards autotuning systems over top.\n\nBy providing wrappers we aim to also allow more easily experimenting with replacing underlying optimizer components as new techniques become available or seem to be a better match for certain systems.\n\n## Contributing\n\nSee [CONTRIBUTING.md](./CONTRIBUTING.md) for details on development environment and contributing.\n\n## Getting Started\n\nThe development environment for MLOS uses [`conda`](https://docs.conda.io/projects/conda/en/latest/user-guide/install/index.html) and [`devcontainers`](https://code.visualstudio.com/docs/remote/containers) to ease dependency management, but not all these libraries are required for deployment.\n\nFor instructions on setting up the development environment please try one of the following options:\n\n- see [CONTRIBUTING.md](./CONTRIBUTING.md) for details on setting up a local development environment\n- launch this repository (or your fork) in a [codespace](https://github.com/features/codespaces), or\n- have a look at one of the autotuning example repositories like [sqlite-autotuning](https://github.com/Microsoft-CISL/sqlite-autotuning) to kick the tires in a [codespace](https://github.com/features/codespaces) in your browser immediately :)\n\n### `conda` activation\n\n1. Create the `mlos` Conda environment.\n\n   ```sh\n   conda env create -f conda-envs/mlos.yml\n   ```\n\n   \u003e See the [`conda-envs/`](./conda-envs/) directory for additional conda environment files, including those used for Windows (e.g. [`mlos-windows.yml`](./conda-envs/mlos-windows.yml)).\n\n   or\n\n   ```sh\n   # This will also ensure the environment is update to date using \"conda env update -f conda-envs/mlos.yml\"\n   make conda-env\n   ```\n\n   \u003e Note: the latter expects a \\*nix environment.\n\n1. Initialize the shell environment.\n\n   ```sh\n   conda activate mlos\n   ```\n\n### Usage Examples\n\n#### `mlos-core`\n\nFor an example of using the `mlos_core` optimizer APIs run the [`BayesianOptimization.ipynb`](./mlos_core/notebooks/BayesianOptimization.ipynb) notebook.\n\n#### `mlos-bench`\n\nFor an example of using the `mlos_bench` tool to run an experiment, see the [`mlos_bench` Quickstart README](./mlos_bench/README.md#quickstart).\n\nHere's a quick summary:\n\n```shell\n./scripts/generate-azure-credentials-config \u003e global_config_azure.jsonc\n\n# run a simple experiment\nmlos_bench --config ./mlos_bench/mlos_bench/config/cli/azure-redis-1shot.jsonc\n```\n\nSee Also:\n\n- [mlos_bench/README.md](./mlos_bench/README.md) for more details on this example.\n- [mlos_bench/config](./mlos_bench/mlos_bench/config/) for additional configuration details.\n- [sqlite-autotuning](https://github.com/Microsoft-CISL/sqlite-autotuning) for a complete external example of using MLOS to tune `sqlite`.\n\n#### `mlos-viz`\n\nFor a simple example of using the `mlos_viz` module to visualize the results of an experiment, see the [`sqlite-autotuning`](https://github.com/Microsoft-CISL/sqlite-autotuning) repository, especially the [mlos_demo_sqlite_teachers.ipynb](https://github.com/Microsoft-CISL/sqlite-autotuning/blob/main/mlos_demo_sqlite_teachers.ipynb) notebook.\n\n## Installation\n\nThe MLOS modules are published to [pypi](https://pypi.org) when new releases are tagged:\n\n- [mlos-core](https://pypi.org/project/mlos-core/)\n- [mlos-bench](https://pypi.org/project/mlos-bench/)\n- [mlos-viz](https://pypi.org/project/mlos-viz/)\n\nTo install the latest release, simply run:\n\n```sh\n# this will install just the optimizer component with SMAC support:\npip install -U mlos-core[smac]\n\n# this will install just the optimizer component with flaml support:\npip install -U \"mlos-core[flaml]\"\n\n# this will install just the optimizer component with smac and flaml support:\npip install -U \"mlos-core[smac,flaml]\"\n\n# this will install both the flaml optimizer and the experiment runner with azure support:\npip install -U \"mlos-bench[flaml,azure]\"\n\n# this will install both the smac optimizer and the experiment runner with ssh support:\npip install -U \"mlos-bench[smac,ssh]\"\n\n# this will install the postgres storage backend for mlos-bench\n# and mlos-viz for visualizing results:\npip install -U \"mlos-bench[postgres]\" mlos-viz\n```\n\nDetails on using a local version from git are available in [CONTRIBUTING.md](./CONTRIBUTING.md).\n\n## See Also\n\n- API and Examples Documentation: \u003chttps://microsoft.github.io/MLOS\u003e\n- Source Code Repository: \u003chttps://github.com/microsoft/MLOS\u003e\n\n### Examples\n\n- [mlos-autotuning-template repository](https://github.com/microsoft/mlos-autotuning-template)\n\n- [sqlite-autotuning](https://github.com/Microsoft-CISL/sqlite-autotuning)\n\n  Working example of tuning `sqlite` with MLOS.\n\nThese can be used as starting points for new autotuning projects outside of the main MLOS repository if you want to keep your tuning experiment configs separate from the MLOS codebase.\n\nAlternatively, we accept PRs to add new examples to the main MLOS repository!\nSee [mlos_bench/config](./mlos_bench/mlos_bench/config/) and [CONTRIBUTING.md](./CONTRIBUTING.md) for more details.\n\n### Publications\n\n- [TUNA: Tuning Unstable and Noisy Cloud Applications](http://aka.ms/mlos/tuna-eurosys-paper) at [EuroSys 2025](https://2025.eurosys.org/accepted-papers.html)\n- [MLOS in Action: Bridging the Gap Between Experimentation and Auto-Tuning in the Cloud](https://www.vldb.org/pvldb/vol17/p4269-kroth.pdf) at [VLDB 2024](https://www.vldb.org/2024/?papers-demo)\n- [Towards Building Autonomous Data Services on Azure](https://dl.acm.org/doi/abs/10.1145/3555041.3589674) in [SIGMOD Companion 2023](https://dl.acm.org/doi/proceedings/10.1145/3555041)\n- [LlamaTune: Sample-efficient DBMS configuration tuning](https://www.microsoft.com/en-us/research/publication/llamatune-sample-efficient-dbms-configuration-tuning) at [VLDB 2022](https://vldb.org/pvldb/volumes/15/)\n- [MLOS: An infrastructure for automated software performance engineering](https://arxiv.org/abs/2006.02155) at [DEEM 2020](https://deem-workshop.github.io/2020/index.html)\n","funding_links":[],"categories":[],"sub_categories":[],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmicrosoft%2Fmlos","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fmicrosoft%2Fmlos","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmicrosoft%2Fmlos/lists"}