{"id":31809928,"url":"https://github.com/outerbounds/hpo-project","last_synced_at":"2025-10-11T05:55:53.660Z","repository":{"id":311811451,"uuid":"1045058326","full_name":"outerbounds/hpo-project","owner":"outerbounds","description":null,"archived":false,"fork":false,"pushed_at":"2025-08-26T18:36:13.000Z","size":106,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2025-08-27T02:09:03.617Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/outerbounds.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2025-08-26T15:38:59.000Z","updated_at":"2025-08-26T18:36:16.000Z","dependencies_parsed_at":"2025-08-27T02:09:26.199Z","dependency_job_id":"a4450bf2-a2b6-4e25-bd58-719dca1477d7","html_url":"https://github.com/outerbounds/hpo-project","commit_stats":null,"previous_names":["outerbounds/hpo-project"],"tags_count":null,"template":false,"template_full_name":null,"purl":"pkg:github/outerbounds/hpo-project","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/outerbounds%2Fhpo-project","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/outerbounds%2Fhpo-project/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/outerbounds%2Fhpo-project/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/outerbounds%2Fhpo-project/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/outerbounds","download_url":"https://codeload.github.com/outerbounds/hpo-project/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/outerbounds%2Fhpo-project/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":279006355,"owners_count":26084088,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-10-11T02:00:06.511Z","response_time":55,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2025-10-11T05:55:51.850Z","updated_at":"2025-10-11T05:55:53.654Z","avatar_url":"https://github.com/outerbounds.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"## Hyperparameter Optimization Project\n\u003cimg style=\"display: block; max-width: 100%; height: auto; margin: auto;\" alt=\"system\" src=\"https://github.com/user-attachments/assets/f9ca6892-879c-46b1-9696-bf691530de1c\" /\u003e\n\nThis repository shows you how to run a hyperparameter optimization (HPO) system as an Outerbounds project.\nThis `README.md` will explain why you'd want to connect these concepts, and will show you how to launch HPO jobs for:\n- classical ML models\n- deep learning models\n- end-to-end system tuning\n\nIf you have never deployed an Outerbounds project, please read the [Outerbounds documentation](https://docs.outerbounds.com/) before continuing.\n\n## Quick start\n\nChange the `platform` in [`obproject.toml`](./obproject.toml) to match your Outerbounds deployment. \n\n[Install uv](https://docs.astral.sh/uv/getting-started/installation/).\n\n```bash\nuv init\nuv add outerbounds optuna numpy pandas \"psycopg[binary]\u003e=3.2.0\" scikit-learn torch torchvision\n```\n\nEnsure you've run your `outerbounds configure ...` command.\nThen, run flows!\n\n```bash\ncd flows/tree\nuv run python flow.py --environment=fast-bakery run --with kubernetes\n```\n\n\u003e For more information about the containerization technology used in this project, see [Fast Bakery: Automatic Containerization](https://outerbounds.com/blog/containerize-with-fast-bakery).\n\n## How to customize this repository for your use cases\n\n### Make a new directory under `/flows`\nTo begin, copy the structure in `/flows/nn` or `/flows/tree`:\n- `config.json` contains system and hyperparameter config options.\n- `flow.py` defines the workflow structure. This should change little across use cases.\n- `objective_fn.py` this is the key piece of the puzzle for a new use case. See examples at https://github.com/optuna/optuna-examples/tree/main.\n- `utils.py` contains small project-specific helpers.\n- `interactive.ipynb` is a starter notebook for running and analyzing hyperparameter tuning runs in a REPL.\n- Symlink to `obproject.toml` at the root of the repository. \n\nIf desired, you can directly modify one of these sub-directories.\n\n### Define and evolve your own objective function\n\nThe key aspect of customization is about defining the objective function. \nCheck out the examples and reach out for assistance if you do not know how to parameterize your task as a tunable optimization problem. \nFrom there, determine the dependencies needed for running the objective function\nand update the `config.json` values accordingly, most notable the Python packages \nsection which `flow.py` will use when building consistent environments across \ncompute backends.\n\n## Advanced\n\n### Detailed set up\n\n#### Deploy the Optuna dashboard application\n\nThe Outerbounds app that will run your Optuna dashboard is defined in [`./deployments/optuna-dashboard/config.yml`](./deployments/optuna-dashboard/config.yml).\nWhen you push to the main branch of this repository, the `obproject-deployer` will create the application in your Outerbounds project branch.\nIf you'd like to manually deploy the application:\n\n```bash\ncd deployments/optuna-dashboard\nuv run outerbounds app deploy --config-file config.yml\n```\n\n#### Local/workstation dependencies\n\n[Install uv](https://docs.astral.sh/uv/getting-started/installation/).\n\nFrom your laptop or Outerbounds workstation run:\n```bash\nuv init\nuv add outerbounds optuna numpy pandas \"psycopg[binary]\u003e=3.2.0\" scikit-learn torch torchvision\n```\n\nConfigure Outerbounds token. Ask in Slack if not sure.\n\n#### Pick a sub-project\n```bash\ncd flows/tree\n# cd flows/nn\n```\n\n#### Setting configs\nBefore running or deploying the workflows, investigate the relationship between the flow and the `config.json` file.\n\nAs long as you haven't changed anything when deploying the application hosting the Optuna dashboard, you do not need to change anything in that file, \nbut it is useful to be familiar with these contents and the way the configuration files are interacting with Metaflow code. \n\n#### Run flows\nThere are two demos implemented within this project base in `flows/tree` and `flows/nn`.\nEach workflow template defines:\n- a `flow.py` containing a `FlowSpec`, \n- a single `config.json` to set system variables and hyperparameter configurations,\n- an `hpo_client.py` containing entrypoints to run and trigger the flow, \n- notebooks showing how to run and analyze results of hyperparameter tuning runs, and\n- the templates show how to define a modular, fully customizable objective function.\n\nFor the rest of this section, we'll use the `flows/nn` template, as everything else is the same as for `flows/tree`.\n\n```bash\ncd flows/nn\n```\n\n#### Use Metaflow directly\n```bash\nuv run python flow.py --environment=fast-bakery run --with kubernetes\nuv run python flow.py --environment=fast-bakery argo-workflows create/trigger\n```\n\n#### Use `hpo_client`\nThe examples also include a convenience wrapper around the workflows in the `hpo_client.py`. \nYou can use this for:\n- running HPO jobs from notebooks, CLI, or other Metaflow flows, or\n- as an example for creating your own experiment entrypoint abstractions.\n\n```bash\nuv run python hpo_client.py -m 1 # blocking\nuv run python hpo_client.py -m 2 # async\nuv run python hpo_client.py -m 3 # trigger deployed flow\n```\n\nThere are three client modes:\n1. Blocking - `python hpo_client.py -m 1`\n2. Async - `python hpo_client.py -m 2`\n3. Trigger - `python hpo_client.py -m 3` \n    - Trigger option also works with a parameter `--namespace/-n`, which determines the namespace within which this code path checks for already-deployed flows.\n\n### Optuna 101\nThis system is an integration between [Optuna](https://optuna.org/), a feature-rich and open-source hyperparameter optimization framework, and Outerbounds. Using it leverages functionality built-into your Outerbounds deployment to run a persistent relational database that tasks and applications can communicate with. The Optuna dashboard is run as an Outerbounds app, enabling sophisticated analysis of hyperparameter tuning runs.  \n\nThe implementation wraps the standard Optuna interface, aiming to balance two goals:\n1. Provide full expressiveness and compatibility with open-source Optuna features.\n2. Provide an opinionated and streamlined interface for launching HPO studies as Metaflow flows. \n\n#### The objective function\nTypically, Optuna programs are developed in Python scripts. \nAn objective function returns 1 or 2 values. \nIt's argument is a [`trial`](https://optuna.readthedocs.io/en/stable/reference/trial.html), \nrepresenting a single execution of the objective function; in other words, a sample drawn from the hyperparameter search space.\n\n```python\ndef objective(trial):\n    x = trial.suggest_float(\"x\", -100, 100)\n    y = trial.suggest_categorical(\"y\", [-1, 0, 1])\n    f1 = x**2 + y\n    f2 = -((x - 2) ** 2 + y)\n    return f1, f2\n```\n\nThe key task of the user who wishes to use the `from outerbounds.hpo import HPORunner` abstraction this project affords is to determine:\n1. How to define the objective function? \n2. What data, model, and code does the objective function depend on?\n3. How many trials do you want to run per study?\n\nWith answers to these questions, you'll be ready to adapt your objective functions as demonstrated in the example [`flows/`](./flows/) and call the `HPORunner` interface to automate HPO workflows.\n\n#### Note on search spaces\nNotice that with Optuna, the user imperatively defines the hyperparameter space in how the `trial` object is used within the `objective` function.\nThe number of variables for which we have `trial.suggest_*` defines the dimensionality of the search space. \nBe judicious with adding parameters. Many algorithms, especially bayesian optimization suffers performance degradation when there are many more than 5-10 parameters being tuned simultaneously.\n\n[Read more](https://optuna.readthedocs.io/en/stable/tutorial/10_key_features/002_configurations.html#configurations).\n\n#### Studies, samplers, and pruners\nTo optimize the hyperparameters, we create a study.\nOptuna implements many optimization algorithm families, called as [`optuna.samplers`](https://optuna.readthedocs.io/en/stable/reference/samplers/index.html). These include grid, random, tree-structure parzen estimators, evolutionary (CMA-ES, NSGA-II), Gaussian processes, Quasi Monte Carlo methods, and more.\n\nFor example, if you wanted to purely random sample - no learning throughout the study - the hyperparameter space 10 times, you'd run:\n```python\nstudy = optuna.create_study(sampler=optuna.samplers.RandomSampler())   \nstudy.optimize(objective, n_trials=10)\n```\n\nSometimes it is desirable to early stop unpromising trials. The mechanism for doing this in Optuna is called [`optuna.pruners`](https://optuna.readthedocs.io/en/stable/reference/pruners.html), which uses intermediate objective function state variables of previous trials to determine a boolean representing whether the trial should be pruned.\n\n#### Resuming studies\nTo resume a study, simply pass in the name of the previous study. \nIf leveraging the Metaflow versioning scheme which uses the Metaflow Run pathspec as the study name - in other words not overriding the study name via configs or CLI - then\nyou can set this value in the config and resume the study. You can also override in the command line using the `hpo_client`'s `--resume-study/-r` option:\n\n```bash\npython hpo_client.py -m 1 -r TreeModelHpoFlow/argo-hposystem.prod.treemodelhpoflow-7ntvz\n```\n\n## TODO\n- Benchmark gRPC vs. pure RDB scaling thresholds. When is it worth it to do gRPC? How hard is that to implement? How do costs scale in each mode? \n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fouterbounds%2Fhpo-project","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fouterbounds%2Fhpo-project","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fouterbounds%2Fhpo-project/lists"}