{"id":13478287,"url":"https://github.com/deephyper/benchmark","last_synced_at":"2025-04-13T03:32:25.835Z","repository":{"id":46235493,"uuid":"414886938","full_name":"deephyper/benchmark","owner":"deephyper","description":"Repository to benchmark DeepHyper on different systems.","archived":false,"fork":false,"pushed_at":"2024-04-04T08:19:16.000Z","size":2050,"stargazers_count":4,"open_issues_count":6,"forks_count":3,"subscribers_count":3,"default_branch":"main","last_synced_at":"2024-10-30T11:42:04.455Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"bsd-2-clause","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/deephyper.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null}},"created_at":"2021-10-08T07:22:35.000Z","updated_at":"2024-06-04T08:23:12.000Z","dependencies_parsed_at":"2023-10-25T18:33:16.556Z","dependency_job_id":"2bdbaaaf-8763-4f85-8786-5cbc925360b3","html_url":"https://github.com/deephyper/benchmark","commit_stats":null,"previous_names":[],"tags_count":1,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/deephyper%2Fbenchmark","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/deephyper%2Fbenchmark/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/deephyper%2Fbenchmark/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/deephyper%2Fbenchmark/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/deephyper","download_url":"https://codeload.github.com/deephyper/benchmark/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":223565596,"owners_count":17166145,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-07-31T16:01:55.030Z","updated_at":"2024-11-07T18:22:32.665Z","avatar_url":"https://github.com/deephyper.png","language":"Python","readme":"# DeepHyper Benchmark\n\n## Table of Contents\n\n- [DeepHyper Benchmark](#deephyper-benchmark)\n  - [Table of Contents](#table-of-contents)\n  - [Introduction](#introduction)\n  - [Organization of the Repository](#organization-of-the-repository)\n  - [Installation](#installation)\n  - [Defining a Benchmark](#defining-a-benchmark)\n  - [Standard Metadata](#standard-metadata)\n  - [List of Benchmarks](#list-of-benchmarks)\n\n## Introduction\n\nThis repository is a collection of machine learning benchmark for DeepHyper.\n\n## Organization of the Repository\n\nThe repository follows this organization:\n\n```bash\n# Python package containing utility code\ndeephyper_benchmark/\n\n# Library of benchmarks\nlib/\n```\n\n## Installation\n\nTo install the DeepHyper benchmark suite, run:\n\n```console\ngit clone https://github.com/deephyper/benchmark.git deephyper_benchmark\ncd deephyper_benchmark/\npip install -e \".\"\n```\n\n## Defining a Benchmark\n\nA benchmark is defined as a sub-folder of the `lib/` folder such as `lib/Benchmark-101/`. Then a benchmark folder needs to follow a python package structure and therefore it needs to contain a `__init__.py` file at its root. In addition, a benchmark folder needs to define a `benchmark.py` script that defines its requirements.\n\nGeneral benchmark structure:\n```\nlib/\n    Benchmark-101/\n        __init__.py\n        benchmark.py\n        data.py\n        model.py\n        hpo.py # Defines hyperparameter optimization inputs (run-function + problem)\n        README.md # Description of the benchmark\n```\n\nThen to use the benchmark:\n\n```python\nimport deephyper_benchmark as dhb\n\ndhb.install(\"Benchmark-101\")\n\ndhb.load(\"Benchmark-101\")\n\nfrom deephyper_benchmark.lib.benchmark_101.hpo import problem, run\n```\n\nAll `run`-functions (i.e., functions returning the objective(s) to be optimized) should follow the **MAXIMIZATION** standard. If a benchmark needs minimization then the negative of the minimized objective can be returned `return -minimized_objective`.\n\nA benchmark inherits from the `Benchmark` class:\n\n```python\nimport os\n\nfrom deephyper_benchmark import *\n\nDIR = os.path.dirname(os.path.abspath(__file__))\n\n\nclass Benchmark101(Benchmark):\n\n    version = \"0.0.1\"\n\n    requires = {\n        \"bash-install\": {\"type\": \"cmd\", \"cmd\": \"cd .. \u0026\u0026 \" + os.path.join(DIR, \"../install.sh\")},\n    }\n\n```\n\nFinally, when testing a benchmark it can be useful to activate the logging:\n\n```python\nimport logging\n\nlogging.basicConfig(\n    # filename=\"deephyper.log\", # Uncomment if you want to create a file with the logs\n    level=logging.INFO,\n    format=\"%(asctime)s - %(levelname)s - %(filename)s:%(funcName)s - %(message)s\",\n    force=True,\n)\n```\n\n## Configuration\n\nBenchmarks can sometimes be configured. The configuration can use environment variables with the prefix `DEEPHYPER_BENCHMARK_`.\n\n## Standard Metadata\n\nBenchmarks must return the following standard metadata when it applies, some metadata are specific to neural networks (e.g., `num_parameters`):\n\n- [ ] `num_parameters`: integer value of the number of parameters in the neural network.\n- [ ] `num_parameters_train`: integer value of the number of **trainable** parameters of the neural network.\n- [ ] `budget`: scalar value (float/int) of the budget consumed by the neural network. Therefore the budget should be defined for each benchmark (e.g., number of epochs in general).\n- [ ] `stopped`: boolean value indicating if the evaluation was stopped before consuming the maximum budget.\n- `train_X`:  scalar value of the training metrics (replace `X` by the metric name, 1 key per metric).\n- `valid_X`: scalar value of the validation metrics (replace `X` by the metric name, 1 key per metric).\n- `test_X`: scalar value of the testing metrics (replace `X` by the metric name, 1 key per metric).\n- [ ] `flops`: number of flops of the model such as computed in `fvcore.nn.FlopCountAnalysis(...).total()` (See [documentation](https://detectron2.readthedocs.io/en/latest/modules/fvcore.html#module-fvcore.nn)).\n- [ ] `latency`: TO BE CLARIFIED\n- [ ] `lc_train_X`: recorded learning curves of the trained model, the `bi` variables are the budget value (e.g., epochs/batches), and the `yi` values are the recorded metric. `X` in `train_X` is replaced by the name of the metric such as `train_loss` or `train_accuracy`. The format is `[[b0, y0], [b1, y1], ...]`.\n- [ ] `lc_valid_X`: Same as `lc_train_X` but for validation data.\n\n\nThe `@profile` decorator should be used on all `run`-functions to collect the `timestamp_start` and `timestamp_end` metadata.\n\n## List of Benchmarks\n\nIn the following table:\n\n- $\\mathbb{R}$ denotes real parameters.\n- $\\mathbb{D}$ denotes discrete parameters.\n- $\\mathbb{C}$ denotes categorical parameters.\n\n| Name       | Description                                                                  | Variable(s) Type                             | Objective(s) Type | Multi-Objective | Multi-Fidelity | Evaluation Duration |\n| ---------- | ---------------------------------------------------------------------------- | -------------------------------------------- | ----------------- | --------------- | -------------- | ------------------- |\n| C-BBO      | Continuous Black-Box Optimization problems.                                  | $\\mathbb{R}^n$                               | $\\mathbb{R}$      | ❌              | ❌             | configurable        |\n| DTLZ       | The modified DTLZ multiobjective test suite.                                 |  $\\mathbb{R}^n$                              |  $\\mathbb{R}$     | ✅              |  ❌            | configurable        |\n| ECP-Candle | Deep Neural-Networks on multiple \"biological\" scales of Cancer related data. | $\\mathbb{R}\\times\\mathbb{D}\\times\\mathbb{C}$ | $\\mathbb{R}$      | ✅              | ✅             | min                 |\n| HPOBench   | Hyperparameter Optimization Benchmark.                                       | $\\mathbb{R}\\times\\mathbb{D}\\times\\mathbb{C}$ | $\\mathbb{R}$      | ✅              | ✅             | ms to min           |\n| JAHSBench  | A slightly modified JAHSBench 201 wrapper.                                   |  $\\mathbb{R}^2\\times\\mathbb{D}\\times\\mathbb{C}^8$ | $\\mathbb{R}$ | ✅              |  ❌            | configurable        |\n| LCu        | Learning curve hyperparameter optimization benchmark.                        |                                              |                   |                 |                |                     |\n| LCbench    | Multi-fidelity benchmark without hyperparameter optimization.                | NA                                           | $\\mathbb{R}$      | ❌              | ✅             | secondes            |\n| PINNBench  | Physics Informed Neural Networks Benchmark.                                  | $\\mathbb{R}\\times\\mathbb{D}\\times\\mathbb{C}$ | $\\mathbb{R}$      | ✅              | ✅             | ms                  |\n|            |                                                                              |                                              |                   |                 |                |                     |\n      \n      \n## List of Optimization Algorithm\n\n- COBYQA: `deephyper_benchmark.search.COBYQA(...)`\n- PyBOBYQA: `deephyper_benchmark.search.PyBOBYQA(...)`\n- TPE: `deephyper_benchmark.search.MPIDistributedOptuna(..., sampler=\"TPE\")`\n- BoTorch: `deephyper_benchmark.search.MPIDistributedOptuna(..., sampler=\"BOTORCH\")`\n- CMAES: `deephyper_benchmark.search.MPIDistributedOptuna(..., sampler=\"CMAES\")`\n- NSGAII: `deephyper_benchmark.search.MPIDistributedOptuna(..., sampler=\"NSGAII\")`\n- QMC: `deephyper_benchmark.search.MPIDistributedOptuna(..., sampler=\"QMC\")`\n- SMAC: `deephyper_benchmark.search.SMAC(...)`\n\n      \n      \n      \n  \n      \n      \n      \n      \n      \n      \n  \n      \n      \n      \n      \n","funding_links":[],"categories":["Python"],"sub_categories":[],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdeephyper%2Fbenchmark","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fdeephyper%2Fbenchmark","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdeephyper%2Fbenchmark/lists"}