{"id":20841026,"url":"https://github.com/benchopt/benchmark_bilevel","last_synced_at":"2025-08-01T15:33:30.669Z","repository":{"id":69250269,"uuid":"438308978","full_name":"benchopt/benchmark_bilevel","owner":"benchopt","description":"Benchmark for bi-level optimization solvers","archived":false,"fork":false,"pushed_at":"2025-05-22T14:32:55.000Z","size":391,"stargazers_count":46,"open_issues_count":5,"forks_count":10,"subscribers_count":5,"default_branch":"main","last_synced_at":"2025-05-22T15:47:22.298Z","etag":null,"topics":["bilevel-optimization","datacleaning","hyperparameter-optimization"],"latest_commit_sha":null,"homepage":"https://benchopt.github.io/results/benchmark_bilevel.html","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/benchopt.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2021-12-14T15:42:41.000Z","updated_at":"2025-05-22T14:35:42.000Z","dependencies_parsed_at":"2023-10-03T11:43:34.323Z","dependency_job_id":"3a82ed90-e555-426d-8d62-699bd0e27849","html_url":"https://github.com/benchopt/benchmark_bilevel","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/benchopt/benchmark_bilevel","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/benchopt%2Fbenchmark_bilevel","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/benchopt%2Fbenchmark_bilevel/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/benchopt%2Fbenchmark_bilevel/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/benchopt%2Fbenchmark_bilevel/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/benchopt","download_url":"https://codeload.github.com/benchopt/benchmark_bilevel/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/benchopt%2Fbenchmark_bilevel/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":268250301,"owners_count":24219802,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-08-01T02:00:08.611Z","response_time":67,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["bilevel-optimization","datacleaning","hyperparameter-optimization"],"created_at":"2024-11-18T01:18:33.808Z","updated_at":"2025-08-01T15:33:30.645Z","avatar_url":"https://github.com/benchopt.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"Bilevel Optimization Benchmark\n===============================\n[![test](https://github.com/benchopt/benchmark_bilevel/workflows/Tests/badge.svg)](https://github.com/benchopt/benchmark_bilevel/actions)\n[![python](https://img.shields.io/badge/python-3.6%2B-blue)](https://www.python.org/downloads/release/python-360/)\n\n*Results can be consulted on https://benchopt.github.io/results/benchmark_bilevel.html*\n\nBenchOpt is a package to simplify, to make more transparent, and\nreproducible the comparison of optimization algorithms.\nThis benchmark is dedicated to solvers for bilevel optimization:\n\n$$\\min_{x} f(x, z^* (x)) \\quad \\text{with} \\quad z^*(x) = \\arg\\min_z g(x, z),$$\n\nwhere $g$ and $f$ are two functions of two variables.\n\nDifferent problems\n------------------\n\nThis benchmark implements three bilevel optimization problems: quadratic problem, regularization selection, and data cleaning.\n\n### 1 - Simulated quadratic bilevel problem\n\n\nIn this problem, the inner and the outer functions are quadratic functions defined on $\\mathbb{R}^{d\\times p}$\n\n$$g(x, z) = \\frac{1}{n}\\sum_{i=1}^n \\frac{1}{2} z^\\top A_i z + \\frac{1}{2} x^\\top B_i x + x^\\top C_i z + a_i^\\top z + b_i^\\top x$$\n\nand\n\n$$f(x, z) = \\frac{1}{m} \\sum_{j=1}^m \\frac{1}{2} z^\\top F_j z + \\frac{1}{2} x^\\top H_j x + x^\\top K_j z + f_j^\\top z + h_j^\\top x$$\n\nwhere $A_i, F_j$ are symmetric positive definite matrices of size $p\\times p$, $B_i, F_j$ are symmetric positive definite matrices of size $d\\times d$, $C_i, K_j$ are matrices of size $d\\times p$, $a_i$, $f_j$ are vectors of size $d$, and $b_i, h_j$ are vectors of size $p$.\n\nThe matrices $A_i, B_i, F_j, H_j$ are randomly generated such that the eigenvalues of $\\frac1n\\sum_i A_i$ are between ``mu_inner``, and ``L_inner_inner``, the eigenvalues of $\\frac1n\\sum_i B_i$ are between ``mu_inner``, and ``L_inner_outer``, the eigenvalues of $\\frac1m\\sum_j F_j$ are between ``mu_inner``, and ``L_outer_inner``, and the eigenvalues of $\\frac1m\\sum_j H_j$ are between ``mu_inner``, and ``L_outer_outer``.\n\nThe matrices $C_i, K_j$ are generated randomly such that the spectral norm of $\\frac1n\\sum_i C_i$ is lower than ``L_cross_inner``, and the spectral norm of $\\frac1m\\sum_j K_j$ is lower than ``L_cross_outer``.\n\nNote that in this setting, the solution of the inner problem is a linear system.\nAs the full batch inner and outer functions can be computed efficiently with the average Hessian matrices, the value function is evaluated in closed form. \n\n\n### 2 - Regularization selection\n\nIn this problem, the inner function $g$ is defined by \n\n\n$$g(x, z) = \\frac{1}{n} \\sum_{i=1}^{n} \\ell(d_i; z) + \\mathcal{R}(x, z)$$\n\nwhere $d_1, \\dots, d_n$ are training data samples, $z$ are the parameters of the machine learning model, and the loss function $\\ell$ measures how well the model parameters $z$ predict the data $d_i$.\nThere is also a regularization $\\mathcal{R}$ that is parametrized by the regularization strengths $x$, which aims at promoting a certain structure on the parameters $z$.\n\nThe outer function $f$ is defined as the unregularized loss on unseen data\n\n$$f(x, z) = \\frac{1}{m} \\sum_{j=1}^{m} \\ell(d'_j; z)$$\n\nwhere the $d'_1, \\dots, d'_m$ are new samples from the same dataset as above.\n\nThere are currently two datasets for this regularization selection problem.\n\n#### Covtype - [*Homepage*](https://archive.ics.uci.edu/dataset/31/covertype*)\n\nThis is a logistic regression problem, where the data have the form $d_i = (a_i, y_i)$ with $a_i\\in\\mathbb{R}^p$ the features and $y_i=\\pm1$ the binary target.\nFor this problem, the loss is $\\ell(d_i, z) = \\log(1+\\exp(-y_i a_i^T z))$, and the regularization is simply given by\n$$\\mathcal{R}(x, z) = \\frac12\\sum_{j=1}^p\\exp(x_j)z_j^2,$$\neach coefficient in $z$ is independently regularized with the strength $\\exp(x_j)$.\n\n#### Ijcnn1 - [*Homepage*](https://www.openml.org/search?type=data\u0026sort=runs\u0026id=1575\u0026status=active)\n\nThis is a multiclass logistic regression problem, where the data is of the form $d_i = (a_i, y_i)$ with  $a_i\\in\\mathbb{R}^p$ are the features and $y_i\\in \\{1,\\dots, k\\}$ is the integer target, with k the number of classes.\nFor this problem, the loss is $\\ell(d_i, z) = \\text{CrossEntropy}(za_i, y_i)$ where $z$ is now a k x p matrix. The regularization is given by \n$$\\mathcal{R}(x, z) = \\frac12\\sum_{j=1}^k\\exp(x_j)\\|z_j\\|^2,$$\neach line in $z$ is independently regularized with the strength $\\exp(x_j)$.\n\n\n### 3 - Data cleaning\n\nThis problem was first introduced by [Franceschi et al. (2017)](https://arxiv.org/abs/1703.01785).\nIn this problem, the data is the MNIST dataset.\nThe training set has been corrupted: with a probability $p$, the label of the image $`y\\in\\{1,\\dots,10\\}`$ is replaced by another random label between 1 and 10.\nWe do not know beforehand which data has been corrupted.\nWe have a clean testing set, which has not been corrupted.\nThe goal is to fit a model on the corrupted training data that has good performances on the test set.\nTo do so, a set of weights -- one per train sample -- is learned as well as the model parameters.\nIdeally, we would want a weight of 0 for data that has been corrupted and a weight of 1 for uncorrupted data.\nThe problem is cast as a bilevel problem with $g$ given by \n\n$$g(x, z) =\\frac1n \\sum_{i=1}^n \\sigma(x_i)\\ell(d_i, z) + \\frac C 2 \\|z\\|^2$$\n\nwhere the $d_i$ are the corrupted training data, $\\ell$ is the loss of a CNN parameterized by $z$, $\\sigma$ is a sigmoid function, and C is a small regularization constant.\nHere the outer variable $x$ is a vector of dimension $n$, and the weight of data $i$ is given by $\\sigma(x_i)$.\nThe test function is\n\n$$f(x, z) =\\frac1m \\sum_{j=1}^n \\ell(d'_j, z)$$\n\nwhere the $d_j$ are uncorrupted testing data.\n\nInstall\n--------\n\nThis benchmark can be run using the following commands:\n\n```bash\n   $ pip install -U benchopt\n   $ git clone https://github.com/benchopt/benchmark_bilevel\n   $ benchopt run benchmark_bilevel\n```\n\nApart from the problem, options can be passed to ``benchopt run`` to restrict the benchmarks to some solvers or datasets, e.g.:\n\n```bash\n   $ benchopt run benchmark_bilevel -s solver1 -d dataset2 --max-runs 10 --n-repetitions 10\n````\n\nYou can also use config files to set the benchmark run:\n\n```bash\n   $ benchopt run benchmark_bilevel --config config/X.yml\n```\n\nwhere ``X.yml`` is a config file. See https://benchopt.github.io/index.html#run-a-benchmark for an example of a config file. This will launch a huge grid search. When available, you can rather use the file ``X_best_params.yml`` to launch an experiment with a single set of parameters for each solver.\n\nUse ``benchopt run -h`` for more details about these options, or visit https://benchopt.github.io/api.html.\n\n### How to contribute to the benchmark?\n\nIf you want to add a solver or a new problem, you are welcome to open an issue or submit a pull request!  \n\n#### 1 - How to add a new solver?\n\nEach solver derives from the [`benchopt.BaseSolver` class](https://benchopt.github.io/user_guide/generated/benchopt.BaseSolver.html) in the [solvers](solvers) folder. The solvers are separated among the stochastic JAX solvers and the others:\n* Stochastic Jax solver: these solvers inherit from the [`StochasticJaxSolver` class](benchmark_utils/stochastic_jax_solver.py) see the detailed explanations in the [template stochastic solver](solvers/template_stochastic_solver.py).\n* Other solver: see the detailed explanation in the [Benchopt documentation](https://benchopt.github.io/tutorials/add_solver.html). An example is provided in the [template solver](solvers/template_solver.py).\n\n#### 2 - How to add a new problem?\n\nIn this benchmark, each problem is defined by a [Dataset class](https://benchopt.github.io/user_guide/generated/benchopt.BaseDataset.html) in the [datasets](datasets) folder. A [template](datasets/template_dataset.py) is provided.\n\nCite\n----\n\nIf you use this benchmark in your research project, please cite the following paper:\n\n```\n   @inproceedings{dagreou2022,\n      title = {A Framework for Bilevel Optimization That Enables Stochastic and Global Variance Reduction Algorithms},\n      booktitle = {Advances in {{Neural Information Processing Systems}} ({{NeurIPS}})},\n      author = {Dagr{\\'e}ou, Mathieu and Ablin, Pierre and Vaiter, Samuel and Moreau, Thomas},\n      year = {2022}\n   }\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fbenchopt%2Fbenchmark_bilevel","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fbenchopt%2Fbenchmark_bilevel","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fbenchopt%2Fbenchmark_bilevel/lists"}