{"id":17686775,"url":"https://github.com/lacerbi/infbench","last_synced_at":"2025-04-22T11:52:09.628Z","repository":{"id":43639634,"uuid":"140290726","full_name":"lacerbi/infbench","owner":"lacerbi","description":"Benchmark of posterior and model inference algorithms for (moderately) expensive likelihoods.","archived":false,"fork":false,"pushed_at":"2022-08-11T17:00:21.000Z","size":68590,"stargazers_count":6,"open_issues_count":0,"forks_count":3,"subscribers_count":4,"default_branch":"master","last_synced_at":"2025-03-29T14:51:10.829Z","etag":null,"topics":["approximate-inference","benchmark","computational-neuroscience","model-selection","modeling"],"latest_commit_sha":null,"homepage":null,"language":"MATLAB","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"gpl-3.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/lacerbi.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE.txt","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2018-07-09T13:38:27.000Z","updated_at":"2024-06-20T03:21:36.000Z","dependencies_parsed_at":"2022-08-12T10:30:55.749Z","dependency_job_id":null,"html_url":"https://github.com/lacerbi/infbench","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lacerbi%2Finfbench","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lacerbi%2Finfbench/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lacerbi%2Finfbench/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lacerbi%2Finfbench/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/lacerbi","download_url":"https://codeload.github.com/lacerbi/infbench/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":250237798,"owners_count":21397399,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["approximate-inference","benchmark","computational-neuroscience","model-selection","modeling"],"created_at":"2024-10-24T10:45:56.052Z","updated_at":"2025-04-22T11:52:09.604Z","avatar_url":"https://github.com/lacerbi.png","language":"MATLAB","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Introduction\n\nThis repository contains code for the inference benchmarks and results reported in several papers on *Variational Bayesian Monte Carlo* (VBMC) [[1,2,3](#references)]. \n\nThe Variational Bayesian Monte Carlo (VBMC) algorithm is available at [this repository](https://github.com/lacerbi/vbmc).\n\n# Inference benchmark (`infbench`)\n\nThe goal of `infbench` is to compare various sample-efficient approximate inference algorithms which have been proposed in the machine learning literature so as to deal with (moderately) expensive and potentially noisy likelihoods. In particular, we want to infer the posterior over model parameters and (an approximation of) the model evidence or marginal likelihood, that is the normalization constant of the posterior. Crucially, we assume that the budget is of the order of up to several hundreds likelihood evaluations.\n\nNotably, this goal is more ambitious than simply finding the maximum of the posterior (MAP), problem that we previously tackled with [Bayesian Adaptive Direct Search](https://github.com/lacerbi/bads) (aka BADS).\n\nOur [first benchmark](#how-to-run-the-original-benchmark-vbmc18) shows that existing inference algorithms perform quite poorly at reconstructing posteriors (or evaluating their normalization constant) with both syntethic and real pdfs that have moderately challenging features, showing that this is a much harder problem [[1,2](#reference)].\n\nOur [second extensive benchmark](#how-to-run-the-extensive-benchmark-vbmc20) shows that the latest version of VBMC (v1.0, June 2020) beats other state-of-the-art methods on real computational problems when dealing with *noisy* log-likelihood evaluations, such as those arising from simulation-based estimation techniques [[3](#reference)].\n\n## How to run the original benchmark (`vbmc18`)\n\nYou can run the benchmark on one test problem in the `vbmc18` problem set as follows:\n```\n\u003e options = struct('SpeedTests',false);\n\u003e [probstruct,history] = infbench_run('vbmc18',testfun,D,[],algo,id,options);\n```\nThe arguments are:\n\n- `testfun` (string) is the test pdf, which can be `'cigar'`, `'lumpy'`, `'studentt'` (synthetic test functions with different properties), or `'goris2015'` (real model fitting problem with neuronal data, see [here](https://github.com/lacerbi/infbench/tree/master/matlab/problems/goris2015)).\n- `D` (integer) is the dimensionality of the problem for the synthetic test functions (typical values are from `D=2` to `D=10`), and `7` or `8` for the `goris2015` test set (corresponding to two different neuron datasets).\n- `algo` (string) is the inference algorithm being tested. Specific settings for the chosen inference algorithm are selected with `'algo@setting'`, e.g. `'agp@debug'`.\n- `id` (integer) is the run id, used to set the random seed. It can be an array, such as `id=1:5`, for multiple consecutive runs.\n- `options` (struct) sets various options for the benchmark. For a fast test, I recommend to set the field `SpeedTests` to `false` since initial speed tests can be quite time consuming.\n\nThe outputs are:\n\n- `probstruct` (struct) describes the current problem in detail.\n- `history` (struct array) summary statistics of the run for each id.\n\n## How to run the extensive benchmark (`vbmc20`)\n\nThe `vbm20` benchmark includes a number of real, challenging models and data largely from computational and cognitive neuroscience, from `D = 3` to `D = 9`. The benchmark is mostly designed to test methods that deal with *noisy* log-likelihood evaluations.\n\nYou can run the benchmark on one test problem in the `vbmc20` problem set as follows:\n```\n\u003e options = struct('SpeedTests',false);\n\u003e [probstruct,history] = infbench_run('vbmc20',testfun,D,noise,algo,id,options);\n```\nThe arguments are:\n\n- `testfun` (string) indicates the tested model, which can be `'wood2010'` (Ricker), `'krajbich2010'` (aDDM), `'acerbi2012'` (Timing), `'acerbidokka2018'` (Multisensory), `'goris2015b'` (Neuronal), `'akrami2018b'` (Rodent). The additional model presented in the Supplement is `'price2018'` (g-and-k).\n- `D` (integer) is the dataset, with `D = 1` for Ricker, Timing, Rodent, g-and-k; `D = 1` or `D = 2` for aDDM and Multisensory; `D = 107` or `D = 108` for Neuronal. For all problems (except Neuronal), add 100 to `D` to obtain a noiseless version of the problem.\n- `algo` (string) is the inference algorithm being tested. For the algorithms tested in the paper [[3](#references)], use `'vbmc@imiqr'`, `'vbmc@viqr'`, `'vbmc@npro'`, `'vbmc@eig'`, `'parallelgp@v3'`, `'wsabiplus@ldet'`.\n- `noise` (double) added Gaussian noise to the log-likelihood. Leave this empty (`noise = []`) for all problems except for Neuronal, for which in the benchmark we used `noise = 2` (all other problems in `vbmc20` are intrinsically noisy).\n\nFor the other input and output arguments, see [above](#how-to-run-the-original-benchmark-vbmc18).\n\nCode used to generate figures in the paper [[3](#references)] is available in [this folder](https://github.com/lacerbi/infbench/tree/master/matlab/figs/vbmc_paper2020). However, you will first need to run the benchmark (due to space limitations, we cannot upload the bulk of the numerical results here).\n\n## References\n\n1. Acerbi, L. (2018). Variational Bayesian Monte Carlo. In *Advances in Neural Information Processing Systems 31*: 8222-8232. ([paper + supplement on arXiv](https://arxiv.org/abs/1810.05558), [NeurIPS Proceedings](https://papers.nips.cc/paper/8043-variational-bayesian-monte-carlo))\n2. Acerbi, L. (2019). An Exploration of Acquisition and Mean Functions in Variational Bayesian Monte Carlo. In *Proc. Machine Learning Research* 96: 1-10. 1st Symposium on Advances in Approximate Bayesian Inference, Montréal, Canada. ([paper in PMLR](http://proceedings.mlr.press/v96/acerbi19a.html))\n3. Acerbi, L. (2020). Variational Bayesian Monte Carlo with Noisy Likelihoods. To appear in *Advances in Neural Information Processing Systems 33*. arXiv preprint arXiv:2006.08655 ([preprint on arXiv](https://arxiv.org/abs/2006.08655)).\n### Contact\n\nThis repository is currently actively used for research, stay tuned for updates:\n\n- [Follow me on Twitter](https://twitter.com/AcerbiLuigi) for updates about my work on model inference and other projects I am involved with;\n- If you have questions or comments about this work, get in touch at \u003cluigi.acerbi@helsinki.fi\u003e (putting 'infbench' in the subject of the email).\n\n### License\n\nThe inference benchmark is released under the terms of the [GNU General Public License v3.0](https://github.com/lacerbi/infbench/blob/master/LICENSE.txt).\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Flacerbi%2Finfbench","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Flacerbi%2Finfbench","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Flacerbi%2Finfbench/lists"}