{"id":13699449,"url":"https://github.com/sbi-benchmark/sbibm","last_synced_at":"2025-05-04T16:34:28.308Z","repository":{"id":40262833,"uuid":"328708749","full_name":"sbi-benchmark/sbibm","owner":"sbi-benchmark","description":"Simulation-based inference benchmark","archived":false,"fork":false,"pushed_at":"2024-03-17T20:15:12.000Z","size":18182,"stargazers_count":89,"open_issues_count":13,"forks_count":35,"subscribers_count":5,"default_branch":"main","last_synced_at":"2024-11-07T18:54:13.566Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"https://sbi-benchmark.github.io","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/sbi-benchmark.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null}},"created_at":"2021-01-11T15:31:01.000Z","updated_at":"2024-09-16T15:04:59.000Z","dependencies_parsed_at":"2024-01-12T19:52:53.191Z","dependency_job_id":"3dc875f7-4b92-44b3-ad4e-f29ec2f17c56","html_url":"https://github.com/sbi-benchmark/sbibm","commit_stats":{"total_commits":63,"total_committers":6,"mean_commits":10.5,"dds":"0.25396825396825395","last_synced_commit":"302f0bf608cb4f9e2ad2f6c813fe0bb8aeb39204"},"previous_names":[],"tags_count":10,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sbi-benchmark%2Fsbibm","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sbi-benchmark%2Fsbibm/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sbi-benchmark%2Fsbibm/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sbi-benchmark%2Fsbibm/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/sbi-benchmark","download_url":"https://codeload.github.com/sbi-benchmark/sbibm/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":224398825,"owners_count":17304661,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-08-02T20:00:33.489Z","updated_at":"2025-05-04T16:34:28.292Z","avatar_url":"https://github.com/sbi-benchmark.png","language":"Python","funding_links":[],"categories":["\u003cspan id=\"head30\"\u003e3.4. Bayesian Inference\u003c/span\u003e","Code Packages and Benchmarks"],"sub_categories":["\u003cspan id=\"head32\"\u003e3.4.2. Approximate Bayesian Computation (ABC)\u003c/span\u003e"],"readme":"[![PyPI\nversion](https://img.shields.io/pypi/v/sbibm)](https://pypi.org/project/sbibm/) ![Python versions](https://img.shields.io/pypi/pyversions/sbibm) [![Contributions welcome](https://img.shields.io/badge/contributions-welcome-brightgreen.svg?style=flat)](https://github.com/sbi-benchmark/sbibm/blob/master/CONTRIBUTING.md) [![Black](https://img.shields.io/badge/code%20style-black-000000.svg?style=flat)](https://github.com/psf/black)\n\n# Simulation-Based Inference Benchmark\n\nThis repository contains a simulation-based inference benchmark framework, `sbibm`, which we describe in the [associated manuscript \"Benchmarking Simulation-based Inference\"](http://proceedings.mlr.press/v130/lueckmann21a.html). A short summary of the paper and interactive results can be found on the project website: https://sbi-benchmark.github.io\n\nThe benchmark framework includes tasks, reference posteriors, metrics, plotting, and integrations with SBI toolboxes. The framework is designed to be highly extensible and easily used in new research projects as we show below.\n\nIn order to emphasize that `sbibm` can be used independently of any particular analysis pipeline, we split the code for reproducing the experiments of the manuscript into a seperate repository hosted at [github.com/sbi-benchmark/results/](https://github.com/sbi-benchmark/results/tree/main/benchmarking_sbi). Besides the pipeline to reproduce the manuscripts' experiments, full results including dataframes for quick comparisons are hosted in that repository.\n\nIf you have questions or comments, please do not hesitate [to contact us](mailto:mail@jan-matthis.de) or [open an issue](https://github.com/sbi-benchmark/sbibm/issues). We [invite contributions](CONTRIBUTING.md), e.g., of new tasks, novel metrics, or wrappers for other SBI toolboxes.\n\n\n## Installation\n\nAssuming you have a working Python environment, simply install `sbibm` via `pip`:\n```commandline\n$ pip install sbibm\n```\n\nODE based models (currently SIR and Lotka-Volterra models) use [Julia](https://julialang.org) via [`diffeqtorch`](https://github.com/sbi-benchmark/diffeqtorch). If you are planning to use these tasks, please additionally follow the [installation instructions of `diffeqtorch`](https://github.com/sbi-benchmark/diffeqtorch#installation). If you are not planning to simulate these tasks for now, you can skip this step.\n\n## Quickstart\n\nA quick demonstration of `sbibm`, see further below for more in-depth explanations:\n\n```python\nimport sbibm\n\ntask = sbibm.get_task(\"two_moons\")  # See sbibm.get_available_tasks() for all tasks\nprior = task.get_prior()\nsimulator = task.get_simulator()\nobservation = task.get_observation(num_observation=1)  # 10 per task\n\n# These objects can then be used for custom inference algorithms, e.g.\n# we might want to generate simulations by sampling from prior:\nthetas = prior(num_samples=10_000)\nxs = simulator(thetas)\n\n# Alternatively, we can import existing algorithms, e.g:\nfrom sbibm.algorithms import rej_abc  # See help(rej_abc) for keywords\nposterior_samples, _, _ = rej_abc(task=task, num_samples=10_000, num_observation=1, num_simulations=100_000)\n\n# Once we got samples from an approximate posterior, compare them to the reference:\nfrom sbibm.metrics import c2st\nreference_samples = task.get_reference_posterior_samples(num_observation=1)\nc2st_accuracy = c2st(reference_samples, posterior_samples)\n\n# Visualise both posteriors:\nfrom sbibm.visualisation import fig_posterior\nfig = fig_posterior(task_name=\"two_moons\", observation=1, samples=[posterior_samples])  \n# Note: Use fig.show() or fig.save() to show or save the figure\n\n# Get results from other algorithms for comparison:\nfrom sbibm.visualisation import fig_metric\nresults_df = sbibm.get_results(dataset=\"main_paper.csv\")\nfig = fig_metric(results_df.query(\"task == 'two_moons'\"), metric=\"C2ST\")\n```\n\n## Tasks\n\nYou can then see the list of available tasks by calling `sbibm.get_available_tasks()`. If we wanted to use, say, the `two_moons` task, we can load it using `sbibm.get_task`, as in:\n\n```python\nimport sbibm\ntask = sbibm.get_task(\"slcp\")\n```\n\nNext, we might want to get `prior` and `simulator`:\n\n```python\nprior = task.get_prior()\nsimulator = task.get_simulator()\n```\n\nIf we call `prior()` we get a single draw from the prior distribution. `num_samples` can be provided as an optional argument. The following would generate 100 samples from the simulator:\n```python\nthetas = prior(num_samples=100)\nxs = simulator(thetas)\n```\n\n`xs` is a `torch.Tensor` with shape `(100, 8)`, since for SLCP the data is eight-dimensional. Note that if required, conversion to and from `torch.Tensor` is very easy: Convert to a numpy array using `.numpy()`, e.g., `xs.numpy()`. For the reverse, use `torch.from_numpy()` on a numpy array.\n\nSome algorithms might require evaluating the pdf of the prior distribution, which can be obtained as a [`torch.Distribution` instance](https://pytorch.org/docs/stable/distributions.html) using `task.get_prior_dist()`, which exposes `log_prob` and `sample` methods. The parameters of the prior can be picked up as a dictionary as parameters using `task.get_prior_params()`.\n\nFor each task, the benchmark contains 10 observations and respective reference posteriors samples. To fetch the first observation and respective reference posterior samples:\n```python\nobservation = task.get_observation(num_observation=1)\nreference_samples = task.get_reference_posterior_samples(num_observation=1)\n```\n\nEvery tasks has a couple of informative attributes, including:\n\n```python\ntask.dim_data               # dimensionality data, here: 8\ntask.dim_parameters         # dimensionality parameters, here: 5\ntask.num_observations       # number of different observations x_o available, here: 10\ntask.name                   # name: slcp\ntask.name_display           # name_display: SLCP\n```\n\nFinally, if you want to have a look at the source code of the task, take a look in `sbibm/tasks/slcp/task.py`. If you wanted to implement a new task, we would recommend modelling them after the existing ones. You will see that each task has a private `_setup` method that was used to generate the reference posterior samples.\n\n\n## Algorithms\n\nAs mentioned in the intro, `sbibm` wraps a number of third-party packages to run various algorithms. We found it easiest to give each algorithm the same interface: In general, each algorithm specifies a `run` function that gets `task` and hyperparameters as arguments, and eventually returns the required `num_posterior_samples`. That way, one can simply import the run function of an algorithm, tune it on any given task, and return metrics on the returned samples. Wrappers for external toolboxes implementing algorithms are in the subfolder `sbibm/algorithms`. Currently, integrations with [`sbi`](https://www.mackelab.org/sbi/), [`pyabc`](https://pyabc.readthedocs.io), [`pyabcranger`](https://github.com/diyabc/abcranger), as well as an experimental integration with [`elfi`](https://github.com/elfi-dev/elfi) are provided.\n\n\n## Metrics\n\nIn order to compare algorithms on the benchmarks, a number of different metrics can be computed. Each task comes with reference samples for each observation. Depending on the benchmark, these are either obtained by making use of an analytic solution for the posterior or a customized likelihood-based approach.\n\nA number of metrics can be computed by comparing algorithm samples to reference samples. In order to do so, a number of different two-sample tests can be computed (see `sbibm/metrics`). These test follow a simple interface, just requiring to pass samples from reference and algorithm.\n\nFor example, in order to compute C2ST:\n```python\nimport torch\nfrom sbibm.metrics.c2st import c2st\nfrom sbibm.algorithms import rej_abc\n\nreference_samples = task.get_reference_posterior_samples(num_observation=1)\nalgorithm_samples, _, _ = rej_abc(task=task, num_samples=10_000, num_simulations=100_000, num_observation=1)\nc2st_accuracy = c2st(reference_samples, algorithm_samples)\n```\n\nFor more info, see `help(c2st)`.\n\n\n## Figures\n\n`sbibm` includes code for plotting results, for instance, to plot metrics on a specific task:\n\n```python\nfrom sbibm.visualisation import fig_metric\n\nresults_df = sbibm.get_results(dataset=\"main_paper.csv\")\nresults_subset = results_df.query(\"task == 'two_moons'\")\nfig = fig_metric(results_subset, metric=\"C2ST\")  # Use fig.show() or fig.save() to show or save the figure\n```\n\nIt can also be used to plot posteriors, e.g., to compare the results of an inference algorithm against reference samples:\n\n```python\nfrom sbibm.visualisation import fig_posterior\nfig = fig_posterior(task_name=\"two_moons\", observation=1, samples=[algorithm_samples])\n```\n\n\n## Results and Experiments\n\nWe host results and the code for reproducing the experiments of the manuscript in a seperate repository at [github.com/sbi-benchmark/results](https://github.com/sbi-benchmark/results/tree/main/benchmarking_sbi): This includes the pipeline to reproduce the manuscripts' experiments as well as dataframes for new comparisons.\n\n\n## Corrections\n\n- [`sbibm v1.1.0` contains a bug fix for the Gaussian Mixture task](https://github.com/sbi-benchmark/sbibm/releases/tag/v1.1.0). [Results from rerunning the task](https://github.com/sbi-benchmark/results/tree/main/gaussian_mixture_rerun).\n\n\n## Citation\n\nThe manuscript is [available through PMLR](http://proceedings.mlr.press/v130/lueckmann21a.html):\n\n```bibtex\n @InProceedings{lueckmann2021benchmarking,\n  title     = {Benchmarking Simulation-Based Inference},\n  author    = {Lueckmann, Jan-Matthis and Boelts, Jan and Greenberg, David and Goncalves, Pedro and Macke, Jakob},\n  booktitle = {Proceedings of The 24th International Conference on Artificial Intelligence and Statistics},\n  pages     = {343--351},\n  year      = {2021},\n  editor    = {Banerjee, Arindam and Fukumizu, Kenji},\n  volume    = {130},\n  series    = {Proceedings of Machine Learning Research},\n  month     = {13--15 Apr},\n  publisher = {PMLR}\n}  \n```\n\n\n## Support\n\nThis work was supported by the German Research Foundation (DFG; SFB 1233 PN 276693517, SFB 1089, SPP 2041, Germany’s Excellence Strategy – EXC number 2064/1 PN 390727645) and the German Federal Ministry of Education and Research (BMBF; project ’[ADIMEM](https://fit.uni-tuebingen.de/Project/Details?id=9199)’, FKZ 01IS18052 A-D).\n\n\n## License\n\nMIT\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsbi-benchmark%2Fsbibm","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fsbi-benchmark%2Fsbibm","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsbi-benchmark%2Fsbibm/lists"}