{"id":15547074,"url":"https://github.com/dionhaefner/bayesian-histograms","last_synced_at":"2025-07-06T23:02:20.837Z","repository":{"id":94355969,"uuid":"402037464","full_name":"dionhaefner/bayesian-histograms","owner":"dionhaefner","description":"Bayesian histograms for estimation of binary rare event rates, with fully automated bin pruning :bar_chart:","archived":false,"fork":false,"pushed_at":"2021-10-14T21:09:36.000Z","size":262,"stargazers_count":28,"open_issues_count":0,"forks_count":2,"subscribers_count":2,"default_branch":"main","last_synced_at":"2025-03-30T02:21:37.509Z","etag":null,"topics":["bayesian","binning","data-mining","histogram","rare-events"],"latest_commit_sha":null,"homepage":"","language":"Jupyter Notebook","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/dionhaefner.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2021-09-01T11:34:10.000Z","updated_at":"2024-11-18T00:59:29.000Z","dependencies_parsed_at":"2023-03-08T16:00:30.433Z","dependency_job_id":null,"html_url":"https://github.com/dionhaefner/bayesian-histograms","commit_stats":{"total_commits":17,"total_committers":1,"mean_commits":17.0,"dds":0.0,"last_synced_commit":"f0caff585d2837ae759e798d1d486a4411ff2fdf"},"previous_names":[],"tags_count":3,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dionhaefner%2Fbayesian-histograms","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dionhaefner%2Fbayesian-histograms/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dionhaefner%2Fbayesian-histograms/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dionhaefner%2Fbayesian-histograms/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/dionhaefner","download_url":"https://codeload.github.com/dionhaefner/bayesian-histograms/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":250487824,"owners_count":21438668,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["bayesian","binning","data-mining","histogram","rare-events"],"created_at":"2024-10-02T13:06:27.655Z","updated_at":"2025-04-23T18:22:15.667Z","avatar_url":"https://github.com/dionhaefner.png","language":"Jupyter Notebook","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Bayesian histograms\n\n**Bayesian histograms** are a nifty tool for data mining if:\n\n- you want to know how the *event rate* (probability) of a binary **rare event** depends on a parameter;\n- you have millions or even **billions of data points**, but few positive samples;\n- you suspect the event rate depends **highly non-linearly** on the parameter;\n- you don't know whether you have *enough data*, so you need **uncertainty information**.\n\nThanks to an adaptive bin pruning algorithm, you don't even have to choose the number of bins, and you should get good results out of the box.\n\nThis is how they look in practice ([see full example below](#usage-example)):\n\n\u003cp align=\"center\"\u003e\n\u003cimg src=\"doc/bayesian-histogram-comp.png?raw=true\" width=\"450px\"\u003e\n\u003c/p\u003e\n\n## Installation\n\n```bash\n$ pip install bayeshist\n```\n\n## Usage example\n\nAssume you have binary samples of a rare event like this:\n\n\u003cp align=\"center\"\u003e\n\u003cimg src=\"doc/samples.png?raw=true\" width=\"450px\"\u003e\n\u003c/p\u003e\n\nCompute and plot a Bayesian histogram:\n\n```python\n\u003e\u003e\u003e from bayeshist import bayesian_histogram, plot_bayesian_histogram\n\n# compute Bayesian histogram from samples\n\u003e\u003e\u003e bin_edges, beta_dist = bayesian_histogram(X, y, bins=100, pruning_method=\"bayes\")\n\n# beta_dist is a `scipy.stats.Beta` object, so we can get the\n# predicted mean event rate for each histogram bin like this:\n\u003e\u003e\u003e bin_mean_pred = best_dist.mean()\n\n# plot it up\n\u003e\u003e\u003e plot_bayesian_histogram(bin_edges, beta_dist)\n```\n\nThe result is something like this:\n\n\u003cp align=\"center\"\u003e\n\u003cimg src=\"doc/bayesian-histogram-comp.png?raw=true\" width=\"450px\"\u003e\n\u003c/p\u003e\n\nSee also [demo.ipynb](demo.ipynb) for a full walkthrough of this example.\n\n## But how do they work?\n\n[Here's the blog post.](https://dionhaefner.github.io/2021/09/bayesian-histograms-for-rare-event-classification/)\n\n## API reference\n\n### `bayesian_histogram`\n\n```python\n\ndef bayesian_histogram(\n    x: np.ndarray,\n    y: np.ndarray,\n    bins: Union[int, Iterable] = 100,\n    x_range: Optional[Tuple[float, float]] = None,\n    prior_params: Optional[Tuple[float, float]] = None,\n    pruning_method: Optional[Literal[\"bayes\", \"fisher\"]] = \"bayes\",\n    pruning_threshold: Optional[float] = None,\n    max_bin_size: Optional[float] = None,\n) -\u003e Tuple[np.ndarray, FrozenDistType]:\n    \"\"\"Compute Bayesian histogram for data x, binary target y.\n\n    The output is a Beta distribution over the event rate for each bin.\n\n    Parameters:\n\n        x:\n            1-dim array of data.\n\n        y:\n            1-dim array of binary labels (0 or 1).\n\n        bins:\n            int giving the number of equally spaced intial bins,\n            or array giving initial bin edges. (default: 100)\n\n        x_range:\n            Range spanned by binning. Not used if `bins` is an array.\n            (default: [min(x), max(x)])\n\n        prior_params:\n            Parameters to use in Beta prior. First value relates to positive,\n            second value to negative samples. [0.5, 0.5] represents Jeffrey's prior, [1, 1] a flat\n            prior. The default is a weakly informative prior based on the global event rate.\n            (default: `[1, num_neg / num_pos]`)\n\n        pruning_method:\n            Method to use to decide whether neighboring bins should be merged or not.\n            Valid values are \"bayes\" (Bayes factor), \"fisher\" (exact Fisher test), or None\n            (no pruning). (default: \"bayes\")\n\n        pruning_threshold:\n            Threshold to use in significance test specified by `pruning_method`.\n            (default: 2 for \"bayes\", 0.2 for \"fisher\")\n\n        max_bin_size:\n            Maximum size (in units of x) above which bins will not be merged\n            (except empty bins). (default: unlimited size)\n\n    Returns:\n\n        bin_edges: Coordinates of bin edges\n        beta_dist: n-dimensional Beta distribution (n = number of bins)\n\n    Example:\n\n        \u003e\u003e\u003e x = np.random.randn(1000)\n        \u003e\u003e\u003e p = 10 ** (-2 + x)\n        \u003e\u003e\u003e y = np.random.rand() \u003c p\n        \u003e\u003e\u003e bins, beta_dist = bayesian_histogram(x, y)\n        \u003e\u003e\u003e plt.plot(0.5 * (bins[1:] + bins[:-1]), beta_dist.mean())\n\n    \"\"\"\n```\n\n### `plot_bayesian_histogram`\n\n```python\ndef plot_bayesian_histogram(\n    bin_edges: np.ndarray,\n    data_dist: FrozenDistType,\n    color: Union[str, Iterable[float], None] = None,\n    label: Optional[str] = None,\n    ax: Any = None,\n    ci: Optional[Tuple[float, float]] = (0.01, 0.99)\n) -\u003e None:\n    \"\"\"Plot a Bayesian histogram as horizontal lines with credible intervals.\n\n    Parameters:\n\n        bin_edges:\n            Coordinates of bin edges\n\n        data_dist:\n            n-dimensional Beta distribution (n = number of bins)\n\n        color:\n            Color to use (default: use next in current color cycle)\n\n        label:\n            Legend label (default: no label)\n\n        ax:\n            Matplotlib axis to use (default: current axis)\n\n        ci:\n            Credible interval used for shading, use `None` to disable shading.\n\n    Example:\n\n        \u003e\u003e\u003e x = np.random.randn(1000)\n        \u003e\u003e\u003e p = 10 ** (-2 + x)\n        \u003e\u003e\u003e y = np.random.rand() \u003c p\n        \u003e\u003e\u003e bins, beta_dist = bayesian_histogram(x, y)\n        \u003e\u003e\u003e plot_bayesian_histogram(bins, beta_dist)\n\n    \"\"\"\n```\n\n## Questions?\n\n[Feel free to open an issue.](https://github.com/dionhaefner/bayesian-histograms/issues)\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdionhaefner%2Fbayesian-histograms","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fdionhaefner%2Fbayesian-histograms","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdionhaefner%2Fbayesian-histograms/lists"}