https://github.com/statmixedml/py-boostlss
An extension of Py-Boost to probabilistic modelling
https://github.com/statmixedml/py-boostlss
distributional-regression gamlss gradient-boosting-machine machine-learning multi-target-regression probabilistic-modeling quantile-estimation uncertainty-estimation
Last synced: 3 months ago
JSON representation
An extension of Py-Boost to probabilistic modelling
- Host: GitHub
- URL: https://github.com/statmixedml/py-boostlss
- Owner: StatMixedML
- License: apache-2.0
- Created: 2022-12-01T08:28:36.000Z (over 3 years ago)
- Default Branch: main
- Last Pushed: 2023-01-19T11:15:36.000Z (over 3 years ago)
- Last Synced: 2024-05-14T00:07:16.953Z (about 2 years ago)
- Topics: distributional-regression, gamlss, gradient-boosting-machine, machine-learning, multi-target-regression, probabilistic-modeling, quantile-estimation, uncertainty-estimation
- Language: Python
- Homepage:
- Size: 5.07 MB
- Stars: 20
- Watchers: 4
- Forks: 0
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
- License: LICENSE
Awesome Lists containing this project
README
# Py-BoostLSS: An extension of Py-Boost to probabilistic modelling
We present a probabilistic extension of the recently introduced [Py-Boost](https://github.com/sb-ai-lab/Py-Boost) approach and model all moments of a parametric multivariate distribution as functions of covariates. This allows us to create probabilistic predictions from which intervals and quantiles of interest can be derived.
## Motivation
Existing implementations of Gradient Boosting Machines, such as [XGBoost](https://github.com/dmlc/xgboost) and [LightGBM](https://github.com/microsoft/LightGBM), are mostly designed for single-target regression tasks. While efficient for low to medium target-dimensions, the computational cost of estimating them becomes prohibitive in high-dimensional settings.
As an example, consider modelling a multivariate Gaussian distribution with `D=100` target variables, where the covariance matrix is approximated using the Cholesky-Decomposition. Modelling all conditional moments (i.e., means, standard-deviations and all pairwise correlations) requires estimation of `D(D + 3)/2 = 5,150` parameters. Because most GBM implementations are based on a *one vs. all estimation strategy*, where a separate tree is grown for each parameter, estimating this many parameters for a large dataset can become computationally extremely expensive.
The recently introduced [Py-Boost](https://github.com/sb-ai-lab/Py-Boost) approach provides a more runtime efficient GBM implementation, making it a good candidate for estimating high-dimensional target variables in a probabilistic setting. Borrowing from the original paper [SketchBoost: Fast Gradient Boosted Decision Tree for Multioutput Problems](https://openreview.net/forum?id=WSxarC8t-T), the following figure illustrates the runtime-efficiency of the Py-Boost model.
Even though the original implementation of Py-Boost also supports estimation of univariate responses, Py-BoostLSS focuses on multi-target probabilistic regression settings. For univariate probabilistic GBMs, we refer to our implementations of [XGBoostLSS](https://github.com/StatMixedML/XGBoostLSS) and [LightGBMLSS](https://github.com/StatMixedML/LightGBMLSS).
## Installation
Since Py-BoostLSS is entirely GPU-based, we first need to install the corresponding PyTorch and CuPy packages. If you are on Windows, it is preferable to install CuPy via conda. All other OS can use pip. You can check your cuda version with `nvcc --version`.
```python
# CuPy (replace with your cuda version)
# Windows only
conda install -c conda-forge cupy cudatoolkit=11.x
# Others
pip install cupy-cuda11x
# PyTorch (replace with your cuda version)
pip3 install torch --extra-index-url https://download.pytorch.org/whl/cu11x
```
Next, you can install Py-BoostLSS.
```python
pip install git+https://github.com/StatMixedML/Py-BoostLSS.git
```
## How to use
We refer to the [examples section](https://github.com/StatMixedML/Py-BoostLSS/tree/main/examples) for example notebooks.
## Available Distributions
Py-BoostLSS currently supports the following distributions. More distribution follow soon.
| Distribution | Usage |Type | Support
| :----------------------------------------------------------: |:--------------: |:--------------------------------: | :-----------------------: |
| Multivariate Normal
(Cholesky) | `MVN()` | Continous
(Multivariate) | $y \in (-\infty,\infty)$ |
| Multivariate Normal
(Low-Rank Approximation) | `MVN_LRA()` | Continous
(Multivariate) | $y \in (-\infty,\infty)$ |
| Multivariate Student-T
(Cholesky) | `MVT()` | Continous
(Multivariate) | $y \in (-\infty,\infty)$ |
| Dirichlet | `DIRICHLET()` | Continous
(Multivariate) | $y \in [0,1]$ |
## Feedback
Please provide feedback on how to improve Py-BoostLSS, or if you request additional distributions to be implemented, by opening a new issue or via the discussion section.
## Acknowledgements
The implementation of Py-BoostLSS relies on the following resources:
- [Py-boost: a research tool for exploring GBDTs](https://github.com/sb-ai-lab/Py-Boost)
- [SketchBoost: Fast Gradient Boosted Decision Tree for Multioutput Problems](https://openreview.net/forum?id=WSxarC8t-T)
We genuinely thank the original authors [Anton Vakhrushev](https://www.kaggle.com/btbpanda) and [Leonid Iosipoi](http://iosipoi.com/) for making their work publicly available.
## Reference Paper
[](https://arxiv.org/abs/2210.06831)
[](https://arxiv.org/abs/2204.00778)
[](https://arxiv.org/abs/1907.03178)