https://github.com/mobeets/group-ard
code for performing Bayesian ARD regression, where covariates have groups
https://github.com/mobeets/group-ard
automatic-relevance-determination bayesian-linear-regression bayesian-regression sparse-regression
Last synced: 2 months ago
JSON representation
code for performing Bayesian ARD regression, where covariates have groups
- Host: GitHub
- URL: https://github.com/mobeets/group-ard
- Owner: mobeets
- Created: 2023-11-22T18:48:58.000Z (over 1 year ago)
- Default Branch: main
- Last Pushed: 2024-03-25T19:52:40.000Z (about 1 year ago)
- Last Synced: 2025-01-12T17:11:38.500Z (4 months ago)
- Topics: automatic-relevance-determination, bayesian-linear-regression, bayesian-regression, sparse-regression
- Language: Python
- Homepage:
- Size: 307 KB
- Stars: 0
- Watchers: 2
- Forks: 1
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
Awesome Lists containing this project
README
## Summary
Suppose we have covariates $X \in \mathbb{R}^{N \times K}$ and observations $y \in \mathbb{R}^N$, where our observation model is $y_i \sim \mathcal{N}(x_i^\top w, \sigma^2)$. Here, $w \in \mathbb{R}^K$ are unknown weights.
In linear regression we want to find the best estimate of the weights given $X$ and $y$. For example, standard linear regression finds the weights $\widehat{w}$ minimizing the sum of the squared residuals:
$$ \|| y - X \widehat{w} \||_2^2 $$
When $N$ is small or $K$ is large, it's often useful to do Bayesian linear regression. This involves choosing a prior on our weights (see [1] for more details). Some common choices or prior are:
1. __Ridge__: $w_i \sim \mathcal{N}(0, \alpha^{-1})$, where $\alpha \in \mathbb{R}$ is called our "inverse prior variance".
2. __Automatic Relevance Determination (ARD)__: $w_i \sim \mathcal{N}(0, \alpha_i^{-1})$. Note that now each covariate has its own inverse prior variance.
Here we consider a third option in between these two, which I will call "Group ARD" (in analogy to Group Lasso [2]). This prior is relevant when our covariates can be grouped. Specifically, we assume the $i^{th}$ covariate has a known group label $c_i \in \\{ 1, 2, \ldots, G\\}$, where $G$ is the total number of groups. The idea is that every covariate in the same group has the same inverse prior variance. In other words:
3. __Group ARD__: $w \sim \mathcal{N}(0, \alpha_{c_i}^{-1})$
We can estimate the inverse prior variances using similar methods proposed in Appendix 1 of [1].
For what it's worth, this model was implemented in [3] but called "Group-sparse Bayesian linear discriminant analysis" (?).
## References
[1] Tipping, Michael E. "Sparse Bayesian learning and the relevance vector machine." Journal of machine learning research 1.Jun (2001): 211-244.
[2] Yuan, Ming, and Yi Lin. "Model selection and estimation in regression with grouped variables." Journal of the Royal Statistical Society Series B: Statistical Methodology 68.1 (2006): 49-67.
[3] Yu, Tianyou, et al. "Grouped automatic relevance determination and its application in channel selection for P300 BCIs." IEEE Transactions on Neural Systems and Rehabilitation Engineering 23.6 (2015): 1068-1077.