https://github.com/mobeets/group-ard

code for performing Bayesian ARD regression, where covariates have groups
https://github.com/mobeets/group-ard

automatic-relevance-determination bayesian-linear-regression bayesian-regression sparse-regression

Last synced: about 2 months ago
JSON representation

code for performing Bayesian ARD regression, where covariates have groups

• Host: GitHub
• URL: https://github.com/mobeets/group-ard
• Owner: mobeets
• Created: 2023-11-22T18:48:58.000Z (about 1 year ago)
• Default Branch: main
• Last Pushed: 2024-03-25T19:52:40.000Z (9 months ago)
• Last Synced: 2024-10-12T15:46:35.616Z (3 months ago)
• Topics: automatic-relevance-determination, bayesian-linear-regression, bayesian-regression, sparse-regression
• Language: Python
• Homepage:
• Size: 307 KB
• Stars: 0
• Watchers: 2
• Forks: 1
• Open Issues: 0
• Metadata Files:
  • Readme: README.md

Awesome Lists containing this project

README

        
## Summary

Suppose we have covariates $X \in \mathbb{R}^{N \times K}$ and observations $y \in \mathbb{R}^N$, where our observation model is $y_i \sim \mathcal{N}(x_i^\top w, \sigma^2)$. Here, $w \in \mathbb{R}^K$ are unknown weights.

In linear regression we want to find the best estimate of the weights given $X$ and $y$. For example, standard linear regression finds the weights $\widehat{w}$ minimizing the sum of the squared residuals:

$$ \|| y - X \widehat{w} \||_2^2 $$

When $N$ is small or $K$ is large, it's often useful to do Bayesian linear regression. This involves choosing a prior on our weights (see [1] for more details). Some common choices or prior are:

1. __Ridge__: $w_i \sim \mathcal{N}(0, \alpha^{-1})$, where $\alpha \in \mathbb{R}$ is called our "inverse prior variance".

2. __Automatic Relevance Determination (ARD)__: $w_i \sim \mathcal{N}(0, \alpha_i^{-1})$. Note that now each covariate has its own inverse prior variance.

Here we consider a third option in between these two, which I will call "Group ARD" (in analogy to Group Lasso [2]). This prior is relevant when our covariates can be grouped. Specifically, we assume the $i^{th}$ covariate has a known group label $c_i \in \\{ 1, 2, \ldots, G\\}$, where $G$ is the total number of groups. The idea is that every covariate in the same group has the same inverse prior variance. In other words:

3. __Group ARD__: $w \sim \mathcal{N}(0, \alpha_{c_i}^{-1})$

We can estimate the inverse prior variances using similar methods proposed in Appendix 1 of [1].

For what it's worth, this model was implemented in [3] but called "Group-sparse Bayesian linear discriminant analysis" (?).

## References

[1] Tipping, Michael E. "Sparse Bayesian learning and the relevance vector machine." Journal of machine learning research 1.Jun (2001): 211-244.

[2] Yuan, Ming, and Yi Lin. "Model selection and estimation in regression with grouped variables." Journal of the Royal Statistical Society Series B: Statistical Methodology 68.1 (2006): 49-67.

[3] Yu, Tianyou, et al. "Grouped automatic relevance determination and its application in channel selection for P300 BCIs." IEEE Transactions on Neural Systems and Rehabilitation Engineering 23.6 (2015): 1068-1077.