Ecosyste.ms: Awesome

An open API service indexing awesome lists of open source software.

Awesome Lists | Featured Topics | Projects

https://github.com/fabricioarendtorres/blasso_sa

Implementation of Simulated Annealing for the Bayesian LASSO.
https://github.com/fabricioarendtorres/blasso_sa

Last synced: about 1 month ago
JSON representation

Implementation of Simulated Annealing for the Bayesian LASSO.

Host: GitHub
URL: https://github.com/fabricioarendtorres/blasso_sa
Owner: FabricioArendTorres
License: mit
Created: 2018-08-15T09:02:33.000Z (over 6 years ago)
Default Branch: master
Last Pushed: 2018-09-06T08:44:52.000Z (about 6 years ago)
Last Synced: 2023-08-29T18:56:04.917Z (about 1 year ago)
Language: Jupyter Notebook
Size: 342 KB
Stars: 0
Watchers: 0
Forks: 0
Open Issues: 0
Metadata Files:
- Readme: README.md
- License: LICENSE

Awesome Lists containing this project

README

# BLASSO_SA
MAP estimation of the Bayesian LASSO via Simulated Annealing.
# NOTE
Depends on a submodule (for sampling from the Generalized Inverse Gaussian), so don't forget to initialize and update the submodules when cloning the repo.
# Sampler
The Simulated Annealing is based on the Gibbs sampler presented in [1] (with marginalized out μ).
$\begin{array}{llcl} &&&A = X^TX + D_\tau^{-1} \\ &&&D_\tau = diag(\tau_1^2,\dots,\tau_p^2) \\ \\ \beta &| \quad \tilde{y}, \sigma^2, \tau_1^2, \dots, \tau_p^2 &\sim& \mathcal{N}(A^{-1} X^T\tilde{y}, \sigma^2 A^{-1}) \\ \sigma^2 &|\quad\tilde{y}, \beta,\tau_1^2,\dots,\tau_p^2 &\sim& InvGamma\Big(\frac{1}{2}(n-1+p), \frac{1}{2} \big((\tilde{y}-X\beta)^T (\tilde{y}-X\beta)+ \beta^TD_\tau^{-1}\beta\big)\Big) \\ 1/\tau_j^2 &|\quad \tilde{y}, \beta, \tau_{-j}^2 &\sim& InvGauss\Big(\mu'=\sqrt{\frac{\lambda^2\sigma^2}{\beta_j^2}}, \lambda'=\lambda^2 \Big) \end{array}$

Cooling down of the posterior conditionals can be achieved by a parameter shift of the distributions. For the Inverse Gaussian distribution we make use of the fact that we can represent an IG as a Generalized Inverse Gaussian
with p=-0.5.

Let T be the current Temperature. Then we can sample according to:
$\large \begin{array}{llcl} &&&T \in \mathbb{R}_{>0} \\ \\ &&&A = X^TX + D_\tau^{-1} \\ &&&D_\tau = diag(\tau_1^2,\dots,\tau_p^2) \\ \\ \beta &| \quad \tilde{y}, \sigma^2, \tau_1^2, \dots, \tau_p^2, T &\sim& \mathcal{N}(A^{-1} X^T\tilde{y}, \sigma^2 T A^{-1}) \\ \sigma^2 &|\quad\tilde{y}, \beta,\tau_1^2,\dots,\tau_p^2, T &\sim& InvGamma\Big(\frac{\frac{1}{2}(n-1+p)+1}{T} -1, \frac{\frac{1}{2} \big((\tilde{y}-X\beta)^T (\tilde{y}-X\beta)+ \beta^TD_\tau^{-1}\beta\big)}{T}\Big) \\ 1/\tau_j^2 &|\quad \tilde{y}, \beta, \tau_{-j}^2, T &\sim& GIG\Big(a'=\frac{\lambda'/\mu'^2}{T}, b'=\frac{\lambda'}{T}, p'=(-\frac{1.5}{T}+1)\Big) \\ &&&\mu'=\sqrt{\frac{\lambda^2\sigma^2}{\beta_j^2}} \\ &&&\lambda'=\lambda^2 \end{array}$

# Sampling from the Normal
We want to avoid having to directly invert A:

$\large \begin{array}{lcl} T\sigma^2 A^{-1} &=& T\sigma^2 (L^TL)^{-1} \\ &=& T\sigma^2 (L^{-T} L^{-1}) \\ &=& (\sqrt{T}\sigma L^{-T}) (L^{-1}\sqrt{T}\sigma) \\ \\ r = \begin{bmatrix} r_{1} \\ \vdots \\ r_{p} \end{bmatrix} &\sim& \mathcal{N}(0, I_p) \\ \\ b = (\sqrt{T}\sigma L^{-T})r &\sim& \mathcal{N}(0, T\sigma^2A^{-1}) \\ \Leftrightarrow L^T b = \sqrt{T}\sigma r \\ \\ \mu &=& A^{-1} X^T \tilde{y} \\ \Leftrightarrow A\mu&=&X^T\tilde{y} \\ \Leftrightarrow LL^T\mu &=& X^T \tilde{y} \\ b + \mu &\sim& \mathcal{N}(A^{-1}X^T\tilde{y}, T\sigma^2A^{-1}) \end{array}$

solve for b by backward subtitution and for μ by forward and backward substitution.

# Example
Example Jupyter Notebook

# References

[1] Park, Trevor, and George Casella. "The bayesian lasso." Journal of the American Statistical Association 103.482 (2008): 681-686.