{"id":32175000,"url":"https://github.com/erichson/spca","last_synced_at":"2026-02-20T16:01:37.279Z","repository":{"id":56937298,"uuid":"126636161","full_name":"erichson/spca","owner":"erichson","description":"Sparse Principal Component Analysis (SPCA) using Variable Projection","archived":false,"fork":false,"pushed_at":"2018-04-07T20:15:09.000Z","size":17741,"stargazers_count":71,"open_issues_count":2,"forks_count":10,"subscribers_count":6,"default_branch":"master","last_synced_at":"2025-12-09T21:57:22.363Z","etag":null,"topics":["dimension-reduction","dimensionality-reduction","pca","spca"],"latest_commit_sha":null,"homepage":"https://arxiv.org/abs/1804.00341","language":"R","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/erichson.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2018-03-24T20:07:37.000Z","updated_at":"2025-10-24T16:43:19.000Z","dependencies_parsed_at":"2022-08-21T06:50:10.346Z","dependency_job_id":null,"html_url":"https://github.com/erichson/spca","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/erichson/spca","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/erichson%2Fspca","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/erichson%2Fspca/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/erichson%2Fspca/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/erichson%2Fspca/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/erichson","download_url":"https://codeload.github.com/erichson/spca/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/erichson%2Fspca/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":29491997,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-02-15T19:29:10.908Z","status":"ssl_error","status_checked_at":"2026-02-15T19:29:10.419Z","response_time":118,"last_error":"SSL_connect returned=1 errno=0 peeraddr=140.82.121.5:443 state=error: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["dimension-reduction","dimensionality-reduction","pca","spca"],"created_at":"2025-10-21T19:20:49.738Z","updated_at":"2026-02-20T16:01:37.273Z","avatar_url":"https://github.com/erichson.png","language":"R","funding_links":[],"categories":[],"sub_categories":[],"readme":"\u003cimg src=\"https://raw.githubusercontent.com/erichson/spca/master/spca.png\" width=\"550\"\u003e\n\n[![Build Status](https://travis-ci.org/erichson/spca.svg?branch=master)](https://travis-ci.org/erichson/spca)\n\n\n\n# SPCA via Variable Projection\n\nSparse principal component analysis (SPCA) is a modern variant of PCA. Specifically, SPCA attempts to find sparse \nweight vectors (loadings), i.e., a weight vector with only a few `active' (nonzero) values. This approach \nleads to an improved interpretability of the model, because the principal components are formed as a \nlinear combination of only a few of the original variables. Further, SPCA avoids overfitting in a \nhigh-dimensional data setting where the number of variables is greater than the number of observations.\n\nThis package provides robust and randomized accelerated SPCA routines in R:\n \n* Sparse PCA: ``spca()``.\n* Randomized SPCA: ``rspca()``.\n* Robust SPCA: ``robspca()``.\n\n## Problem Formulation\n\nSuch a parsimonious model is obtained by introducing prior information like sparsity promoting regularizers.\nMore concreatly, given a data matrix ``X`` with shape ``(m,p)``, SPCA attemps to minimize the following\noptimization problem:\n\n```\nminimize f(A,B) = 1/2⋅‖X - X⋅B⋅Aᵀ‖² + α⋅‖B‖₁ + 1/2⋅β‖B‖², subject to AᵀA = I.\n```\n\nThe matrix ``B`` is the sparse weight (loadings) matrix and ``A`` is an orthonormal matrix.\nHere we use a combination of the l1 and l2 norm as a sparsity-promoting regularizer, also known as the elastic net. \nThen, the principal components ``Z`` are then formed as\n\n```\nZ = X %*% B.\n```\n\nSpecifically, the interface of the SPCA function is:\n\n```R\nspca(X, k, alpha=1e-4, beta=1e-4, center=TRUE, scale=TRUE,  max_iter=1000, tol=1e-4, verbose=TRUE)\n```\nThe description of the arguments is listed in the following:\n\n* ``X`` is a real ``n`` by ``p`` data matrix (or data frame), where ``n`` denotes the number of observations and ``p`` the number of variables.\n\n* ``k`` specifies the target rank, i.e., number of components to be computed.\n\n* ``alpha`` is a sparsity controlling parameter. Higher values lead to sparser components.\n\n* `` beta`` controls the amount of ridge shrinkage to apply in order to improve conditioning.\n\n* ``center`` logical value which indicates whether the variables should be shifted to be zero centered (TRUE by default).\n\n* ``scale``logical value which indicates whether the variables should be scaled to have unit variance (FALSE by default).\n\n* ``max_iter`` maximum number of iterations to perform before exiting.\n\n* ``tol`` stopping tolerance for reconstruction error.\n\n* ``verbose`` logical value which indicates whether progress is printed.\n\nA list with the following components is returned:\n\n* ``loadings`` sparse loadings (weight) vector.\n* ``transform`` the approximated inverse transform to rotate the scores back to high-dimensional space.\n* ``scores`` the principal component scores.\n* ``eigenvalues`` the approximated eigenvalues; \n* ``center, scale`` the centering and scaling used.\n\n\n## Installation\n\nInstall the sparsepca package via CRAN\n```R\ninstall.packages(\"sparsepca\")\n```\n\nYou can also install the development version from GitHub using [devtools](https://cran.r-project.org/package=devtools):\n\n```r\ndevtools::install_github(\"erichson/spca\")\n```\n\n\n\n## Example: Sparse PCA\n\nOne of the most striking demonstrations of PCA are eigenfaces. The aim is to extract the most dominant correlations between different faces from a large set of facial images. Specifically, the resulting columns of the rotation matrix (i.e., the eigenvectors) represent `shadows' of the faces, the so-called eigenfaces. Specifically, the eigenfaces reveal both inner face features (e.g., eyes, nose, mouth) and outer features (e.g., head shape, hairline, eyebrows). These features can then be used for facial recognition and classification.\n\nIn the following we use the down-sampled cropped Yale face database B. The dataset comprises 2410 grayscale images of 38 different people, cropped and aligned, and can be loaded as \n\n```r\nutils::download.file('https://github.com/erichson/data/blob/master/R/faces.RData?raw=true', 'faces.RData', \"wget\")\nload(\"faces.RData\")\n```\nFor computational convenience the 96x84 faces images are stored as column vectors of the data matrix. For instance, the first face can be displayed as\n\n```r\nface \u003c- matrix(rev(faces[ , 1]), nrow = 84, ncol = 96)\nimage(face, col = gray(0:255 / 255))\n```\n\nIn order to approximate the ``k=25`` dominant eigenfaces you can use the standard PCA function in R:\n\n```r\nfaces.pca \u003c- prcomp(t(faces), k = 25, center = TRUE, scale. = TRUE)\n```\n\nNote, that the data matrix needs to be transposed, so that each column corresponds to a pixel location rather than to a person. Here, the analysis is performed on the correlation matrix by setting the argument \\code{scale = TRUE}. The mean face is provided as the attribute ``center``.\n\nIn the following, we use the SPCA package:\n\n```r\nlibrary(\"spca\")\n```\n\nFirst, we set the tuning parameters ``alpha=0`` and ``beta=0``, which reduces to PCA:\n\n```r\nspca.results \u003c- spca(t(faces), k=25, alpha=0, beta=0, center=TRUE, scale=TRUE)\n```\n\nThe summary of the analysis is as follows:\n\n```r\nsummary(spca.results) \n\n                            PC1       PC2      PC3      PC4      PC5    ...    \nExplained variance       2901.541  2706.701  388.080  227.062  118.407  ...\nStandard deviations        53.866    52.026   19.700   15.069   10.882  ... \nProportion of variance      0.360     0.336    0.048    0.028    0.015  ...\nCumulative proportion       0.360     0.695    0.744    0.772    0.786  ... \n```\n\nJust the first 5 PCs explain about 79% of the total variation in the data, while the first 25 PCs explain more then 90%. Finally, the eigenvectors can be visualized as eigenfaces, e.g., the first eigenvector (eigenface) is displayed as follows\n\n```r\nlayout(matrix(1:25, 5, 5, byrow = TRUE))\nfor(i in 1:25) {\n  par(mar = c(0.5,0.5,0.5,0.5))\n  img \u003c- matrix(spca.results$loadings[,i], nrow=84, ncol=96)\n  image(img[,96:1], col = gray((255:0)/255), axes=FALSE, xaxs=\"i\", yaxs=\"i\", xaxt='n', yaxt='n',ann=FALSE )\n}\n```\n\n\u003cimg src=\"https://raw.githubusercontent.com/erichson/spca/master/plots/eigenfaces.png\" width=\"500\"\u003e\n\nThe eigenfaces encode the holistic facial features as well as the illumination. \n\nIn  many application, however, it is favorable to obtain a sparse representation. This is because a sparse representation is easier to interpret.  Further, this avoids overfitting in a high-dimensional data setting where the number of variables is greater than the number of observations. SPCA attempts to find sparse weight vectors (loadings), i.e., the approximate eigenvecotrs with only a few `active' (nonzero) values. We can compute the sparse eigenvectors as follows:\n\n```r\nrspca.results \u003c- rspca(t(faces), k=25, alpha=1e-4, beta=1e-1, verbose=1, max_iter=1000, tol=1e-4, center=TRUE, scale=TRUE)\n```\n\nthe objective values for each iteration can be plotted as:\n\n```r\nplot(log(rspca.results$objective), col='red', xlab='Number of iterations', ylab='Objective value')\n```\n\nNote, that we have use here the randomized accelerated SPCA algorithm! The randomized algorithm eases the computational demands and is suitable if the input data feature some low-rank structure. For more details about randomized methods see, for instance, [Randomized Matrix Decompositions using R](http://arxiv.org/abs/1608.02148).\nNow, ``summary(rspca.results)`` reveals that the first 5 PCs only explain about 67% of the total variation. However, we yield a parsimonious representation of the data:\n\n```r\nlayout(matrix(1:25, 5, 5, byrow = TRUE))\nfor(i in 1:25) {\n    par(mar = c(0.5,0.5,0.5,0.5))\n    img \u003c- matrix(rspca.results$loadings[,i], nrow=84, ncol=96)\n    image(img[,96:1], col = gray((255:0)/255), axes=FALSE, xaxs=\"i\", yaxs=\"i\", xaxt='n', yaxt='n',ann=FALSE )\n}\n```\n\n\u003cimg src=\"https://raw.githubusercontent.com/erichson/spca/master/plots/sparseeigenfaces.png\" width=\"500\"\u003e\n\nUnlike PCA, the sparse loadings contextualize localized features. If desired the solution can be made even sparser by increasing the tuning parameter ``alpha``:\n\n```r\nrspca.results \u003c- rspca(t(faces), k=25, alpha=2e-4, beta=2e-1, verbose=1, max_iter=1000, tol=1e-4, center=TRUE, scale=TRUE)\n```\nWe yield the following sparse loadings:\n\n```r\nlayout(matrix(1:25, 5, 5, byrow = TRUE))\nfor(i in 1:25) {\n    par(mar = c(0.5,0.5,0.5,0.5))\n    img \u003c- matrix(rspca.results$loadings[,i], nrow=84, ncol=96)\n    image(img[,96:1], col = gray((255:0)/255), axes=FALSE, xaxs=\"i\", yaxs=\"i\", xaxt='n', yaxt='n',ann=FALSE )\n}\n```\n\n\u003cimg src=\"https://raw.githubusercontent.com/erichson/spca/master/plots/sparsereigenfaces.png\" width=\"500\"\u003e\n\n\n\n## Computational Comparison\n\nNow, let us evaluate the computatinal performace. Here we consider for comparision also the SPCA function from the ``elasticnet`` package. First, we create some artificial data where we know the ground truth: \n\n```r\nm \u003c- 100000\nV1 \u003c- rnorm(m, 0, 290)\nV2 \u003c- rnorm(m, 0, 300)\nV3 \u003c- -0.1*V1 + 0.1*V2 + rnorm(m,0,100)\nX \u003c- cbind(V1,V1,V1,V1, V2,V2,V2,V2, V3,V3)\nX \u003c- X + matrix(rnorm(length(X),0,1), ncol = ncol(X), nrow = nrow(X))\n```\nHence, we expect to find 3 components. The first and second should have 4 non-zero loadings and the third 2 non-zero loadings. As a sanity check you can plot the loadings:\n\n```r\nout \u003c- rspca(X, k=3, alpha=1e-4, beta=1e-4, center = TRUE, scale = FALSE, verbose=0)\nprint(out)\n\nStandard deviations:\n[1] 599.686 578.621 137.740\n\nEigenvalues:\n[1] 359623.56 334802.29  18972.17\n\nSparse loadings:\n        [,1]   [,2]  [,3]\n [1,]  0.000 -0.498 0.000\n [2,]  0.000 -0.498 0.000\n [3,]  0.000 -0.498 0.000\n [4,]  0.000 -0.498 0.000\n [5,] -0.499  0.000 0.000\n [6,] -0.499  0.000 0.000\n [7,] -0.499  0.000 0.000\n [8,] -0.499  0.000 0.000\n [9,]  0.000  0.000 0.678\n[10,]  0.000  0.000 0.678\n```\n\nNext, we use the ``microbenchmark`` package for timing:\n\n```r\nlibrary(\"microbenchmark\")\n\ntiming_spca = microbenchmark(\n  'Elastinet SPCA' = elasticnet::spca(X, K = 3, type = \"predictor\", sparse = \"varnum\", para = c(4, 4, 2)),\n  'Deterministic SPCA' = spca(X, k=3, alpha=1e-4, beta=1e-4, verbose=0),\n  'Randomized SPCA'  = rspca(X, k=3, alpha=1e-4, beta=1e-4, verbose=0, o=0, q=0),\n  times = 20\n)\n\nautoplot(timing_spca, log = TRUE, y_max = 1.05 * max(timing_spca$time))\n```\nThe performance is summarized in the following plot.\n\n\u003cimg src=\"https://raw.githubusercontent.com/erichson/spca/master/plots/timeing_art.png\" width=\"800\"\u003e\n\nWe clearly see the computational advantage of the SPCA algorithm using variable projection compare to the implementation provided by the elastinet package. Further, we see that there is little difference between the randomized and the deterministic algorithms here. However, the performance is pronaunced for bigger datasets such as the face data. \n\n```r\ntiming_spca = microbenchmark(\n  'Deterministic SPCA' = spca(t(faces), k=25, alpha=1e-4, beta=1e-1, verbose=0, max_iter=1000, tol=1e-4),\n  'Randomized SPCA'  = rspca(t(faces), k=25, alpha=1e-4, beta=1e-1, verbose=0, max_iter=1000, tol=1e-4),\n  times = 15\n)\n\nautoplot(timing_spca, log = FALSE, y_max = 1.2 * max(timing_spca$time))\n```\nClearly, the randomized algorithm shows some substantial speedups over the deterministic algorithm.\n\n\u003cimg src=\"https://raw.githubusercontent.com/erichson/spca/master/plots/timeing.png\" width=\"800\"\u003e\n\n\n\n\n\n\n## Example: Robust SPCA\n\nIn the following we demonstrate the robust SCPA which allows to capture some grossly corrupted entries in the data. The idea is to separate the input data into a low-rank component and a sparse component. The latter aims to capture potential outliers in the data. For the face data, we proceed as follows:\n\n```r\nrobspca.results \u003c- robspca(t(faces), k=25, alpha=1e-4, beta=1e-2, gamma=0.9, verbose=1, max_iter=1000, tol=1e-4, center=TRUE, scale=TRUE)\n```\n\nThe tuning parameter ``gamma`` is the sparsity controlling parameter for the sparse error matrix. Smaller values lead to a larger amount of noise removeal. We yield the following sparse loadings:\n\n```r\nlayout(matrix(1:25, 5, 5, byrow = TRUE))\nfor(i in 1:25) {\n    par(mar = c(0.5,0.5,0.5,0.5))\n    img \u003c- matrix(robspca.results$loadings[,i], nrow=84, ncol=96)\n    image(img[,96:1], col = gray((255:0)/255), axes=FALSE, xaxs=\"i\", yaxs=\"i\", xaxt='n', yaxt='n',ann=FALSE )\n}\n```\n\n\u003cimg src=\"https://raw.githubusercontent.com/erichson/spca/master/plots/robustsparseeigenfaces.png\" width=\"500\"\u003e\n\n\nFurther, we can visualize the captured outliers for each face. Here we show the outliers for the first 25 faces:\n\n```r\nlayout(matrix(1:25, 5, 5, byrow = TRUE))\nfor(i in 1:25) {\n  par(mar = c(0.5,0.5,0.5,0.5))\n  img \u003c- matrix(robspca.results$sparse[i,], nrow=84, ncol=96)\n  image(img[,96:1], col = gray((255:0)/255), axes=FALSE, xaxs=\"i\", yaxs=\"i\", xaxt='n', yaxt='n',ann=FALSE )\n}\n```\n\n\u003cimg src=\"https://raw.githubusercontent.com/erichson/spca/master/plots/sparsecomp.png\" width=\"500\"\u003e\n\nIt can be seen that the robust SPCA algorithms captures some of the specularities and other errors in the data. \n\n\n\n\n## References\n\n* [N. Benjamin Erichson, et al. \"Sparse Principal Component Analysis via Variable Projection.\" (2018)](https://arxiv.org/abs/1804.00341)\n\n* [N. Benjamin Erichson, et al. \"Randomized Matrix Decompositions using R.\" (2016)](http://arxiv.org/abs/1608.02148)\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ferichson%2Fspca","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Ferichson%2Fspca","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ferichson%2Fspca/lists"}