{"id":18501598,"url":"https://github.com/const-ae/sparsematrixstats","last_synced_at":"2025-04-09T14:14:49.124Z","repository":{"id":38999880,"uuid":"211092842","full_name":"const-ae/sparseMatrixStats","owner":"const-ae","description":"Implementation of the matrixStats API for sparse matrices","archived":false,"fork":false,"pushed_at":"2024-06-12T11:08:33.000Z","size":842,"stargazers_count":54,"open_issues_count":6,"forks_count":3,"subscribers_count":5,"default_branch":"devel","last_synced_at":"2025-04-02T12:12:26.657Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":"R","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"other","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/const-ae.png","metadata":{"files":{"readme":"README.Rmd","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2019-09-26T13:13:48.000Z","updated_at":"2024-06-29T03:15:41.000Z","dependencies_parsed_at":"2024-06-18T22:40:19.417Z","dependency_job_id":"f6622f26-003b-4d7a-9273-8e023926182a","html_url":"https://github.com/const-ae/sparseMatrixStats","commit_stats":{"total_commits":247,"total_committers":7,"mean_commits":"35.285714285714285","dds":0.07287449392712553,"last_synced_commit":"4c6e8142914b0c73de29830fa08ba2a52b202a08"},"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/const-ae%2FsparseMatrixStats","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/const-ae%2FsparseMatrixStats/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/const-ae%2FsparseMatrixStats/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/const-ae%2FsparseMatrixStats/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/const-ae","download_url":"https://codeload.github.com/const-ae/sparseMatrixStats/tar.gz/refs/heads/devel","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248054193,"owners_count":21039952,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-06T13:54:22.858Z","updated_at":"2025-04-09T14:14:49.091Z","avatar_url":"https://github.com/const-ae.png","language":"R","funding_links":[],"categories":[],"sub_categories":[],"readme":"---\noutput: github_document\n---\n\n\u003c!-- README.md is generated from README.Rmd. Please edit that file --\u003e\n\n```{r, include = FALSE}\nknitr::opts_chunk$set(\n  collapse = TRUE,\n  comment = \"#\u003e\",\n  fig.path = \"man/figures/README-\",\n  out.width = \"100%\"\n)\n```\n\n# sparseMatrixStats \u003ca href='https://github.com/const-ae/sparseMatrixStats'\u003e\u003cimg src='man/figures/logo.png' align=\"right\" height=\"209\" /\u003e\u003c/a\u003e\n\n\u003c!-- badges: start --\u003e\n[![codecov](https://codecov.io/gh/const-ae/sparseMatrixStats/branch/master/graph/badge.svg)](https://codecov.io/gh/const-ae/sparseMatrixStats)\n\n\u003c!-- badges: end --\u003e\n\nThe goal of `sparseMatrixStats` is to make the API of [matrixStats](https://github.com/HenrikBengtsson/matrixStats) available\nfor sparse matrices.\n\n## Installation\n\nYou can install the release version of *[sparseMatrixStats](https://bioconductor.org/packages/sparseMatrixStats)* from BioConductor:\n\n``` r\nif (!requireNamespace(\"BiocManager\", quietly = TRUE))\n    install.packages(\"BiocManager\")\n\nBiocManager::install(\"sparseMatrixStats\")\n```\n\nAlternatively, you can get the development version of the package from [GitHub](https://github.com/const-ae/sparseMatrixStats) with:\n\n``` r\n# install.packages(\"devtools\")\ndevtools::install_github(\"const-ae/sparseMatrixStats\")\n```\n\nIf you have trouble with the installation, see the end of the README.\n\n## Example\n\n```{r include=FALSE}\nset.seed(1)\n```\n\n\n```{r}\nlibrary(sparseMatrixStats)\n```\n\n\n```{r}\nmat \u003c- matrix(0, nrow=10, ncol=6)\nmat[sample(seq_len(60), 4)] \u003c- 1:4\n# Convert dense matrix to sparse matrix\nsparse_mat \u003c- as(mat, \"dgCMatrix\")\nsparse_mat\n```\n\nThe package provides an interface to quickly do common operations on the rows or columns. For example calculate\nthe variance:\n\n```{r}\napply(mat, 2, var)\nmatrixStats::colVars(mat)\nsparseMatrixStats::colVars(sparse_mat)\n```\n\nOn this small example data, all methods are basically equally fast, but if we have a much larger dataset, the \noptimizations for the sparse data start to show.\n\nI generate a dataset with 10,000 rows and 50 columns that is 99% empty\n\n```{r}\nbig_mat \u003c- matrix(0, nrow=1e4, ncol=50)\nbig_mat[sample(seq_len(1e4 * 50), 5000)] \u003c- rnorm(5000)\n# Convert dense matrix to sparse matrix\nbig_sparse_mat \u003c- as(big_mat, \"dgCMatrix\")\n```\n\nI use the `bench` package to benchmark the performance difference:\n\n```{r}\nbench::mark(\n  sparseMatrixStats=sparseMatrixStats::colVars(big_sparse_mat),\n  matrixStats=matrixStats::colVars(big_mat),\n  apply=apply(big_mat, 2, var)\n)\n```\n\n\nAs you can see `sparseMatrixStats` is ca. 35 times fast than `matrixStats`, which in turn is 7 times faster than the `apply()` version.\n\n\n\n\n\n# API\n\nThe package now supports all functions from the `matrixStats` API for column sparse matrices (`dgCMatrix`). And thanks to the [`MatrixGenerics`](https://bioconductor.org/packages/MatrixGenerics/) it can be easily integrated along-side [`matrixStats`](https://cran.r-project.org/package=matrixStats) and [`DelayedMatrixStats`](https://bioconductor.org/packages/DelayedMatrixStats/).\nNote that the `rowXXX()` functions are called by transposing the input and calling the corresponding `colXXX()` function. Special optimized implementations are available for `rowSums2()`, `rowMeans2()`, and `rowVars()`.\n\n```{r, echo=FALSE, results=\"asis\"}\nmatrixStats_functions \u003c- sort(\n  c(\"colsum\", \"rowsum\", grep(\"^(col|row)\", \n                             getNamespaceExports(\"matrixStats\"), \n                             value = TRUE)))\nDelayedMatrixStats_functions \u003c- grep(\"^(col|row)\", getNamespaceExports(\"DelayedMatrixStats\"), value=TRUE)\nDelayedArray_functions \u003c- grep(\"^(col|row)\", getNamespaceExports(\"DelayedArray\"), value=TRUE)\nsparseMatrixStats_functions \u003c- grep(\"^(col|row)\", getNamespaceExports(\"sparseMatrixStats\"), value=TRUE)\n\nnotes \u003c- c(\"colAnyMissings\"=\"Not implemented because it is deprecated in favor of `colAnyNAs()`\",\n           \"rowAnyMissings\"=\"Not implemented because it is deprecated in favor of `rowAnyNAs()`\",\n           \"colsum\"=\"Base R function\",\n           \"rowsum\"=\"Base R function\",\n           \"colWeightedMedians\"=\"Only equivalent if `interpolate=FALSE`\",\n           \"rowWeightedMedians\"=\"Only equivalent if `interpolate=FALSE`\",\n           \"colWeightedMads\"=\"Sparse version behaves slightly differently, because it always uses `interpolate=FALSE`.\",\n           \"rowWeightedMads\"=\"Sparse version behaves slightly differently, because it always uses `interpolate=FALSE`.\")\n\napi_df \u003c- data.frame(\n  Method = paste0(matrixStats_functions, \"()\"),\n  matrixStats = ifelse(matrixStats_functions %in% matrixStats_functions, \"✔\", \"❌\"),\n  sparseMatrixStats = ifelse(matrixStats_functions %in%sparseMatrixStats_functions, \"✔\", \"❌\"),\n  Notes = ifelse(matrixStats_functions %in% names(notes), notes[matrixStats_functions], \"\"),\n  stringsAsFactors = FALSE\n)\nknitr::kable(api_df, row.names = FALSE)\n\n```\n\n\n\n# Installation Problems\n\n`sparseMatrixStats` uses features from C++14 and as the standard is more than 6 years old, I thought this wouldn't cause problems. In most circumstances this is true, but there are reoccuring reports, that the installation fails for some people and that is of course annoying. The typical error message is:\n```\nError: C++14 standard requested but CXX14 is not defined\n```\n\nThe main reason that the installation fails is that the compiler is too old. Sufficient support for C++14 came in \n\n* `clang` version 3.4\n* `gcc` version 4.9\n\nAccordingly, you must have a compiler available that is at least that new. If you run on the command line\n\n```{bash eval=FALSE, include=TRUE}\n$ gcc --version\n```\n\nand it says 4.8, you will have to install a newer compiler. At the end of the section, I have collected a few tips to install an appropriate version on different distributions.\n\nIf you have recent version of `gcc` (\u003e=4.9) or `clang` (\u003e= 3.4) installed, but you still see the error message\n```\nError: C++14 standard requested but CXX14 is not defined\n```\nthe problem is that R doesn't yet know about it.\n\nThe solution is to either create a `~/.R/Makevars` file and define\n```\nCXX14 = g++\nCXX14FLAGS = -g -O2 $(LTO)\nCXX14PICFLAGS = -fpic\nCXX14STD = -std=gnu++14\n```\nor simply call \n```{r eval=FALSE, include=TRUE}\nwithr::with_makevars(\n new = c(CXX14 = \"g++\", CXX14FLAGS = \"-g -O2 $(LTO)\", \n         CXX14PICFLAGS = \"-fpic\", CXX14STD = \"-std=gnu++14\"), \n code = {\n   BiocManager::install(\"sparseMatrixStats\")\n })\n```\n\n\n\n### Update Compiler\n\n\n#### CentOS / Scientic Linux / RHEL\n\nOne of the main culprits causing trouble is CentOS 7. It is popular in scientific computing and is still supported until 2024. It does, however, by default come with a very old version of `gcc` (4.8.5).\n\nTo install a more recent compiler, we can use [devtoolset](https://www.softwarecollections.org/en/scls/rhscl/devtoolset-7/). First, we enable the Software Collection Tools and then install for example `gcc` version 7:\n\n```{bash eval=FALSE}\n$ yum install centos-release-scl\n$ yum install devtoolset-7-gcc*\n```\n\nWe can now either activate the new compiler for an R session   \n```{bash eval=FALSE}\n$ scl enable devtoolset-7 R\n```\nand then call \n```{r eval=FALSE, include=TRUE}\nwithr::with_makevars(\n new = c(CXX14 = \"g++\", CXX14FLAGS = \"-g -O2 $(LTO)\", \n         CXX14PICFLAGS = \"-fpic\", CXX14STD = \"-std=gnu++14\"), \n code = {\n   BiocManager::install(\"sparseMatrixStats\")\n })\n```\n\nor we refer to the full path of the newly installed g++ from a standard R session\n\n```{r eval=FALSE, include=TRUE}\nwithr::with_makevars(\n new = c(CXX14 = \"/opt/rh/devtoolset-7/root/usr/bin/g++\", \n         CXX14FLAGS = \"-g -O2 $(LTO)\", CXX14PICFLAGS = \"-fpic\",\n         CXX14STD = \"-std=gnu++14\"), \n code = {\n   BiocManager::install(\"sparseMatrixStats\")\n })\n```\n\nNote, that our shenanigans are only necessary once, when we install `sparseMatrixStats`. After the successful installation of the package, we can use R as usual.\n\n\n#### Debian\n\nAll Debian releases later than Jessie (i.e. Stretch, Buster, Bullseye) are recent enough and should install sparseMatrixStats without problems.\n\nI was able to install `sparseMatrixStats` on Debian Jessie (which comes with `gcc` version 4.9.2) by providing the necessary Makefile arguments\n```{r eval=FALSE, include=TRUE}\nwithr::with_makevars(\n new = c(CXX14 = \"g++\", \n         CXX14FLAGS = \"-g -O2 $(LTO)\", CXX14PICFLAGS = \"-fpic\",\n         CXX14STD = \"-std=gnu++14\"), \n code = {\n   BiocManager::install(\"sparseMatrixStats\")\n })\n```\n\nDebian Wheezy comes with `gcc` 4.7, which does not support C++14. On the other hand, the last R release that was backported to Wheezy is 3.2.5 (see information on [CRAN](https://cloud.r-project.org/bin/linux/debian/#debian-wheezy-oldoldoldstable)). Thus, if you are still on Wheezy, I would encourage you to update your OS.\n\n\n#### Ubuntu\n\nSince 16.04, Ubuntu comes with a recent enough compiler.\n\nUbuntu 14.04 comes with `gcc` 4.8.5, but updating to `gcc-5` is easy:\n\n```{bash eval=FALSE, include=TRUE}\n$ sudo add-apt-repository ppa:ubuntu-toolchain-r/test\n$ sudo apt-get update\n$ sudo apt-get install gcc-5 g++-5\n```\nAfter that, you can install `sparseMatrixStats` with a custom Makevars variables that refer to the new compiler\n```{r eval=FALSE, include=TRUE}\nwithr::with_makevars(\n new = c(CXX14 = \"g++-5\", \n         CXX14FLAGS = \"-g -O2 $(LTO)\", CXX14PICFLAGS = \"-fpic\",\n         CXX14STD = \"-std=gnu++14\"), \n code = {\n   BiocManager::install(\"sparseMatrixStats\")\n })\n```\n\n#### MacOS\n\nNo trouble reported so far. Just do:\n\n```{r eval=FALSE, include=TRUE}\nBiocManager::install(\"sparseMatrixStats\")\n```\n\n#### Windows\n\nIt is important that you have [RTools40](https://cran.r-project.org/bin/windows/Rtools/) installed. After that, you shouldn't have any troubles installing `sparseMatrixStats` directly from Bioconductor:\n\n```{r eval=FALSE, include=TRUE}\nBiocManager::install(\"sparseMatrixStats\")\n```\n\n\n### But I still have a problems\n\n1. *Please* make sure to carefully read the full problem section.\n2. Make sure that you are using at least R 4.0.0.\n3. Make sure your compiler is new enough to support C++14 (ie. `gcc` \u003e= 4.9 and `clang` \u003e= 3.4)\n\nIf your problems nonetheless persist, please file an [issue](https://github.com/const-ae/sparseMatrixStats/issues/) including the following information:\n\n* Operating system with exact version (e.g. 'Linux Ubuntu 18.04')\n* Compiler and compiler version (e.g. 'gcc version 7.2.5')\n* The output of `sessionInfo()`\n* Information if you have a `~/.R/Makevars` file and what it contains\n* The exact call that you use to install `sparseMatrixStats` including the full error message\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fconst-ae%2Fsparsematrixstats","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fconst-ae%2Fsparsematrixstats","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fconst-ae%2Fsparsematrixstats/lists"}