https://github.com/koheiw/proxyc

R package for large-scale similarity/distance computation
https://github.com/koheiw/proxyc

data-science distance-measures r similarity-measures

Last synced: 3 months ago
JSON representation

R package for large-scale similarity/distance computation

• Host: GitHub
• URL: https://github.com/koheiw/proxyc
• Owner: koheiw
• License: gpl-3.0
• Created: 2018-12-17T19:58:04.000Z (about 6 years ago)
• Default Branch: master
• Last Pushed: 2024-04-25T11:39:11.000Z (9 months ago)
• Last Synced: 2024-04-28T03:20:53.917Z (9 months ago)
• Topics: data-science, distance-measures, r, similarity-measures
• Language: R
• Size: 995 KB
• Stars: 28
• Watchers: 6
• Forks: 6
• Open Issues: 4
• Metadata Files:
  • Readme: README.Rmd
  • Changelog: NEWS.md
  • License: LICENSE

Awesome Lists containing this project

README

        
---

output: github_document

---

```{r, echo = FALSE}

knitr::opts_chunk$set(

  collapse = TRUE,

  comment = "##",

  fig.path = "man/figures/",

  fig.width = 9,

  fig.height = 3,

  warning = FALSE,

  dpi = 150

)

```

# proxyC: R package for large-scale similarity/distance computation

[![CRAN Version](https://www.r-pkg.org/badges/version/proxyC)](https://CRAN.R-project.org/package=proxyC)

[![Downloads](https://cranlogs.r-pkg.org/badges/proxyC)](https://CRAN.R-project.org/package=proxyC)

[![Total Downloads](https://cranlogs.r-pkg.org/badges/grand-total/proxyC?color=orange)](https://CRAN.R-project.org/package=proxyC)

[![R build status](https://github.com/koheiw/proxyC/workflows/R-CMD-check/badge.svg)](https://github.com/koheiw/proxyC/actions)

[![codecov](https://codecov.io/gh/koheiw/proxyC/branch/master/graph/badge.svg)](https://app.codecov.io/gh/koheiw/proxyC)

**proxyC** computes proximity between rows or columns of large matrices efficiently in C++. It is optimized for large sparse matrices using the Armadillo and Intel TBB libraries. Among several built-in similarity/distance measures, computation of correlation, cosine similarity and Euclidean distance is particularly fast.

This code was originally written for [**quanteda**](https://github.com/quanteda/quanteda) to compute similarity/distance between documents or features in large corpora, but separated as a stand-alone package to make it available for broader data scientific purposes.

## Install 

Since **proxyC** v0.4.0, it requires the Intel oneAPI Threading Building Blocks for parallel computing. Windows and Mac users can download a binary package from CRAN, but Linux users must install the library by executing the commands below:

```{bash, eval=FALSE}

# Fedora, CentOS, RHEL

sudo yum install tbb-devel

# Debian and Ubuntu

sudo apt install libtbb-dev

```

```{r eval=FALSE}

install.packages("proxyC")

```

## Performance

```{r}

require(Matrix)

require(microbenchmark)

require(ggplot2)

require(magrittr)

# Set number of threads

options("proxyC.threads" = 8)

# Make a matrix with 99% zeros

sm1k <- rsparsematrix(1000, 1000, 0.01) # 1,000 columns

sm10k <- rsparsematrix(1000, 10000, 0.01) # 10,000 columns

# Convert to dense format

dm1k <- as.matrix(sm1k) 

dm10k <- as.matrix(sm10k)

```

## Cosine similarity between columns

With sparse matrices, **proxyC** is roughly 10 to 100 times faster than **proxy**. 

```{r, cahce=TRUE}

bm1 <- microbenchmark(

    "proxy 1k" = proxy::simil(dm1k, method = "cosine"),

    "proxyC 1k" = proxyC::simil(sm1k, margin = 2, method = "cosine"),

    "proxy 10k" = proxy::simil(dm10k, method = "cosine"),

    "proxyC 10k" = proxyC::simil(sm10k, margin = 2, method = "cosine"),

    times = 10

)

autoplot(bm1)

```

##  Cosine similarity greater than 0.9

If `min_simil` is used, **proxyC** becomes even faster because small similarity scores are floored to zero.

```{r, cahce=TRUE}

bm2 <- microbenchmark(

    "proxyC all" = proxyC::simil(sm1k, margin = 2, method = "cosine"),

    "proxyC min_simil" = proxyC::simil(sm1k, margin = 2, method = "cosine", min_simil = 0.9),

    times = 10

)

autoplot(bm2)

```

Flooring by `min_simil` makes the resulting object much smaller.

```{r, cahce=TRUE}

proxyC::simil(sm10k, margin = 2, method = "cosine") %>% 

  object.size() %>% 

  print(units = "MB")

proxyC::simil(sm10k, margin = 2, method = "cosine", min_simil = 0.9) %>% 

  object.size() %>% 

  print(units = "MB")

```

## Top-10 correlation

If `rank` is used, **proxyC** only returns top-n values. 

```{r, cahce=TRUE}

bm3 <- microbenchmark(

    "proxyC rank" = proxyC::simil(sm1k, margin = 2, method = "correlation", rank = 10),

    "proxyC all" = proxyC::simil(sm1k, margin = 2, method = "correlation"),

    times = 10

)

autoplot(bm3)

```