{"id":14989198,"url":"https://github.com/ing-bank/sparse_dot_topn","last_synced_at":"2025-04-11T14:39:16.603Z","repository":{"id":39006525,"uuid":"98980344","full_name":"ing-bank/sparse_dot_topn","owner":"ing-bank","description":"Python package to accelerate the sparse matrix multiplication and top-n similarity selection","archived":false,"fork":false,"pushed_at":"2025-03-31T07:48:21.000Z","size":450,"stargazers_count":404,"open_issues_count":3,"forks_count":85,"subscribers_count":19,"default_branch":"master","last_synced_at":"2025-04-04T08:49:53.175Z","etag":null,"topics":["cosine-similarity","cython","scipy","sparse-matrix"],"latest_commit_sha":null,"homepage":"","language":"C++","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/ing-bank.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGES.md","contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":".github/CODEOWNERS","security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2017-08-01T08:50:48.000Z","updated_at":"2025-03-31T07:48:19.000Z","dependencies_parsed_at":"2022-08-02T20:30:56.645Z","dependency_job_id":"b0446728-3162-4add-b886-b7716ceace4f","html_url":"https://github.com/ing-bank/sparse_dot_topn","commit_stats":{"total_commits":161,"total_committers":20,"mean_commits":8.05,"dds":0.3229813664596274,"last_synced_commit":"ef5f09f7e9497fba89709a762d14553a4e512d5e"},"previous_names":[],"tags_count":18,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ing-bank%2Fsparse_dot_topn","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ing-bank%2Fsparse_dot_topn/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ing-bank%2Fsparse_dot_topn/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ing-bank%2Fsparse_dot_topn/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/ing-bank","download_url":"https://codeload.github.com/ing-bank/sparse_dot_topn/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248420083,"owners_count":21100309,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["cosine-similarity","cython","scipy","sparse-matrix"],"created_at":"2024-09-24T14:17:51.002Z","updated_at":"2025-04-11T14:39:16.581Z","avatar_url":"https://github.com/ing-bank.png","language":"C++","funding_links":[],"categories":[],"sub_categories":[],"readme":"# sparse\\_dot\\_topn\n\n[![MacOS](https://github.com/ing-bank/sparse_dot_topn/actions/workflows/macos.yml/badge.svg)](https://github.com/ing-bank/sparse_dot_topn/actions/workflows/macos.yml)\n[![Linux](https://github.com/ing-bank/sparse_dot_topn/actions/workflows/linux.yml/badge.svg)](https://github.com/ing-bank/sparse_dot_topn/actions/workflows/linux.yml)\n[![Windows](https://github.com/ing-bank/sparse_dot_topn/actions/workflows/windows.yml/badge.svg)](https://github.com/ing-bank/sparse_dot_topn/actions/workflows/windows.yml)\n[![License](https://img.shields.io/github/license/ing-bank/sparse_dot_topn)](https://github.com/ing-bank/sparse_dot_topn/blob/master/LICENSE)\n[![ruff](https://img.shields.io/endpoint?url=https://raw.githubusercontent.com/charliermarsh/ruff/main/assets/badge/v1.json)](https://github.com/charliermarsh/ruff)\n\n\n[![Release_date](https://img.shields.io/github/release-date/ing-bank/sparse_dot_topn)](https://github.com/ing-bank/sparse_dot_topn/releases)\n[![PyPi](https://img.shields.io/pypi/v/sparse-dot-topn.svg)](https://pypi.org/project/sparse-dot-topn/)\n[![Downloads](https://pepy.tech/badge/sparse_dot_topn)](https://pepy.tech/project/sparse_dot_topn)\n\n**sparse\\_dot\\_topn** provides a fast way to performing a sparse matrix multiplication followed by top-n multiplication result selection.\n\nComparing very large feature vectors and picking the best matches, in practice often results in performing a sparse matrix multiplication followed by selecting the top-n multiplication results.\n\n**sparse\\_dot\\_topn** provides a (parallelised) sparse matrix multiplication implementation that integrates selecting the top-n values, resulting in a significantly lower memory footprint and improved performance.\nOn Apple M2 Pro over two 20k x 193k TF-IDF matrices **sparse\\_dot\\_topn** can be up to 6 times faster when retaining the top 10 values per row and utilising 8 cores.\nSee the benchmark directory for details.\n\n## Usage\n\n`sp_matmul_topn` supports `{CSR, CSC, COO}` matrices with `{32, 64}bit {int, float}` data.\nNote that `COO` and `CSC` inputs are converted to the `CSR` format and are therefore slower.\nTwo options to further reduce memory requirements are `threshold` and `density`.\nOptionally, the values can be sorted such that the first column for a given row contains the largest value.\nNote that `sp_matmul_topn(A, B, top_n=B.shape[1])` is equal to `sp_matmul(A, B)` and `A.dot(B)`.\n\n**If you are migrating from `v0.*` please see the migration guide below for details.**\n\n```python\nimport scipy.sparse as sparse\nfrom sparse_dot_topn import sp_matmul, sp_matmul_topn\n\nA = sparse.random(1000, 100, density=0.1, format=\"csr\")\nB = sparse.random(100, 2000, density=0.1, format=\"csr\")\n\n# Compute C and retain the top 10 values per row\nC = sp_matmul_topn(A, B, top_n=10)\n\n# or paralleslised matrix multiplication without top-n selection\nC = sp_matmul(A, B, n_threads=2)\n# or with top-n selection\nC = sp_matmul_topn(A, B, top_n=10, n_threads=2)\n\n# If you are only interested in values above a certain threshold\nC = sp_matmul_topn(A, B, top_n=10, threshold=0.8)\n\n# If you set the threshold we cannot easily determine the number of non-zero\n# entries beforehand. Therefore, we allocate memory for `ceil(top_n * A.shap[0] * density)`\n# non-zero entries. You can set the expected density to reduce the amount pre-allocated\n# entries. Note that if we allocate too little an expensive copy(ies) will need to hapen.\nC = sp_matmul_topn(A, B, top_n=10, threshold=0.8, density=0.1)\n```\n\n## Installation\n\n**sparse\\_dot\\_topn** provides wheels for CPython 3.8 to 3.12 for:\n\n* Windows (64bit)\n* Linux (64bit)\n* MacOS (x86 and ARM)\n\n```shell\npip install sparse_dot_topn\n```\n\n**sparse\\_dot\\_topn** relies on a C++ extension for the computationally intensive multiplication routine.\n**Note that the wheels vendor/ships OpenMP with the extension to provide parallelisation out-of-the-box.**\n**This may cause issues when used in combination with other libraries that ship OpenMP like PyTorch.**\nIf you run into any issues with OpenMP see INSTALLATION.md for help or run the function without specifying the `n_threads` argument.\n\nInstalling from source requires a C++17 compatible compiler.\nIf you have a compiler available it is advised to install without the wheel as this enables architecture specific optimisations.\n\nYou can install from source using:\n\n```shell\npip install sparse_dot_topn --no-binary sparse_dot_topn\n```\n\n### Build configuration\n\n**sparse\\_dot\\_topn** provides some configuration options when building from source.\nBuilding from source can enable architecture specific optimisations and is recommended for those that have a C++ compiler installed.\nSee INSTALLATION.md for details.\n\n## Distributing the top-n multiplication of two large O(10M+) sparse matrices over a cluster\n\nThe top-n multiplication of two large O(10M+) sparse matrices can be broken down into smaller chunks.\nFor example, one may want to split sparse matrices into matrices with just 1M rows, and do the\nthe (top-n) multiplication of all those matrix pairs.\nReasons to do this are to reduce the memory footprint of each pair, and to employ available distributed computing power.\n\nThe pairs can be distributed and calculated over a cluster (eg. we use a spark cluster).\nThe resulting matrix-products are then zipped and stacked in order to reproduce the full matrix product.\n\nHere's an example how to do this, where we are matching 1000 rows in sparse matrix A against 600 rows in sparse matrix B,\nand both A and B are split into chunks.\n\n```python\nimport numpy as np\nimport scipy.sparse as sparse\nfrom sparse_dot_topn import sp_matmul_topn, zip_sp_matmul_topn\n\n# 1a. Example matching 1000 rows in sparse matrix A against 600 rows in sparse matrix B.\nA = sparse.random(1000, 2000, density=0.1, format=\"csr\", dtype=np.float32, random_state=rng)\nB = sparse.random(600, 2000, density=0.1, format=\"csr\", dtype=np.float32, random_state=rng)\n\n# 1b. Reference full matrix product with top-n\nC_ref = sp_matmul_topn(A, B.T, top_n=10, threshold=0.01, sort=True)\n\n# 2a. Split the sparse matrices. Here A is split into three parts, and B into five parts.\nAs = [A[i*200:(i+1)*200] for i in range(5)]\nBs = [B[:100], B[100:300], B[300:]]\n\n# 2b. Perform the top-n multiplication of all sub-matrix pairs, here in a double loop.\n# E.g. all sub-matrix pairs could be distributed over a cluster and multiplied there.\nCs = [[sp_matmul_topn(Aj, Bi.T, top_n=10, threshold=0.01, sort=True) for Bi in Bs] for Aj in As]\n\n# 2c. top-n zipping of the C-matrices, done over the index of the B sub-matrices.\nCzip = [zip_sp_matmul_topn(top_n=10, C_mats=Cis) for Cis in Cs]\n\n# 2d. stacking over zipped C-matrices, done over the index of the A sub-matrices\n# The resulting matrix C equals C_ref.\nC = sparse.vstack(Czip, dtype=np.float32)\n```\n\n## Migrating to v1.\n\n**sparse\\_dot\\_topn** v1 is a significant change from `v0.*` with a new bindings and API.\nThe new version adds support for CPython 3.12 and now supports both ints as well as floats.\nInternally we switched to a max-heap to collect the top-n values which significantly reduces memory-footprint.\nThe former implementation had `O(n_columns)` complexity for the top-n selection where we now have `O(top-n)` complexity.\n**`awesome_cossim_topn` has been deprecated and will be removed in a future version.**\n\nUsers should switch to `sp_matmul_topn` which is largely compatible:\n\nFor example:\n\n```python\nC = awesome_cossim_topn(A, B, ntop=10)\n```\n\ncan be replicated using:\n\n```python\nC = sp_matmul_topn(A, B, top_n=10, threshold=0.0, sort=True)\n```\n\n### API changes\n1. `ntop` has been renamed to `topn`\n2. `lower_bound` has been renamed to `threshold`\n3. `use_threads` and `n_jobs` have been combined into `n_threads`\n4. `return_best_ntop` option has been removed\n5. `test_nnz_max` option has been removed\n6. `B` is auto-transposed when its shape is not compatible but its transpose is.\n\nThe output of `return_best_ntop` can be replicated with:\n\n```python\nC = sp_matmul_topn(A, B, top_n=10)\nbest_ntop = np.diff(C.indptr).max()\n```\n\n### Default changes\n\n1. `threshold` no longer `0.0` but disabled by default\n\nThis enables proper functioning for matrices that contain negative values.\nAdditionally a different data-structure is used internally when collecting non-zero results that has a much lower memory-footprint than previously.\nThis means that the effect of the `threshold` parameter on performance and memory requirements is negligible. \nIf the `threshold` is `None` we pre-compute the number of `non-zero` entries, this can significantly reduce the required memory at a mild (~10%) performance penalty.\n\n2. `sort = False`, the result matrix is no longer sorted by default\n\nThe matrix is returned with the same column order as if not filtering of the top-n results has taken place.\nThis means that when you set `top_n` equal to the number of columns of `B` you obtain the same result as normal multiplication,\ni.e. `sp_matmul_topn(A, B, top_n=B.shape[1])` is equal to `A.dot(B)`.\n\n## Contributing\n\nContributions are very welcome, please see CONTRIBUTING for details.\n\n### Contributors\n\nThis package was developed and is maintained by authors (previously) affiliated with ING Analytics Wholesale Banking Advanced Analytics.\nThe original implementation was based on modified version of Scipy's CSR multiplication implementation.\nYou can read about it in a [blog](https://medium.com/@ingwbaa/https-medium-com-ingwbaa-boosting-selection-of-the-most-similar-entities-in-large-scale-datasets-450b3242e618) [(mirror)](https://www.sun-analytics.nl/posts/2017-07-26-boosting-selection-of-most-similar-entities-in-large-scale-datasets/) written by Zhe Sun.\n\n* [Zhe Sun](https://github.com/ymwdalex/)\n* [Ahmet Erdem](https://github.com/aerdem4)\n* [Stephane Collot](https://github.com/stephanecollot)\n* [Particular Miner](https://github.com/ParticularMiner) (no ING affiliation)\n* [Ralph Urlus](https://github.com/RUrlus)\n* [Max Baak](https://github.com/mbaak)\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fing-bank%2Fsparse_dot_topn","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fing-bank%2Fsparse_dot_topn","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fing-bank%2Fsparse_dot_topn/lists"}