{"id":22188431,"url":"https://github.com/vmikk/goclust","last_synced_at":"2025-03-24T20:14:46.251Z","repository":{"id":231562652,"uuid":"782014940","full_name":"vmikk/goclust","owner":"vmikk","description":"Clustering tool for sparse matrices produced by USEARCH","archived":false,"fork":false,"pushed_at":"2024-05-10T14:02:03.000Z","size":42,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":2,"default_branch":"main","last_synced_at":"2024-05-10T19:35:05.654Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/vmikk.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-04-04T13:27:46.000Z","updated_at":"2024-05-30T14:38:12.694Z","dependencies_parsed_at":"2024-05-30T14:54:10.361Z","dependency_job_id":null,"html_url":"https://github.com/vmikk/goclust","commit_stats":null,"previous_names":["vmikk/single_linkage","vmikk/goclust"],"tags_count":2,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/vmikk%2Fgoclust","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/vmikk%2Fgoclust/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/vmikk%2Fgoclust/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/vmikk%2Fgoclust/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/vmikk","download_url":"https://codeload.github.com/vmikk/goclust/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":245344005,"owners_count":20599867,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-12-02T11:10:34.840Z","updated_at":"2025-03-24T20:14:46.219Z","avatar_url":"https://github.com/vmikk.png","language":"Go","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Clustering tool for sparse matrices produced by USEARCH\n\n## Motivation\n\nThe 32-bit version of USEARCH cannot process large distance matrices due to memory limitations. \nThis can be a significant bottleneck when working with large sequence datasets. \nTo overcome this limitation, we present a tool that performs clustering similarly to the `usearch -cluster_aggd`. \nCurrently, only single linkage and complete linkage methods are implemented.  \n\n## Quick start\n\nFirst, use USEARCH to calculate the distance matrix for your sequences with a maximum distance cutoff:\n```bash\nusearch -calc_distmx seqs.fa -tabbedout mx.txt -maxdist 0.3\n```\n\nNext, perform the clustering using the `goclust` tool:\n```bash\ngoclust --input mx.txt --output clusters.txt --cutoff 0.01 --method single\n```\n\nThis command is an alternative to the USEARCH clustering command:\n```bash \nusearch -cluster_aggd mx.txt -clusterout clusters.txt -id 0.99 -linkage min\n```\n\n\n## Description\n\nThe input for clustering is a \"sparse\" distance matrix \nestimated by `usearch -calc_distmx`, \nwhich only stores a subset of distances, \nomitting pairs with low identities as determined by the `maxdist` threshold. \nThis significantly reduces the time and space required to compute \nand store a matrix for large sequence sets. \nMissing entries in the matrix are assumed to be at the maximum possible distance of 1.0.\n\n## Installation\n\nDownload the `goclust` binary:\n\n```bash\nwget https://github.com/vmikk/goclust/releases/download/0.1/goclust\nchmod +x goclust\n./goclust\n``` \n\n## Usage\n\nThe `goclust` tool is designed for clustering sequences based on a sparse distance matrix.   \n\nUsage example:\n```bash\ngoclust --cutoff \u003cfloat\u003e --includeequal=\u003cbool\u003e --method \u003cstring\u003e --input \u003cfile\u003e --output \u003cfile\u003e\n```\n\nParameters:\n\n- `--cutoff`: This parameter specifies the distance cutoff for clustering. The value must be a floating-point number greater than 0. Clusters are formed by linking sequences that have a pairwise distance less than this cutoff. A lower cutoff value will result in a larger number of smaller clusters, while a higher cutoff may produce fewer, larger clusters.\n\n- `--input`: The path to the input file containing pairwise distances. This file should be a \"sparse\" matrix generated by `usearch -calc_distmx`, where each row contains the distances between a pair of sequences.\n\n- `--output`: The path to the output file where the cluster assignments will be saved. The output file will list each sequence along with its assigned cluster label.\n\n- `--includeequal`: This option determines whether distances equal to the specified cutoff should be included in the clustering process. By default, this option is set to true (`--includeequal=true`), allowing sequences with pairwise distances exactly equal to the cutoff to be included in the same cluster. Setting this option to false (`--includeEqual=false`) changes the clustering to only consider pairwise distances strictly greater than the cutoff value, potentially leading to more, smaller clusters.\n\n- `--method`: Specifies the clustering method to use. Choose `single` for single linkage where a sequence joins a cluster if it is close to any sequence within the cluster, allowing larger clusters with no upper bound on diameter. Choose `complete` for complete linkage (equivalent to maximum linkage), where all sequences in a cluster must be within a certain distance threshold from each other, resulting in generally smaller clusters. The default setting is `single`.\n\n## Benchmarks\n\n### Equivalency of results\n\nClustering results obtained with `goclust` closely match \nthose obtained with `usearch -cluster_aggd`, except for the differences in cluster labels.\nThe [Rand index](https://en.wikipedia.org/wiki/Rand_index) between the two methods is 1, indicating perfect agreement.\n\n### Performance benchmark\n\n**Input data**: `mx.txt` - sparse distance matrix, 24MB, 1,468 unique sequences, 841,080 lines.  \n\nPerformance comparisons are conducted using \n`goclust` v.0.2 (ex-`single_linkage`), \n`usearch` v.11.0.667 (i86linux32), \nand `hyperfine` v.1.18.0:  \n\n```bash\nhyperfine \\\n  --warmup 3 --runs 5 \\\n  --export-markdown SING_BENCH.md \\\n  \"usearch -cluster_aggd mx.txt -clusterout clusters_USEARCH.txt -id 0.99 -linkage min\" \\\n  \"./goclust --input mx.txt --output clusters_SL.txt --cutoff 0.01 --method single\"\n```\n\nThe benchmark results are as follows:\n\n| Command                 |      Mean [s] | Min [s] | Max [s] |    Relative |\n|:------------------------| -------------:| -------:| -------:| -----------:|\n| `usearch -cluster_aggd` | 2.593 ± 0.467 |   2.220 |   3.160 | 5.96 ± 3.93 |\n| `goclust`               | 0.435 ± 0.276 |   0.218 |   0.881 |        1.00 |\n\n\nProcessing of a larger file (11GB, 29,278 unique sequences, 393,645,092 lines), which `usearch -cluster_aggd` fails to handle due to memory limitations, takes approximately 144 seconds.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fvmikk%2Fgoclust","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fvmikk%2Fgoclust","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fvmikk%2Fgoclust/lists"}