{"id":50557168,"url":"https://github.com/g-insana/proteomecluster","last_synced_at":"2026-06-04T08:02:09.875Z","repository":{"id":358069177,"uuid":"1191165357","full_name":"g-insana/ProteomeCluster","owner":"g-insana","description":"NextFlow pipelines and Python scripts for comprehensive analyses of proteomes data, providing: Clustering via MMseqs2; Labelling, filtering and aggregation of clusters; Alignments and Variation analyses; Meta clustering comparisons","archived":false,"fork":false,"pushed_at":"2026-05-15T15:07:19.000Z","size":6918,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2026-05-15T16:29:45.861Z","etag":null,"topics":["alignments","mmseqs2","nextflow-pipeline","proteome","proteome-clustering","proteomes","variation","variome"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/g-insana.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":".github/FUNDING.yml","license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null},"funding":{"custom":["https://paypal.me/GiuseppeInsana","http://mediaglyphs.org/?p=donations","https://www.kraken.com/pay/giuseppeinsana"]}},"created_at":"2026-03-25T01:26:23.000Z","updated_at":"2026-05-15T15:07:23.000Z","dependencies_parsed_at":null,"dependency_job_id":null,"html_url":"https://github.com/g-insana/ProteomeCluster","commit_stats":null,"previous_names":["g-insana/proteomecluster"],"tags_count":1,"template":false,"template_full_name":null,"purl":"pkg:github/g-insana/ProteomeCluster","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/g-insana%2FProteomeCluster","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/g-insana%2FProteomeCluster/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/g-insana%2FProteomeCluster/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/g-insana%2FProteomeCluster/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/g-insana","download_url":"https://codeload.github.com/g-insana/ProteomeCluster/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/g-insana%2FProteomeCluster/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":33895175,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-26T15:22:16.424Z","status":"online","status_checked_at":"2026-06-04T02:00:06.755Z","response_time":64,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["alignments","mmseqs2","nextflow-pipeline","proteome","proteome-clustering","proteomes","variation","variome"],"created_at":"2026-06-04T08:02:08.822Z","updated_at":"2026-06-04T08:02:09.862Z","avatar_url":"https://github.com/g-insana.png","language":"Python","funding_links":["https://paypal.me/GiuseppeInsana","http://mediaglyphs.org/?p=donations","https://www.kraken.com/pay/giuseppeinsana"],"categories":[],"sub_categories":[],"readme":"# ProteomeCluster - Proteome Clustering \u0026 Comparisons Framework\n[![DOI](https://zenodo.org/badge/1191165357.svg)](https://doi.org/10.5281/zenodo.20208646)\n\nPipelines and tools for clustering, comparing and analyzing proteomes at scale.\n\nProteomeCluster combines a **Nextflow pipeline** with **Python utilities** to perform clustering, filtering, alignments and other comparative analyses of proteomes.\n\n---\n\n## 🚀 Overview\n\nProteomeCluster enables:\n\n- High-performance **clustering** using MMseqs2\n- Flexible **cluster filtering and annotation**\n- Optional **sequence extraction and multiple alignment**\n- Advanced **comparison of clustering strategies**\n- Exploration of **merge/split cluster events**\n- Analyses of **protein frequency** across isolates\n- Inter- and intra- cluster **sequence variation**\n- **Variome analysis**\n\nThe pipeline is designed to run both **locally** and on **HPC systems (SLURM/LSF)**.\nThe single processes can also be run independently via the provided python scripts.\n\n---\n\n## ✨ Features\n\n### 🔹 Basic Workflow\n\n- Clusters proteins using `mmseqs2`\n- Labels cluster members by proteome of origin\n- Filters clusters based on proteome representation\n\n**Outputs:**\n(for each species or dataset)\n- `*_clusters.tsv`\n  - Cluster membership\n  - Sequence length range and mode\n- `*_matrix.tsv`\n  - presence absence matrix with row and column counts\n\n---\n\n### 🔹 Extended Workflow\n\nOptional downstream processing after clustering:\n\n- Extract sequences per cluster\n- Align them by cluster\n\n**Outputs:**\n\n- Cluster FASTA files\n- Cluster alignment files\n\n---\n\n### 🔹 Meta-Clustering Comparison\n\nCompare different clustering runs (e.g. parameter tuning):\n\n- Compute similarity metrics\n- Identify overlapping clusters\n- Detect cluster **merge/split events**\n\n---\n\n### 🔹 Variation Analysis (WIP)\n\n- Framework for building and analyzing **variomes**\n\n---\n\n## 📦 Requirements\n\n- [MMseqs2](https://github.com/soedinglab/MMseqs2)\n- [Nextflow](https://www.nextflow.io/)\n- Python ≥ 3.8\n\nOptional:\n\n- [ffdb](https://github.com/g-insana/ffdb.py) (for sequence extraction)\n- [Biopython](https://biopython.org/) (for alignment processing)\n- Clustal Omega (`clustalo`) (for multiple sequence alignments)\n- [scikit-learn](https://scikit-learn.org) (for meta-clustering metrics)\n\n---\n\n## 📥 Installation\n\n```bash\ngit clone https://github.com/g-insana/ProteomeCluster.git\n```\n\n---\n\n## ⚡ Quickstart 1 2 3\n\n### 1. Prepare Input\n\nPlace proteome FASTA files under:\n```\nproteomesdir/\n```\n\nYou can organize them in subfolders (e.g. per species or per dataset) and they will be processed in parallel.\n\n---\n\n### 2. Run Pipeline\neither locally or on HPC, choosing your preferred parameters, editing the following scripts - or [running directly from command line](#command-line)\n\n#### Basic workflow\n\n```bash\n./cluster_only-local.sh\n./cluster_only-slurm.sh\n```\n\n#### Extended workflow\n\n```bash\n./cluster_and_align-local.sh\n./cluster_and_align-slurm.sh\n```\n\n---\n\n### 3. Results\n\nOutput files will be written to:\n```\noutdir/\n```\n\n---\n\n### 🧪 Test Dataset\n\nIncluded:\n\n- 10 proteomes from [UniProtKB](https://https://www.uniprot.org)\n- belonging to two species\n\nRun immediately without setup.\n\n---\n\n## Command Line\n\nInstead of the provided scripts you can for example run it on command line, e.g.:\n\n```bash\nnextflow run cluster_and_align.nf \\\n  --min_seq_id 0.9 \\\n  --coverage 0.9 \\\n  --proteomes_threshold 3 \\\n  -profile local -resume\n```\n\nSee full list of parameters below in the [Documentation](#-documentation) section.\n\n---\n\n## 🧠 Best Practices\n\n- Start with **strict thresholds** (for coverage, sequence identity, covmode) for high-confidence clusters\n- Use **meta-clustering tools** (see below for the Documentation) to tune parameters and compare with the high-confidence clustering\n- Align `maxForks` and threads on HPC systems to your resources and needs\n- Use `--extract` and `--align` only if needed\n\n---\n\n## 📚 Documentation\n\n### Nextflow Pipeline\n#### Core Parameters\n\n| Parameter | Description |\n|----------|------------|\n| `--proteomesdir` | Input proteome directory |\n| `--outdir` | Output directory |\n| `--proteomes_threshold` | Minimum proteomes per cluster |\n| `--extract` | Extract sequences |\n| `--align` | Perform alignments |\n\n---\n\n#### MMseqs2 Clustering\n\n| Parameter | Description |\n|----------|------------|\n| `--covmode` | Coverage mode (0=bidirectional, 1=target, 2=query) |\n| `--coverage` | Minimum alignment coverage (0–1) |\n| `--min_seq_id` | Minimum sequence identity (0–1) |\n| `--seq_id_mode` | Identity calculation mode |\n\n---\n\n#### Performance\n\n| Parameter | Description |\n|----------|------------|\n| `--clusterthreads` | Threads for clustering |\n| `--labelthreads` | Threads for labeling |\n| `--extractthreads` | Threads for extraction |\n| `--alignthreads` | Threads for alignment |\n| `--clustalothreads` | Threads for Clustal Omega |\n| `--maxForks` | Max parallel processes |\n\n---\n\n#### Advanced\n\n```bash\n--mmseqs_additional \"--clust-hash 1 --kmer-per-seq 100 ...\"\n```\n(custom MMseqs2 parameters for tuning speed vs sensitivity vs accuracy)\n\n### Cluster Comparison Tools\n\nTwo scripts are provided:\n\n### 1️⃣ compare_clusters.py\n\nCompare two clustering results:\n- compute overlap between protein identifiers of the two clusterings\n- map clusters according to Jaccard and Szymkiewicz–Simpson thresholds (IoU and containment)\n- write the mapping between clusters of the two clustering results\n- identify split/merge/complex events\n- compute similarity metrics (`adjusted_rand_score` and `adjusted_mutual_info_score`)\n\n```bash\ncompare_clusters.py clusters_file1.tsv clusters_file2.tsv\n```\n\n#### Key Options\n\n| Option | Description |\n|-------|------------|\n| `-m` | Minimum proteomes per cluster |\n| `-j` | Minimum Jaccard index (default 0.3) |\n| `-c` | Containment threshold: Szymkiewicz–Simpson coefficient (default 0.9) |\n| `-t` | Max proteomes per cluster |\n| `-p` | Protein ID format |\n| `-o` | Output prefix |\n\n#### Optional Flags\n\n- `--skip_metrics`\n- `--skip_writing_overlap`\n- `--skip_writing_mapping`\n\n---\n\n### 2️⃣ inspect_merge_splits.py\n\nAnalyze cluster merge/split events:\n- extract clusters involved (sampling if desired)\n- align them\n- prepare list of alignments for visual comparison\n\n```bash\ninspect_merge_splits.py mergesplits.tsv mapping.tsv file1.tsv file2.tsv\n```\n\n#### Key Options (all optional)\n\n| Option | Description |\n|-------|------------|\n| `-os` | Output suffix |\n| `-op` | Output prefix |\n| `-m` | Max sampled events |\n| `-t` | Threads |\n\n\n### Variation Analysis\n(WIP)\n\n### Individual processes\n\nInstead of running the Nextflow pipeline you can use the individual Python scripts to execute specific steps independently. This is useful for debugging, custom workflows, or integrating ProteomeCluster components into other pipelines.\n\nBelow is a brief documentation for each script. Complete documentation is available (simply click on the script name below).\n\n---\n\n## 🏷️ Labelling\n\n### [`label_clusters.py`](docs/label_clusters.md)\n\nThis script labels the second column of a TSV (protein IDs) with the FASTA filename(s) (proteome identifiers) where each protein appears, compacts cluster IDs to sequential integers, and flags the representative protein for each cluster. Input row order is preserved.\n\n```bash\nusage: label_clusters.py [-h] -f FASTA_DIR -i INPUT_FILE -o OUT_FILE [-p PREFIX] [-e EXTENSION]\n                         [-n NOLABEL] [-s] [-u] [-t THREADS] [-b BATCHSIZE] [-c CHUNKSIZE] [-q]\n```\n\n---\n\n## 🔍 Filtering\n\n### [`filter_clusters.py`](docs/filter_clusters.md)\n\nScript to extract lists of protein identifiers for clusters defined in a tsv file\n    (like the output files of `label_clusters.py`)\nThe file is assumed to contain at least three columns, with the first three\nbeing `cluster_id`, `protein_id` and `proteomes`.\n\n\n```bash\nusage: filter_clusters.py [-h] -i INPUT_FILE -o OUT_FILE [-m MINPROTEOMES] [-t TOPPROTEOMES] [-q] [-s]\n                          [-a]\n```\n\n---\n\n## 📦 Extraction\n\n### [`extract_clusters.py`](docs/extract_clusters.md)\n\nThis script extracts FASTA sequences for protein identifiers grouped by cluster from a TSV (like the output of `filter_clusters.py`), retrieving sequences from ffdb-indexed FASTA files and writing one FASTA file per cluster.\n\n```bash\nusage: extract_clusters.py [-h] (-d FASTA_DIR | -s SINGLE_FILE | -w URL) -i INPUT_FILE -o OUT_DIR [-u]\n                           [-a] [-ut] [-m MINPROTEINS] [-r RESTRICT] [-c CLUSTER [CLUSTER ...]]\n                           [-p PREFIX] [-e EXTENSION] [-t THREADS] [-f] [-q]\n\n```\n\n---\n\n## 🧬 Alignment\n\n### [`dedup_align_fastafiles.py`](docs/dedup_align_fastafiles.md)\n\nWrapper around Clustal Omega that can find and deduplicate sequences across FASTA files, run alignments and optionally create mapping files of duplicates.\n\n```bash\nusage: dedup_align_fastafiles.py [-h]\n                                 (-d INPUT_FASTA_DIR | -l INPUT_FASTA_LIST | -s FASTA_FILE [FASTA_FILE ...])\n                                 [-t THREADS] [-e EXTENSION] [-a] [-q] [-m] [-c] -o OUT_DIR\n                                 [-at ALIGN_THREADS] [-f] [-st {Protein,DNA}]\n                                 [-of {fasta,clustal,stockholm}]\n```\n\n---\n\n## 🧮 Presence/Absence Matrix\n\n### [`presence_matrix.py`](docs/presence_matrix.md)\n\nScript to create a presence/absence (and count) matrix where the columns\nare proteomes, the rows are clusters and the values are either 0 if there are no proteins from a\ncolumn proteome in the row cluster or a positive integer corresponding to the number of proteins from\nthat proteome in the cluster.\n\n```bash\nusage: presence_matrix.py [-h] -i INPUT_FILE -p PROTEOMES -o OUT_FILE [-m MEMBERS_FILE] [-l LABELS]\n                          [-a ADDITIONAL] [-t] [-c] [-s]\n```\n\n---\n\n## ⚡FASTA Indexing (Extra Utility)\n\n### [`index_fastafiles.py`](docs/index_fastafiles.md)\n\nIndexes FASTA files for fast random access. It supports indexing a single file, a list of files or all the files with a given extension present in a directory.\n\n```bash\nusage: index_fastafiles.py [-h] (-i INPUT_FILE | -l LIST | -d FASTA_DIR) [-e EXTENSION] [-f] [-q]\n                           [-p PATTERN] [-t THREADS]\n```\n\n---\n\n## 📜 License\n\nProteomeCluster is licensed under the [Apache License 2.0](https://www.apache.org/licenses/LICENSE-2.0.html)\n\n(c) Copyright [Giuseppe Insana](https://insana.net), 2024-\n\n---\n\n## 🤝 Contributions\n\nContributions, issues, and feature requests are welcome.\n\n---\n\n## 📌 Citation\n\nIf you use ProteomeCluster in your research, please consider citing this repository:\n```\nGiuseppe Insana. (2026). g-insana/ProteomeCluster: Initial Release (v1.0.0). Zenodo. https://doi.org/10.5281/zenodo.20208647\n```\n\nBibtex:\n```\n@software{giuseppe_insana_2026_20208647,\n  author       = {Giuseppe Insana},\n  title        = {g-insana/ProteomeCluster: Initial Release},\n  month        = may,\n  year         = 2026,\n  publisher    = {Zenodo},\n  version      = {v1.0.0},\n  doi          = {10.5281/zenodo.20208647},\n  url          = {https://doi.org/10.5281/zenodo.20208647},\n}\n```\n\n---\n\n## 🏗️ Used in\n\n[Early terminated transcripts and missing proteins reflect artifacts in bacterial proteomes](https://github.com/g-insana/ett_ms)\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fg-insana%2Fproteomecluster","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fg-insana%2Fproteomecluster","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fg-insana%2Fproteomecluster/lists"}