{"id":25938641,"url":"https://github.com/karel-brinda/phylign","last_synced_at":"2025-03-04T03:37:48.016Z","repository":{"id":151420291,"uuid":"438872294","full_name":"karel-brinda/Phylign","owner":"karel-brinda","description":"Alignment against all pre-2019 bacteria on laptops within a few hours (former MOF-Search)","archived":false,"fork":false,"pushed_at":"2024-04-17T12:18:10.000Z","size":7439,"stargazers_count":23,"open_issues_count":13,"forks_count":4,"subscribers_count":6,"default_branch":"main","last_synced_at":"2024-08-17T06:41:15.216Z","etag":null,"topics":["assemblies","bacteria","bioinformatics","compression","genomes","indexing","k-mers","mof","phylogenetic-compression","phylogeny"],"latest_commit_sha":null,"homepage":"http://brinda.eu/mof","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"other","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/karel-brinda.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null}},"created_at":"2021-12-16T05:36:22.000Z","updated_at":"2024-08-09T03:48:35.000Z","dependencies_parsed_at":"2023-06-12T12:45:18.495Z","dependency_job_id":"0df3d726-34f1-48eb-8f77-345e7abe2558","html_url":"https://github.com/karel-brinda/Phylign","commit_stats":{"total_commits":491,"total_committers":4,"mean_commits":122.75,"dds":0.4684317718940937,"last_synced_commit":"c940d78def623d8dc4e34c9790ce1014c6d21ced"},"previous_names":["karel-brinda/mof-search"],"tags_count":2,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/karel-brinda%2FPhylign","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/karel-brinda%2FPhylign/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/karel-brinda%2FPhylign/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/karel-brinda%2FPhylign/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/karel-brinda","download_url":"https://codeload.github.com/karel-brinda/Phylign/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":241780472,"owners_count":20019058,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["assemblies","bacteria","bioinformatics","compression","genomes","indexing","k-mers","mof","phylogenetic-compression","phylogeny"],"created_at":"2025-03-04T03:37:47.427Z","updated_at":"2025-03-04T03:37:48.000Z","avatar_url":"https://github.com/karel-brinda.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Phylign – alignment to all pre-2019 bacteria (former MOF-Search)\n\n\u003cp\u003e\n\u003ca href=\"https://brinda.eu/mof\"\u003e\n    \u003cimg src=\"docs/logo_wbg.svg\" align=\"left\" style=\"width:100px;\" /\u003e\n\u003c/a\u003e\nAlignment to\n\u003ca href=\"https://doi.org/10.1371/journal.pbio.3001421\"\u003eall pre-2019 bacteria from ENA\u003c/a\u003e\non standard desktop and laptops computers.\nPhylign uses\n\u003ca href=\"https://brinda.eu/mof\"\u003e\nphylogenetically compressed assemblies and their \u003ci\u003ek\u003c/i\u003e-mer\nindexes\u003c/a\u003e\nto align batches of queries to them by\n\u003ca href=\"https://github.com/lh3/minimap2\"\u003eMinimap 2\u003c/a\u003e,\nall within only several hours.\n\u003c/p\u003e\u003cbr/\u003e\n\n[![Info](https://img.shields.io/badge/Project-Info-blue)](https://brinda.eu/mof)\n[![Paper DOI](https://img.shields.io/badge/paper-10.1101/2023.04.15.536996-14dc3d.svg)](https://doi.org/10.1101/2023.04.15.536996)\n[![GitHub release](https://img.shields.io/github/release/karel-brinda/phylign.svg)](https://github.com/karel-brinda/phylign/releases/)\n[![DOI](https://zenodo.org/badge/DOI/10.5281/zenodo.10828248.svg)](https://doi.org/10.5281/zenodo.10828248)\n[![CI Tests](https://github.com/karel-brinda/phylign/actions/workflows/ci.yaml/badge.svg)](https://github.com/karel-brinda/phylign/actions/)\n\n\u003ch2\u003eContents\u003c/h2\u003e\n\n\u003c!-- vim-markdown-toc GFM --\u003e\n\n* [1. Introduction](#1-introduction)\n  * [Citation](#citation)\n* [2. Requirements](#2-requirements)\n  * [2a) Hardware](#2a-hardware)\n  * [2b) Dependencies](#2b-dependencies)\n* [3. Installation](#3-installation)\n  * [3a) Step 1: Install dependencies](#3a-step-1-install-dependencies)\n  * [3b) Step 2: Clone the repository](#3b-step-2-clone-the-repository)\n  * [3c) Step 3: Run a simple test](#3c-step-3-run-a-simple-test)\n  * [3d) Step 4: Download the database](#3d-step-4-download-the-database)\n* [4. Usage](#4-usage)\n  * [4a) Step 1: Copy or symlink your queries](#4a-step-1-copy-or-symlink-your-queries)\n  * [4b) Step 2: Adjust configuration](#4b-step-2-adjust-configuration)\n  * [4c) Step 3: Clean up intermediate files](#4c-step-3-clean-up-intermediate-files)\n  * [4d) Step 4: Run the pipeline](#4d-step-4-run-the-pipeline)\n  * [4e) Step 5: Analyze your results](#4e-step-5-analyze-your-results)\n* [5. Additional information](#5-additional-information)\n  * [5a) List of workflow commands](#5a-list-of-workflow-commands)\n  * [5b) Directories](#5b-directories)\n  * [5c) File formats](#5c-file-formats)\n  * [5d) Running on a cluster](#5d-running-on-a-cluster)\n  * [5e) Known limitations](#5e-known-limitations)\n* [6. License](#6-license)\n* [7. Contacts](#7-contacts)\n\n\u003c!-- vim-markdown-toc --\u003e\n\n\n## 1. Introduction\n\nThe central idea behind Phylign, enabling alignment locally at such a large\nscale, is [**phylogenetic compression**](https://brinda.eu/mof)\n([paper](https://doi.org/10.1101/2023.04.15.536996)) - a technique based on\nusing estimated evolutionary history to guide compression and search of large\ngenome collections using existing algorithms and data structures.\n\nIn short, input data are reorganized according to the topology of the estimated\nphylogenies, which makes data highly locally compressible even using basic\ntechniques. Existing software packages for compression, indexing, and search\n- in this case [XZ](https://tukaani.org/xz/),\n[COBS](https://github.com/iqbal-lab-org/cobs), and\n[Minimap2](https://github.com/lh3/minimap2) - are then used as low-level tools.\nThe resulting performance gains come from a wide range of benefits of\nphylogenetic compression, including easy parallelization, small memory\nrequirements, small database size, better memory locality, and better branch\nprediction.\n\nFor more information about phylogenetic compression and the implementation\ndetails of Phylign, see the [corresponding\npaper](https://www.biorxiv.org/content/10.1101/2023.04.15.536996v2) (including\nits [supplementary\nmaterial](https://www.biorxiv.org/content/biorxiv/early/2023/04/18/2023.04.15.536996/DC1/embed/media-1.pdf)\nand visit the [associated website](https://brinda.eu/mof).\n\n\n### Citation\n\n\u003e K. Břinda, L. Lima, S. Pignotti, N. Quinones-Olvera, K. Salikhov, R. Chikhi, G. Kucherov, Z. Iqbal, and M. Baym. **[Efficient and Robust Search of Microbial Genomes via Phylogenetic Compression.](https://doi.org/10.1101/2023.04.15.536996)** *bioRxiv* 2023.04.15.536996, 2023. https://doi.org/10.1101/2023.04.15.536996\n\n\n## 2. Requirements\n\n### 2a) Hardware\n\nPhylign requires a standard desktop or laptop computer with an \\*nix system,\nand it can also run on a cluster. The minimal hardware requirements are **12 GB\nRAM** and approximately **120 GB of disk space** (102 GB for the database and\na margin for intermediate files).\n\n\n### 2b) Dependencies\n\nPhylign is implemented as a [Snakemake](https://snakemake.github.io)\npipeline, using the Conda system to manage non-standard dependencies. Ensure\nyou have [Conda](https://docs.conda.io/en/latest/miniconda.html) installed with\nthe following packages:\n\n* [GNU Time](https://www.gnu.org/software/time/) (on Linux present by default; on OS X, install with `brew install gnu-time`).\n* [Python](https://www.python.org/) (\u003e=3.7)\n* [Snakemake](https://snakemake.github.io) (\u003e=6.2.0)\n* [Mamba](https://mamba.readthedocs.io/) (\u003e= 0.20.0) - optional, but recommended\n\nAdditionally, Phylign uses standard Unix tools like\n[GNU Make](https://www.gnu.org/software/make/),\n[cURL](https://curl.se/),\n[XZ Utils](https://tukaani.org/xz/), and\n[GNU Gzip](https://www.gnu.org/software/gzip/).\nThese tools are typically included in standard \\*nix installations. However, in\nminimal setups (e.g., virtualization, continuous integration), you might need\nto install them using the corresponding package managers.\n\n\n## 3. Installation\n\n### 3a) Step 1: Install dependencies\n\nMake sure you have Conda and GNU Time installed. On Linux:\n\n```bash\nsudo apt-get install conda\n```\n\nOn OS X (using Homebrew):\n\n```bash\nbrew install conda\nbrew install gnu-time\n```\n\nInstall Python (\u003e=3.7), Snakemake (\u003e=6.2.0), and Mamba (optional but\nrecommended) using Conda:\n\n```bash\nconda install -y -c bioconda -c conda-forge \\\n    \"python\u003e=3.7\" \"snakemake\u003e=6.2.0\" \"mamba\u003e=0.20.0\"\n```\n\n\n### 3b) Step 2: Clone the repository\n\nClone the Phylign repository from GitHub and navigate into the directory:\n\n```bash\n git clone https://github.com/karel-brinda/phylign\n cd phylign\n```\n\n\n### 3c) Step 3: Run a simple test\n\nRun the following command to ensure the pipeline works for sample queries and\n3 batches (this will also install all additional dependencies using Conda):\n\n```bash\nmake test\n```\n\nMake sure the test returns 0 (success) and that you see the expected output\nmessage:\n\n```bash\n Success! Test run produced the expected output.\n```\n\n\n### 3d) Step 4: Download the database\n\nDownload all phylogenetically compressed assemblies and COBS *k*-mer indexes\nfor the [661k-HQ collection](https://doi.org/10.1371/journal.pbio.3001421) by:\n\n```bash\nmake download\n```\n\nThe downloaded files will be located in the `asms/` and `cobs/` directories.\n\n\n*Notes:*\n* The compressed assemblies comprise *all* the genomes from the 661k\n  collection.The COBS indexes comprise only those genomes that passed quality\n  control.\n\n\n## 4. Usage\n\n### 4a) Step 1: Copy or symlink your queries\n\nRemove the default test files or your old files in the `input/` directory and\ncopy or symlink (recommended) your query files. The supported input formats are\nFASTA and FASTQ, possibly gzipped. All query files will be preprocessed and\nmerged together.\n\n*Notes:*\n* All query names have to be unique among all query files.\n* Queries should not contain non-ACGT characters. All non-`ACGT` characters in your query sequences will be translated to `A`.\n\n\n### 4b) Step 2: Adjust configuration\n\nEdit the [`config.yaml`](config.yaml) file for your desired search. All\navailable options are documented directly there.\n\n### 4c) Step 3: Clean up intermediate files\n\nRun `make clean` to clean intermediate files from the previous runs. This\nincludes COBS matching files, alignment files, and various reports.\n\n### 4d) Step 4: Run the pipeline\n\nSimply run `make`, which will execute Snakemake with the corresponding\nparameters. If you want to run the pipeline step by step, run `make match`\nfollowed by `make map`.\n\n### 4e) Step 5: Analyze your results\n\nCheck the output files in `output/` (for more info about formats, see\n[5c) File formats](#5c-file-formats)).\n\nIf the results do not correspond to what you expected and you need to re-adjust\nyour search parameters, go to Step 2. If only the mapping part is affected by\nthe changes, you proceed more rapidly by manually removing the files in\n`intermediate/05_map` and `output/` and running directly `make map`.\n\n\n## 5. Additional information\n\n### 5a) List of workflow commands\n\nPhylign is executed via [GNU Make](https://www.gnu.org/software/make/),\nwhich handles all parameters and passes them to Snakemake.\n\nHere's a list of all implemented commands (to be executed as `make {command}`):\n\n\n```bash\n####################\n# General commands #\n####################\n   all                Run everything (the default rule)\n   test               Quick test using 3 batches\n   help               Print help messages\n   clean              Clean intermediate search files\n   cleanall           Clean all generated and downloaded files\n##################\n# Pipeline steps #\n##################\n   conda              Create the conda environments\n   download           Download the assemblies and COBS indexes\n   download_asms      Download only the assemblies\n   download_cobs      Download only the COBS indexes\n   match              Match queries using COBS (queries -\u003e candidates)\n   map                Map candidates to assemblies (candidates -\u003e alignments)\n#############\n# Reporting #\n#############\n   config             Print configuration without comments\n   report             Generate Snakemake report\n###########\n# Cluster #\n###########\n   cluster_slurm      Submit to a SLURM cluster\n   cluster_lsf        Submit to LSF cluster\n   cluster_lsf_test   Submit the test pipeline to LSF cluster\n##################\n# For developers #\n##################\n   format             Reformat Python and Snakemake files\n   checkformat        Check source code format\n```\n\n*Note:* `make format` requires\n[YAPF](https://github.com/google/yapf) and\n[Snakefmt](https://github.com/snakemake/snakefmt), which can be installed by\n`conda install -c conda-forge -c bioconda yapf snakefmt`.\n\n\n### 5b) Directories\n\n* `asms/`, `cobs/` Downloaded assemblies and COBS indexes\n* `input/` Queries, to be provided within one or more FASTA/FASTQ files,\n  possibly gzipped (`.fa`)\n* `intermediate/` Intermediate files\n   * `00_queries_preprocessed/` Preprocessed queries\n   * `01_queries_merged/` Merged queries\n   * `02_cobs_decompressed/` Decompressed COBS indexes (temporary, used only in\n     the disk mode is used)\n   * `03_match/` COBS matches\n   * `04_filter/` Filtered candidates\n   * `05_map/` Minimap2 alignments\n* `logs/` Logs and benchmarks\n* `output/` The resulting files (in a headerless SAM format)\n\n\n\n### 5c) File formats\n\n**Input files:** FASTA or FASTQ files possibly compressed by gzipped. The files\nare searched in the `input/` directory, as files with the following suffixes:\n`.fa`, `.fasta`, `.fq`, `.fastq` (possibly with `.gz` at the end).\n\n**Output files:**\n\n* `output/{name}.sam_summary.gz`: output alignments in a headerless SAM format\n* `output/{name}.sam_summary.stats`: statistics about your computed alignments\n  in TSV\n\nSAM headers are omitted as all search experiments\ngenerate hits across large numbers of assemblies (many\nof them being spurious). As a result, SAM headers then\ndominate the outputs. Nevertheless, we note that, in\nprinciple, the SAM headers can always be recreated from the\nFASTA files in `asms/`, although this functionality is not\ncurrently implemented.\n\n\n### 5d) Running on a cluster\n\nRunning on a cluster is much faster as the jobs produced by this pipeline are\nquite light and usually start running as soon as they are scheduled.\n\n**For LSF clusters:**\n\n1. Test if the pipeline is working on a LSF cluster: `make cluster_lsf_test`;\n2. Configure you queries and run the full pipeline: `make cluster_lsf`;\n\n\n### 5e) Known limitations\n\n* **Swapping if the number of queries too high.** If the number of queries is\n  too high (e.g., 10M Illumina reads), the auxiliary Python scripts start to\n  use too much memory, which may result in swapping. Try to keep the number of\n  queries moderate and ideally their names short.\n* **No support for ambiguous characters in queries.** Queries are expected to\n  be over the ACGT alphabet. All non-ACGT characters in queries are first\n  converted to A.\n* **Too many reported hits.** When queries have too many equally good hits in\n  the database, even if the threshold on the maximum number of hits is chosen\n  low – for instance 10 – the program will take top 10 + ties, which can be\n  a huge number (especially for short sequences).\n\n\n## 6. License\n\n[MIT](https://github.com/karel-brinda/phylign/blob/master/LICENSE)\n\n\n\n## 7. Contacts\n\n* [Karel Brinda](https://brinda.eu) \\\u003ckarel.brinda@inria.fr\\\u003e\n* [Leandro Lima](https://github.com/leoisl) \\\u003cleandro@ebi.ac.uk\\\u003e\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fkarel-brinda%2Fphylign","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fkarel-brinda%2Fphylign","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fkarel-brinda%2Fphylign/lists"}