{"id":25938667,"url":"https://github.com/karel-brinda/miniphy","last_synced_at":"2026-03-10T09:02:08.382Z","repository":{"id":151415422,"uuid":"388855191","full_name":"karel-brinda/MiniPhy","owner":"karel-brinda","description":"Phylogenetic compression of extremely large genome collections [661k ↘𝟭𝟲𝗚𝗶𝗕 | BIGSIdata ↘𝟰𝟴𝗚𝗶𝗕 | AllTheBact'23 ↘𝟳𝟱𝗚𝗶𝗕]","archived":false,"fork":false,"pushed_at":"2025-04-01T18:24:35.000Z","size":6138,"stargazers_count":22,"open_issues_count":14,"forks_count":1,"subscribers_count":6,"default_branch":"main","last_synced_at":"2025-04-01T18:51:09.354Z","etag":null,"topics":["assemblies","bacteria","bioinformatics","compression","genomes","indexing","k-mers","mof","phylogenetic-compression","phylogeny","prophyle"],"latest_commit_sha":null,"homepage":"https://brinda.eu/mof","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"other","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/karel-brinda.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2021-07-23T16:03:18.000Z","updated_at":"2025-02-28T22:53:14.000Z","dependencies_parsed_at":"2023-10-30T10:23:14.499Z","dependency_job_id":"76522c12-c01b-4dea-a4fc-e75a08b061b9","html_url":"https://github.com/karel-brinda/MiniPhy","commit_stats":{"total_commits":358,"total_committers":3,"mean_commits":"119.33333333333333","dds":"0.16201117318435754","last_synced_commit":"487262ed19539e5188bcd47ef3d59d187a93045b"},"previous_names":["karel-brinda/miniphy"],"tags_count":4,"template":false,"template_full_name":null,"purl":"pkg:github/karel-brinda/MiniPhy","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/karel-brinda%2FMiniPhy","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/karel-brinda%2FMiniPhy/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/karel-brinda%2FMiniPhy/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/karel-brinda%2FMiniPhy/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/karel-brinda","download_url":"https://codeload.github.com/karel-brinda/MiniPhy/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/karel-brinda%2FMiniPhy/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":30328269,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-03-10T05:25:20.737Z","status":"ssl_error","status_checked_at":"2026-03-10T05:25:17.430Z","response_time":106,"last_error":"SSL_read: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["assemblies","bacteria","bioinformatics","compression","genomes","indexing","k-mers","mof","phylogenetic-compression","phylogeny","prophyle"],"created_at":"2025-03-04T03:37:49.888Z","updated_at":"2026-03-10T09:02:08.321Z","avatar_url":"https://github.com/karel-brinda.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# MiniPhy – Minimization via Phylogenetic compression (former MOF-Compress)\n\n\n\u003cp\u003e\n\u003ca href=\"https://brinda.eu/mof\"\u003e\n    \u003cimg src=\"docs/logo_wbg.svg\" align=\"left\" style=\"width:100px;\" /\u003e\n\u003c/a\u003e\nWorkflow for \u003ca href=\"http://brinda.eu/mof\"\u003ephylogenetic compression\u003c/a\u003e\nof microbial genomes, producing highly compressed \u003ccode\u003e.tar.xz\u003c/code\u003e genome archives.\nMiniPhy first estimates the evolutionary history\nof user-provided genomes\nand then uses it for guiding their compression using XZ.\nThe resulting archives can be distributed to users or\nre-compressed/indexed by other methods.\nFor more information,\nsee the \u003ca href=\"https://brinda.eu/mof\"\u003ewebsite of phylogenetic compression\u003c/a\u003e\nand the \u003ca href=\"https://doi.org/10.1101/2023.04.15.536996\"\u003eassociated paper\u003c/a\u003e.\n\u003c/p\u003e\u003cbr/\u003e\n\n[![Info](https://img.shields.io/badge/Project-Info-blue)](https://brinda.eu/mof)\n[![Paper DOI](https://img.shields.io/badge/paper-10.1101/2023.04.15.536996-14dc3d.svg)](https://doi.org/10.1101/2023.04.15.536996)\n[![GitHub release](https://img.shields.io/github/release/karel-brinda/miniphy.svg)](https://github.com/karel-brinda/miniphy/releases/)\n[![DOI](https://zenodo.org/badge/DOI/10.5281/zenodo.10636846.svg)](https://doi.org/10.5281/zenodo.10636846)\n[![MiniPhy test](https://github.com/karel-brinda/miniphy/actions/workflows/main.yaml/badge.svg)](https://github.com/karel-brinda/miniphy/actions/)\n\n\n\u003ch2\u003eContents\u003c/h2\u003e\n\n\u003c!-- vim-markdown-toc GFM --\u003e\n\n* [1. Introduction](#1-introduction)\n* [2. Dependencies](#2-dependencies)\n    * [2a. Essential dependencies](#2a-essential-dependencies)\n    * [2b. Protocol-specific dependencies](#2b-protocol-specific-dependencies)\n* [3. Installation](#3-installation)\n* [4. Usage](#4-usage)\n    * [4a. Basic example](#4a-basic-example)\n    * [4b. Adjusting configuration](#4b-adjusting-configuration)\n    * [4c. List of implemented protocols](#4c-list-of-implemented-protocols)\n    * [4d. List of workflow commands](#4d-list-of-workflow-commands)\n    * [4e. Running on a cluster](#4e-running-on-a-cluster)\n    * [4f. Troubleshooting](#4f-troubleshooting)\n* [5. Citation](#5-citation)\n* [6. Issues](#6-issues)\n* [7. Changelog](#7-changelog)\n* [8. License](#8-license)\n* [9. Contacts](#9-contacts)\n\n\u003c!-- vim-markdown-toc --\u003e\n\n\n## 1. Introduction\n\nThe user provides files of files for individual batches\nin the `input/` directory\nand specifies the requested compression protocols in the\n[configuration file](config.yaml).\nIt is assumed that the input genomes are provided as batches of\nphylogenetically related genomes, of up to approx. 10k genomes per batch\n(for more information on batching strategies,\nsee the [paper](http://doi.org/10.1101/2023.04.15.536996)).\nUpon the execution by `make`,\nMiniPhy performs phylogenetic compression\nof the assemblies or associated de Bruijn graphs.\nAll the compressed outputs and the calculated statistics\nare then placed in `output/`.\n\n\n\n## 2. Dependencies\n\n### 2a. Essential dependencies\n\n* [Conda](https://docs.conda.io/en/latest/miniconda.html) (unless the use of Conda is switched off in the configuration) and ideally also [Mamba](https://mamba.readthedocs.io/) (\u003e= 0.20.0)\n* [GNU Make](https://www.gnu.org/software/make/)\n* [Python](https://www.python.org/) (\u003e=3.7)\n* [Snakemake](https://snakemake.github.io) (\u003e=6.2.0)\n* [XZ](https://tukaani.org/xz/)\n\nand can be installed by Conda by\n```bash\nconda install -c conda-forge -c bioconda -c defaults \\\n  make \"python\u003e=3.7\" \"snakemake-minimal\u003e=6.2.0\" \"mamba\u003e=0.20.0\"\n```\n\n### 2b. Protocol-specific dependencies\n\nThese are installed automatically by\nSnakemake when they are requested;\nfor instance, ProPhyle is not installed unless Protocol 3 is used.\nThe specifications of individual environments\ncan be found in [`workflow/envs/`](workflow/envs/),\nand they contain:\n[Attotree](https://github.com/karel-brinda/attotree),\n[ETE 3](http://etetoolkit.org/),\n[SeqTK](https://github.com/lh3/seqtk),\n[xopen](https://pypi.org/project/xopen/),\n[Pandas](https://pandas.pydata.org/),\n[Jellyfish 2](https://github.com/gmarcais/Jellyfish),\n[ProphAsm](https://github.com/prophyle/prophasm),\nand [ProPhyle](https://prophyle.github.io).\n\n\nAll non-essential dependencies across all protocols can also be\ninstalled at once by `make conda`.\n\n\n\n## 3. Installation\n\nClone and enter the repository by\n\n```bash\ngit clone https://github.com/karel-brinda/miniphy\ncd miniphy\n```\n\nAlternatively, the repository can also be installed using cURL by\n```bash\nmkdir miniphy\ncd miniphy\ncurl -L https://github.com/karel-brinda/miniphy/tarball/main \\\n    | tar xvf - --strip-components=1\n```\n\n\n## 4. Usage\n\n### 4a. Basic example\n\n* ***Step 1: Provide lists of input files.*** \\\n  For every batch, create a txt list of input files in the `input/`\n  directory (i.e., as `input/{batch_name}.txt`. Use either absolute paths (recommended),\n  or paths relative to the root of the Github repository (not relative to the txt files).\n\n  Such a list can be generated, for instance, by `find` by\n  ```bash\n  find ~/dir_with_my_genomes -name '*.fa' \u003e input/my_first_batch.txt\n  ```\n  The supported input file formats include FASTA and FASTQ (possibly compressed by GZip).\n\n* ***Step 2 (optional): Provide corresponding phylogenies.*** \\\n  Instead of estimating phylogenies by [Attotree](https://github.com/karel-brinda/attotree)\n  (similar functionality like [Mashtree](https://github.com/lskatz/mashtree)),\n  it is possible to supply custom phylogenies in the Newick format.\n  The tree files should be named `input/{batch_name}.nw`,\n  and the leave names inside should correspond\n  to FASTA filenames (without FASTA suffixes).\n\n* ***Step 3 (optional): Adjust configuration.*** \\\n  By editing [`config.yaml`](config.yaml) it is possible to specify\n  compression protocols, data analyzes,\n  and low-level parameters (see below).\n\n* ***Step 4: Run the pipeline.*** \\\n  Run the pipeline by `make`; this is run\n  Snakemake with the corresponding parameters.\n\n* ***Step 5: Retrieve the output files.*** \\\n  All output files will be located in `output/`.\n\n\n### 4b. Adjusting configuration\n\nThe workflow can be configured via the [`config.yaml`](./config.yaml) file, and\nall options are documented directly there. The configurable functionality includes:\n* switching off Conda,\n* protocols to use (asm, dGSs, dBGs with propagation),\n* analyzes to include (sequence and *k*-mer statistics),\n* *k* for de Bruijn graph and *k*-mer counting,\n* Attotree parameters (phylogeny estimation),\n* XZ parameters (low-level compression), or\n* JellyFish parameters (*k*-mer counting).\n\n\n### 4c. List of implemented protocols\n\n\u003ctable\u003e\n\n\u003cthead\u003e\n  \u003ctd\u003eProtocol\n  \u003ctd\u003eRepresentation\n  \u003ctd\u003eDescription\n  \u003ctd\u003eProduct\n\n\n\u003ctr\u003e\n\n  \u003ctd\u003e\n    \u003cb\u003eProtocol\u0026nbsp;1\u003cbr /\u003e\n    (default)\u003c/b\u003e\n\n  \u003ctd\u003e\n    Assemblies\n\n  \u003ctd\u003e\n    Left-to-right reordering of the assemblies according to the phylogeny\n\n  \u003ctd\u003e\n    \u003ccode\u003eoutput/asm/{batch}.tar.xz\u003c/code\u003e\u003cbr/\u003e\n    original assemblies in FASTA \u003csup\u003e\u003cb\u003e(1)\u003c/b\u003e\u003c/sup\u003e\n\n\n\u003ctr\u003e\n\n  \u003ctd\u003e\n    \u003cb\u003eProtocol\u0026nbsp;2\u003c/b\u003e\u003cbr /\u003e\n    (optional)\n\n  \u003ctd\u003e\n    de Bruijn graphs\n\n  \u003ctd\u003e\n    \u003ca href=\"https://doi.org/10.1186/s13059-021-02297-z\"\u003eSimplitigs\u003c/a\u003e\n    from individual assemblies, left-to-right reordering of their files\n\n  \u003ctd\u003e\n    \u003ccode\u003eoutput/pre/{batch}.tar.xz\u003c/code\u003e\u003cbr/\u003e\n    with simplitig text files,\n    representing individual de Bruijn graphs\n\n\n\u003ctr\u003e\n\n  \u003ctd\u003e\n    \u003cb\u003eProtocol\u0026nbsp;3\u003c/b\u003e\u003cbr /\u003e\n    (optional)\n\n  \u003ctd\u003e\n    de Bruijn graphs\n\n  \u003ctd\u003e\n    Bottom-up \u003ci\u003ek\u003c/i\u003e-mer propagation using \u003ca href=\"http://prophyle.github.io\"\u003eProPhyle\u003c/a\u003e,\n    \u003ca href=\"https://doi.org/10.1186/s13059-021-02297-z\"\u003esimplitigs\u003c/a\u003e\n    at individual nodes of the tree, and left-to-right re-ordering of the obtained files\n\n  \u003ctd\u003e\n    \u003ccode\u003eoutput/post/{batch}.tar.xz\u003c/code\u003e\u003cbr/\u003e\n    \u003ccode\u003eoutput/post/{batch}.nw\u003c/code\u003e\u003cbr/\u003e\n    simplitig text files per individual nodes of the tree \u003csup\u003e\u003cb\u003e(2)\u003c/b\u003e\u003c/sup\u003e\n\n\u003c/table\u003e\n\n\n\u003csmall\u003e\n  \u003csup\u003e\u003cb\u003e(1)\u003c/b\u003e\u003c/sup\u003e In FASTA 1-line format and all sequences converted to uppercase\n  (unless switche off in the configuration).\n  \u003cbr /\u003e\n  \u003csup\u003e\u003cb\u003e(2)\u003c/b\u003e\u003c/sup\u003e The original de Bruijn graphs can\n  be obtained by merging \u003ci\u003ek\u003c/i\u003e-mer sets along\n  the respetive root-to-leaf paths.\n\u003c/small\u003e\n\n\n### 4d. List of workflow commands\n\nMiniPhy is executed via [GNU Make](https://www.gnu.org/software/make/), which handles all parameters and passes them to Snakemake.\nHere's a list of all implemented commands (to be executed as `make {command}`):\n\n\n```yaml\n######################\n## General commands ##\n######################\n    all                  Run everything (the default subcommand)\n    help                 Print help messages\n    conda                Create the conda environments\n    clean                Clean all output archives and files with statistics\n    cleanall             Clean everything but Conda, Snakemake, and input files\n    cleanallall          Clean completely everything\n###############\n## Reporting ##\n###############\n    viewconf             View configuration without comments\n    reports              Create html report\n####################\n## For developers ##\n####################\n    test                 Run the workflow on test data (P1)\n    bigtest              Run the workflow on test data (P1, P2, P3)\n    format               Reformat all source code\n    checkformat          Check source code format\n```\n\n*Note:* `make format` and `make checkformat` require\n[YAPF](https://github.com/google/yapf) and\n[Snakefmt](https://github.com/snakemake/snakefmt), which can be installed by\n`conda install -c conda-forge -bioconda yapf snakefmt`.\n\n\n\n### 4e. Running on a cluster\n\nCluster-related parameters for Snakemake can be added via the `SMK_CLUSTER_ARGS` Make variable.\n\nExample:\n```bash\nmake SMK_CLUSTER_ARGS=\"--profile my_snakemake_cluster_profile\"\n```\n\n\n### 4f. Troubleshooting\n\nTests can be run by `make test` (just Protocol 1) or `make bigtest` (all the protocols).\n\n\n## 5. Citation\n\n\u003e K. Brinda, L. Lima, S. Pignotti, N. Quinones-Olvera, K. Salikhov, R. Chikhi, G. Kucherov, Z. Iqbal, and M. Baym. **[Efficient and Robust Search of Microbial Genomes via Phylogenetic Compression](https://doi.org/10.1101/2023.04.15.536996).** *bioRxiv* 2023.04.15.536996, 2023. https://doi.org/10.1101/2023.04.15.536996\n\n```bibtex\n@article {PhylogeneticCompression,\n   author  = {Karel B{\\v r}inda and Leandro Lima and Simone Pignotti\n               and Natalia Quinones-Olvera and Kamil Salikhov and Rayan Chikhi\n               and Gregory Kucherov and Zamin Iqbal and Michael Baym},\n   title   = {Efficient and Robust Search of Microbial Genomes via Phylogenetic Compression},\n   journal = {bioRxiv},\n   elocation-id = {2023.04.15.536996},\n   year    = {2023},\n   doi     = {10.1101/2023.04.15.536996},\n   url     = {https://www.biorxiv.org/content/early/2023/04/16/2023.04.15.536996}\n}\n```\n\n\n## 6. Issues\n\nPlease use [Github issues](https://github.com/karel-brinda/miniphy/issues).\n\n\n\n## 7. Changelog\n\nSee [Releases](https://github.com/karel-brinda/miniphy/releases).\n\n\n\n## 8. License\n\n[MIT](https://github.com/karel-brinda/miniphy/blob/master/LICENSE)\n\n\n\n## 9. Contacts\n\n* [Karel Brinda](https://brinda.eu) \\\u003ckarel.brinda@inria.fr\\\u003e\n* [Leandro Lima](https://github.com/leoisl) \\\u003cleandro@ebi.ac.uk\\\u003e\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fkarel-brinda%2Fminiphy","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fkarel-brinda%2Fminiphy","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fkarel-brinda%2Fminiphy/lists"}