https://github.com/ratschlab/metagraph
Scalable annotated de Bruijn graphs for DNA indexing, alignment, and assembly
https://github.com/ratschlab/metagraph
alignment assembly dna graph metagenomics search
Last synced: 29 days ago
JSON representation
Scalable annotated de Bruijn graphs for DNA indexing, alignment, and assembly
- Host: GitHub
- URL: https://github.com/ratschlab/metagraph
- Owner: ratschlab
- License: gpl-3.0
- Created: 2016-08-10T15:13:50.000Z (over 9 years ago)
- Default Branch: master
- Last Pushed: 2026-01-12T12:08:33.000Z (about 1 month ago)
- Last Synced: 2026-01-12T19:35:42.562Z (about 1 month ago)
- Topics: alignment, assembly, dna, graph, metagenomics, search
- Language: C++
- Homepage: http://metagraph.ethz.ch
- Size: 75.1 MB
- Stars: 215
- Watchers: 16
- Forks: 26
- Open Issues: 22
-
Metadata Files:
- Readme: README.md
- License: LICENSE
- Authors: AUTHORS
- Copyright: COPYRIGHT
Awesome Lists containing this project
README
# Metagenome Graph Project
[](https://github.com/ratschlab/metagraph/releases)
[](https://bioconda.github.io/recipes/metagraph/README.html)
[](#conda)
[](#docker)
[](#install-from-sources)
[](https://metagraph.ethz.ch/static/docs/index.html)
MetaGraph is a tool for scalable construction of annotated genome graphs and sequence-to-graph alignment.
The default index representations in MetaGraph are extremely scalable and support building graphs with trillions of nodes and millions of annotation labels.
At the same time, the provided workflows and their careful implementation, combined with low-level optimizations of the core data structures, enable exceptional query and alignment performance.
#### Main features:
* Large-scale indexing of sequences
* [Python API](https://metagraph.ethz.ch/static/docs/api.html) for querying in the server mode
* Encoding [**k-mer counts**](https://metagraph.ethz.ch/static/docs/quick_start.html#index-k-mer-counts) (e.g., expression values) and [**k-mer coordinates**](https://metagraph.ethz.ch/static/docs/quick_start.html#index-k-mer-coordinates) in source sequences (e.g., for lossless encoding of genomes)
* **Sequence alignment** against very large annotated graphs (sub-k seeding allows using arbitrarily short seeds)
* Scalable cleaning of very large de Bruijn graphs (to remove sequencing errors)
* Support for custom alphabets (e.g., {A,C,G,T,N} or amino acids)
* Algorithms for [differential assembly](https://metagraph.ethz.ch/static/docs/sequence_assembly.html#differential-assembly)
#### Design choices in MetaGraph:
* Use of succinct data structures and efficient representation schemes for extremely high scalability
* Algorithmic choices that work efficiently with succinct data structures (e.g., always prefer batched operations)
* Modular support of different graph and annotation representations
* Use of generic and extensible interfaces to support adding custom index representations / algorithms with little code overhead.
## Documentation
Online documentation is available at https://metagraph.ethz.ch/static/docs/index.html. Offline sources are [here](metagraph/docs/source).
## Install
### Conda
Install the [latest release](https://github.com/ratschlab/metagraph/releases/latest) on Linux or Mac OS X with Anaconda:
```
conda install -c bioconda -c conda-forge metagraph
```
### Docker
If docker is available on the system, immediately get started with
```
docker pull ghcr.io/ratschlab/metagraph:master
docker run -v ${HOME}:/mnt ghcr.io/ratschlab/metagraph:master \
metagraph build -v -k 10 -o /mnt/transcripts_1000 /mnt/transcripts_1000.fa
```
and replace `${HOME}` with a directory on the host system to map it under `/mnt` in the container.
By default, it executes the binary compiled for the `DNA` alphabet {A,C,G,T}.
To run the binary compiled for the `DNA5` or `Protein` alphabet, just replace `metagraph` with `metagraph_DNA5` or `metagraph_Protein`, respectively, e.g.:
```
docker run -v ${HOME}:/mnt ghcr.io/ratschlab/metagraph:master \
metagraph_Protein build -v -k 10 -o /mnt/graph /mnt/protein.fa
```
One can see that running MetaGraph with docker is very easy. Also, the following command (or similar) may be handy to see what directory is mounted in the container:
```
docker run -v ${HOME}:/mnt ghcr.io/ratschlab/metagraph:master ls /mnt
```
For more complex workflows, consider running docker in the interactive mode:
```
$ docker run -it --entrypoint /bin/bash -v ${HOME}:/mnt ghcr.io/ratschlab/metagraph:master
root@5c42291cc9cf:/# ls /mnt/
root@5c42291cc9cf:/# metagraph --version
```
All different versions of the container image are listed [here](https://github.com/ratschlab/metagraph/pkgs/container/metagraph).
### Install From Sources
To compile from source (e.g., for builds with custom alphabet or other configurations), see [documentation online](https://metagraph.ethz.ch/static/docs/installation.html#install-from-source).
## Typical workflow
1. Build de Bruijn graph from Fasta files, FastQ files, or [KMC k-mer counters](https://github.com/refresh-bio/KMC/):\
`./metagraph build`
2. Annotate graph using the column compressed annotation:\
`./metagraph annotate`
3. Transform the built annotation to a different annotation scheme:\
`./metagraph transform_anno`
4. Query annotated graph\
`./metagraph query`
### Example
```
DATA="../tests/data/transcripts_1000.fa"
./metagraph build -k 12 -o transcripts_1000 $DATA
./metagraph annotate -i transcripts_1000.dbg --anno-filename -o transcripts_1000 $DATA
./metagraph query -i transcripts_1000.dbg -a transcripts_1000.column.annodbg $DATA
./metagraph stats -a transcripts_1000.column.annodbg transcripts_1000.dbg
```
### Print usage
`./metagraph`
### Build graph
* #### Simple build
```bash
./metagraph build -v --parallel 30 -k 20 --mem-cap-gb 10 \
-o /graph /*.fasta.gz \
2>&1 | tee /log.txt
```
* #### Build with disk swap (use to limit the RAM usage)
```bash
./metagraph build -v --parallel 30 -k 20 --mem-cap-gb 10 --disk-swap \
-o /graph /*.fasta.gz \
2>&1 | tee /log.txt
```
#### Build from k-mers filtered with KMC
```bash
K=20
./KMC/kmc -ci5 -t4 -k$K -m5 -fm .fasta.gz .cutoff_5 ./KMC
./metagraph build -v -p 4 -k $K --mem-cap-gb 10 -o graph .cutoff_5.kmc_pre
```
### Annotate graph
```bash
./metagraph annotate -v --anno-type row --fasta-anno \
-i primates.dbg \
-o primates \
~/fasta_zurich/refs_chimpanzee_primates.fa
```
### Convert annotation to Multi-BRWT
1) Cluster columns
```bash
./metagraph transform_anno -v --linkage --greedy \
-o linkage.txt \
--subsample R \
-p NCORES \
primates.column.annodbg
```
Requires `N*R/8 + 6*N^2` bytes of RAM, where `N` is the number of columns and `R` is the number of rows subsampled.
2) Construct Multi-BRWT
```bash
./metagraph transform_anno -v -p NCORES --anno-type brwt \
--linkage-file linkage.txt \
-o primates \
--parallel-nodes V \
-p NCORES \
primates.column.annodbg
```
Requires `M*V/8 + Size(BRWT)` bytes of RAM, where `M` is the number of rows in the annotation and `V` is the number of nodes merged concurrently.
### Query graph
```bash
./metagraph query -v -i /graph.dbg \
-a /annotation.column.annodbg \
--min-kmers-fraction-label 0.8 --labels-delimiter ", " \
query_seq.fa
```
### Align to graph
```bash
./metagraph align -v -i /graph.dbg query_seq.fa
```
### Assemble sequences
```bash
./metagraph assemble -v /graph.dbg \
-o assembled.fa \
--unitigs
```
### Assemble differential sequences
```bash
./metagraph assemble -v /graph.dbg \
--unitigs \
-a /annotation.column.annodbg \
--diff-assembly-rules diff_assembly_rules.json \
-o diff_assembled.fa
```
See [`metagraph/tests/data/example.diff.json`](metagraph/tests/data/example.diff.json) and [`metagraph/tests/data/example_simple.diff.json`](metagraph/tests/data/example_simple.diff.json) for sample files.
### Get stats
Stats for graph
```bash
./metagraph stats graph.dbg
```
Stats for annotation
```bash
./metagraph stats -a annotation.column.annodbg
```
Stats for both
```bash
./metagraph stats -a annotation.column.annodbg graph.dbg
```
## Developer Notes
### Build a docker container
Simply run `docker build .`
### Makefile
The `Makefile` in the top level source directory can be used to build and test `metagraph` more conveniently. The following
arguments are supported:
* `env`: environment in which to compile/run (`""`: on the host, `docker`: in a docker container)
* `alphabet`: compile metagraph for a certain alphabet (e.g. `DNA` or `Protein`, default `DNA`)
* `additional_cmake_args`: additional arguments to pass to cmake.
Examples:
```
# compiles metagraph in a docker container for the `DNA` alphabet
make build-metagraph env=docker alphabet=DNA
```
### Update and create a new release
Creating a new version release is done in three steps:
1. Update package.json and set the version
2. Add a tag with that new version
3. Make a new release on github
## License
Metagraph is distributed under the GPLv3 License (see LICENSE).
Please find further information in the AUTHORS and COPYRIGHTS files.