{"id":25596870,"url":"https://github.com/nanoporetech/dtr-phage-pipeline","last_synced_at":"2025-04-13T02:33:17.935Z","repository":{"id":48813447,"uuid":"225891710","full_name":"nanoporetech/DTR-phage-pipeline","owner":"nanoporetech","description":null,"archived":false,"fork":false,"pushed_at":"2024-03-11T22:09:49.000Z","size":11215,"stargazers_count":16,"open_issues_count":3,"forks_count":6,"subscribers_count":25,"default_branch":"master","last_synced_at":"2025-04-06T08:02:14.164Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mpl-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/nanoporetech.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2019-12-04T14:54:59.000Z","updated_at":"2024-04-02T02:05:49.000Z","dependencies_parsed_at":"2025-02-21T12:45:08.540Z","dependency_job_id":null,"html_url":"https://github.com/nanoporetech/DTR-phage-pipeline","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/nanoporetech%2FDTR-phage-pipeline","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/nanoporetech%2FDTR-phage-pipeline/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/nanoporetech%2FDTR-phage-pipeline/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/nanoporetech%2FDTR-phage-pipeline/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/nanoporetech","download_url":"https://codeload.github.com/nanoporetech/DTR-phage-pipeline/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248657795,"owners_count":21140842,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2025-02-21T12:34:59.443Z","updated_at":"2025-04-13T02:33:17.889Z","avatar_url":"https://github.com/nanoporetech.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"![ONT_logo](/ONT_logo.png)\n\n-----------------------------\n\nDTR phage pipeline\n==================\nThe DTR phage pipeline is a Snakemake pipeline for obtaining polished, full-length  dsDNA bacteriophage genomes with direct terminal repeats (DTRs) from enviromental samples. \n\nGetting started\n===============\n## Dependencies\nThe pipeline relies heavily  on `conda` to supply [Snakemake](https://snakemake.readthedocs.io/en/stable/index.html) and the software packages required for various steps in the analysis. If `conda` is not currently installed on your system (highly encouraged!), please first follow the installation instructions for installing [miniconda](https://docs.conda.io/en/latest/miniconda.html).\n\n[Snakemake](https://snakemake.readthedocs.io/en/stable/index.html) is a workflow management system that uses Python-based language that is ideally-suited for providing reproducible and scalable bioinformatic pipelines. If you are unfamiliar with Snakemake, this [tutorial](https://snakemake.readthedocs.io/en/stable/tutorial/basics.html) is a good place to start. \n\n## Installation\nFirst clone the repositorty to the desired location and enter the cloned directory:\n```\ngit clone https://github.com/nanoporetech/DTR-phage-pipeline.git\ncd DTR-phage-pipeline\n```\nNext, create a `conda` environment where you will run `Snakemake`:\n```\nconda env create -f environment.yml\n```\nThat's all. Ideally, `conda` shoud take care of all the remaining dependencies when each specific Snakemake step (described below) is executed.\n\n### If conda is slow\n\nFor speed tips related to conda, see *mamba* section below.\n\n### Testing\n\nTo verify that everything is set up correctly, simply run the workflow without\nmodifying config.yaml:\n\n```\nsnakemake --use-conda -p -r -j 10\n```\n\n## Pipeline configuration\n\nThe pipeline determines the input files, output path, and the various software parameters from the file `config.yml`, which contains key:value pairs that control how the pipeline runs. \n1. Attention must be paid to the first section of `config.yml` ('Editing mandatory'), as these might change from run to run.\n    - `sample`: name of the sample (e.g. 25m)\n    - `stype`: sample type (e.g. 1D or filtered_02um, etc)\n    - `version`: run version (for changing run parameters)\n    - `input_fasta`: FASTA file containing nanopore reads to be analyzed\n    - `input_summary`: Sequencing summary file associated with the FASTA file\n    - `output_root`: Base directory where pipeline results will be written\n    - `max_threads`: Maximum number of threads to allocate for each rule\n    - `MEDAKA:model`: Medaka model to use for polishing the draft genome (e.g. `r941_min_high_g303`). See Medaka [documentation](https://github.com/nanoporetech/medaka#models) for details on selecting the proper model for you basecalled reads.\n    - `MEDAKA:threads`: Number of threads to use for each Medaka genome polishing. Should be \u003c\u003c `max_threads` so that multiple genomes can be polished in parallel.\n    - `KAIJU:db`: Path to Kaiju database to use for taxonomic annotation. Folder should contain the `*.fmi`, `names.dmp`, and `nodes.dmp` files. These be downloaded from the [Kaiju web server](http://kaiju.binf.ku.dk/server). nr+euk is recommended.\n    - `KAIJU:switch`: Parameters to use for Kaiju annotation. These parameters should suffice in most cases.\n    - `SKIP_BINS:\u003csample\u003e:\u003cstype\u003e:\u003cversion\u003e`: List of k-mer bins to skip for analysis. HDBSCAN assigns all unbinned reads to bin_id = -1, which should always be skipped. Occasionally downstream errors in the pipeline indicate that additional bad k-mer bins should also be skipped. \n\n2. The second section of `config.yml` ('Editing optional') contains software parameters that can be adjusted, but should perform sufficiently well in most cases using the default values.\n    - 'BIN_AVA_ALGN:CLUST:max_recursion': If you get an error in this step from\n      reaching Python's max recursion depth, you can try increasing the limit\n      here if you have the system resources available.\n## Pipeline execution\nThe full pipeline, starting from raw reads and ending with nanopore-polished phage genomes, can be executed in one step. \n\n```\nsnakemake --use-conda -p -j \u003cnproc\u003e -r \n```\nWhere `\u003cnproc\u003e` is the maximum number of cores for all tasks in the pipeline. The output files for this workflow are placed in a path according to the variables set in the `config.yml` file. In this description, `\u003crun_output_dir\u003e` will refer to the path consisting of `\u003coutput_root\u003e/\u003csample\u003e/\u003cstype\u003e/\u003cversion\u003e` as defined in the `config.yml` file.\n\n### all_kmer_count_and_bin\nThe first step provides summary plots for the input reads, identifies reads containing DTR sequences, creates k-mer count vectors, embeds these vectors into 2-d via [UMAP](https://github.com/lmcinnes/umap), and calls bins in the embedding via [HDBSCAN](https://github.com/lmcinnes/HDBSCAN). \n\nThe outputs from this step will fall into three directories:\n\n * `\u003crun_output_dir\u003e/reads_summary`\n    * `reads.summary.stats.png`: Read length and qscore distributions for all input reads\n * `\u003crun_output_dir\u003e/dtr_reads`\n    * `output.dtr.fasta`: FASTA file of all DTR-containing reads\n    * `output.dtr.hist.png`: Read length distribution of all DTR-containing reads\n    * `output.dtr.stats.tsv`: Statistics for each DTR-containing read\n * `\u003crun_output_dir\u003e/kmer_binning`\n    * `bin_membership.tsv`: bin assignments for each DTR-containing read\n    * `kmer_comp.tsv`: 5-mer count vectors for each DTR-containing read\n    * `kmer_comp.umap.tsv`: x- and y-coordinates for each read in the 2-d embedding\n    * `kmer_comp.umap.*.png`: Variety of scatter plots of [UMAP](https://github.com/lmcinnes/umap) embedding of k-mer count vectors, annoted by features including GC-content, read length, and bin assigned by [HDBSCAN](https://github.com/lmcinnes/HDBSCAN)\n    * `kmer_comp.umap.bins.tsv`: Mean x- and y-coordinates for each bin assigned by HDBSCAN (for finding bins in 2-d embedding)\n    \n### all_kaiju\nThe next (optional) step annotates reads using Kaiju and is not strictly required for producing polished genomes. However, it can be informative for verifying the integrity of the k-mer bins and for other downstream analyses.\n\nSome of the output files for this step include:\n* `\u003crun_output_dir\u003e/kaiju`\n    * `results.html`: Krona dynamic plot of annotated taxonomic composition of DTR-containing reads\n    * `results.taxa.tsv`: Per-read annotation results\n* `\u003crun_output_dir\u003e/kmer_binning`\n    * `kmer_comp.umap.nr_euk.[0-6].png`: Scatter plots of UMAP embedding of k-mer count vectors, annotated by various levels of taxonomic annotation\n\n### all_populate_kmer_bins\nThe next step simply populates subdirectories with the binned reads as assigned by HDBSCAN.\n\nThese subdirectories are located in the `kmer_binning` directory:\n* `\u003crun_output_dir\u003e/kmer_binning/bins/\u003cbin_id\u003e`\n* Each bin subdirectory contains a list of binned read names (`read_list.txt`) and an associated FASTA file of reads (`\u003cbin_id\u003e.reads.fa`)\n\n### all_alignment_clusters\nNext, each k-mer bin is refined by all-vs-all aligning all reads within a bin. The resulting alignment scores are clustered hierarchically and refined alignment clusters are called from the clustering.\n\nThe bin refinement results for each k-mer bin are also placed in `kmer_binning` directory:\n* `\u003crun_output_dir\u003e/kmer_binning/refine_bins/align_clusters/\u003cbin_id\u003e`\n* Each bin refinement procedure generates an alignment (`\u003cbin_id\u003e.ava.paf`), clustering heatmap (`\u003cbin_id\u003e.clust.heatmap.png`), and information on alignment cluster assignments (`\u003cbin_id\u003e.clust.info.tsv`)\n\n### all_polish_and_annotate\nNext, a single read is selected from each valid alignment cluster and is polished by the remaining reads in the alignment cluster. Polishing is first done using multiple rounds of [Racon](https://github.com/isovic/racon) (3x by default), then is finished using a single round of [Medaka](https://github.com/nanoporetech/medaka) polishing.\n\nThis step produces polished output in the following directories:\n* `\u003crun_output_dir\u003e/kmer_binning/refine_bins/align_cluster_reads/\u003cclust_id\u003e` simply contains the reads corresponding to each alignment cluster\n* `\u003crun_output_dir\u003e/kmer_binning/refine_bins/align_cluster_polishing/racon/\u003cclust_id\u003e` contains the Racon polishing output for each alignment cluster\n* `\u003crun_output_dir\u003e/kmer_binning/refine_bins/align_cluster_polishing/medaka/\u003cclust_id\u003e` contains the Medaka polishing output for each alignment cluster\n    * The critical output file in each of these `medaka/\u003cclust_id\u003e` folders is the `\u003cclust_id\u003e.ref_read.medaka.fasta` file containing the __Medaka-polished genome produced from this alignment cluster__. Subsequent Snakemake rules analyze, aggregate, and deduplicate these polished genomes.\n    * `\u003cclust_id\u003e.ref_read.medaka.prodigal.cds.*` files describe the coding sequence annotations from [Prodigal](https://github.com/hyattpd/Prodigal) for this Medaka-polished genome.\n    * `\u003cclust_id\u003e.ref_read.strands.*` files describe the strand abundance for reads in each alignment cluster. Clusters containing \u003e80% reads from a single strand should be treated with suspicion.\n    * `\u003cclust_id\u003e.ref_read.dtr.aligns.*` files describe the results of aligning the DTR sequence from each corresponding k-mer bin to the polished genome from each alignment cluster. If the DTR sequences all align to the same reference positions, the DTR is fixed. However, if they align all over the reference genome, this suggests that a headful DNA packaging mechanism was used. \n\n### all_combine_dedup_summarize\nNext, we finish up the genome discovery portion of the pipeline by running a series of aggregations and evaluations of the final polished genome sequences. \n\nAll output from this step is written to a single directory:\n* `\u003crun_output_dir\u003e/kmer_binning/refine_bins/align_cluster_polishing`\n    * `polished.seqs.fasta`: The combined set of genomes from each alignment cluster\n    * `polished.seqs.unique.fasta`: Same as above but only containing unique sequences after the deduplication step\n    * `polished.stats.tsv`: Various statistics for each polished genome, including length, GC content, DTR details, cluster strand abundance, CDS annotation statistics, circular permutation status, and many others\n    * `polished.stats.unique.tsv`: Same as above but only containing unique sequences after the deduplication step\n    * `polished.unique.cds.summary.all.png`: Summary plots of summary statistics for the coding sequences (CDS) annotated by Prodigal for each unique polished genome\n    * `polished.unique.cds.summary.dtr_npol10.png`: Same as above but only including polished genomes with a confirmed DTR and at least 10 reads used for polishing\n\n### all_linear_concatemer_reads\nFinally, we run one final step to query the sequencing dataset for linear concatemer reads that could represent interesting mobile elements in the environmental sample.\n\nAll output from this step is written to a single directory:\n* `\u003crun_output_dir\u003e/concatemers`\n    * `concats.fasta`: All identified concatemeric reads found in the input reads\n    * `concats.tsv`: Statistic for each concatemeric read found, including readname, length, repeat size, and repeat copy count\n    * `concats.contours.png`: Scatter plot with density contours showing the relationship between the observed repeat length and copy count in all identified concatemeric reads.\n\n## Licence and Copyright\n\n(c) 2019 Oxford Nanopore Technologies Ltd.\n\nThis Source Code Form is subject to the terms of the Mozilla Public\nLicense, v. 2.0. If a copy of the MPL was not distributed with this\nfile, You can obtain one at http://mozilla.org/MPL/2.0/.\n\n## FAQs and tips\n\n### *mamba*\nThe conda package manager is a powerful tool for replicating bioinformatics,\nbut it can be slow for large envirnonments. If you find installing any of the \nenvironments is taking too long, try using\n[mamba](https://github.com/TheSnakePit/mamba).\n\nMamba is a fast drop-in replacement for conda. It is still in development, but\nso far has proven to e useful. Simply install in your base envirnoment:\n\n```\nconda install mamba\n```\n\nand then replace `conda` with `mamba` in commands that install packages. EG:\n```\nmamba env create -f environment.yml\n```\n\nTo get snakemake to use mamba, pass `--conda-frontend mamba` to your command:\n```\nsnakemake --use-conda --conda-frontend mamba -p -r -j \u003cnprocs\u003e\n```\n\n### The \"TBD\" Error\nIf you re-run the workflow from certain points, because there are multiple\n\"checkpoints\" where the number of output files is unknown until runtime, \nsnakemake will sometimes get confused about a rule's input. The exact error\nwill depend upon where in the workflow it occurs.\n\nOne know case gives this error:\n```\n[Errno 2] No such file or directory: 't'\n```\n\nbut there may be others. They all share the odditiy that the rule's input is \nmarked as TBD in the snakemake output and log. For example:\n\n```\nrule aggregate_prodigal_statistics:\n    input: \u003cTBD\u003e\n    output:\n        test/test-output/25m/1D/v2/kmer_binning/refine_bins/align_cluster_polishing/polished.cds.summary.tsv\n    jobid: 21\n```\n\nThe workaround is to ask snakemake explicitly to create the output file of the\nfailed rule. This is done by jsut appending the full (with path) file name to\nthe snakemake command. Simply copy the file path from the output reported after\n\"TBD\". For the example error above, you would run something\nlike this:\n\n```\nsnakemake --use-conda --conda-frontend mamba -p -r -j 10\ntest/test-output/25m/1D/v2/kmer_binning/refine_bins/align_cluster_polishing/polished.cds.summary.tsv\n```\n\nThe single step should run OK, then you can resume the rest of the workflow with the `all` rule.\n\n\n## References and Supporting Information\n\nThis pipeline is described in Genome Research:\n\nBeaulaurier J., Luo E., Eppley J., Den Uyl P., Dai X., Turner D.J., Pendelton M., Juul S., Harrington E., DeLong E.F. Assembly-free single-molecule sequencing recovers complete virus genomes from natural microbial communities. Genome Research (2020). [doi:10.1101/gr.251686.119](https://genome.cshlp.org/content/early/2020/02/19/gr.251686.119.abstract)\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fnanoporetech%2Fdtr-phage-pipeline","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fnanoporetech%2Fdtr-phage-pipeline","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fnanoporetech%2Fdtr-phage-pipeline/lists"}