{"id":15060293,"url":"https://github.com/ruanjue/smartdenovo","last_synced_at":"2025-04-10T05:50:58.885Z","repository":{"id":45156767,"uuid":"47866716","full_name":"ruanjue/smartdenovo","owner":"ruanjue","description":"Ultra-fast de novo assembler using long noisy reads","archived":false,"fork":false,"pushed_at":"2021-02-24T05:00:23.000Z","size":347,"stargazers_count":134,"open_issues_count":19,"forks_count":30,"subscribers_count":14,"default_branch":"master","last_synced_at":"2025-03-24T07:04:20.244Z","etag":null,"topics":["assembler","pacbio"],"latest_commit_sha":null,"homepage":"","language":"C","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"gpl-3.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/ruanjue.png","metadata":{"files":{"readme":"README-tools.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE.txt","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2015-12-12T06:02:18.000Z","updated_at":"2025-03-20T02:44:04.000Z","dependencies_parsed_at":"2022-07-13T16:47:45.486Z","dependency_job_id":null,"html_url":"https://github.com/ruanjue/smartdenovo","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ruanjue%2Fsmartdenovo","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ruanjue%2Fsmartdenovo/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ruanjue%2Fsmartdenovo/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ruanjue%2Fsmartdenovo/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/ruanjue","download_url":"https://codeload.github.com/ruanjue/smartdenovo/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248166926,"owners_count":21058480,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["assembler","pacbio"],"created_at":"2024-09-24T22:56:23.302Z","updated_at":"2025-04-10T05:50:58.859Z","avatar_url":"https://github.com/ruanjue.png","language":"C","funding_links":[],"categories":[],"sub_categories":[],"readme":"wtzmo\n-----\n\nThe main overlaper of SMARTdenovo. Seed index -\u003e Seed map -\u003e Alignment.\n\nwtzmo uses homopolymer compressed kmer as seeds. To trun off homopolymer\ncompression, see wtzmo -H 0. A seed contains [read_id, strand] in 4 bytes.\nPlease note that, wtzmo only keep the smaller between one kmer and its reverse\ncomplementary, if equals, skips. Seeds are sorted by multiple value composed of\nkmer, read_id and strand. A hashtable records {kmer_value =\u003e\nregion_in_seed_array}. In implement, I scan the reads twice, first count kmer\nin hastable and create a empty but reffered array, then fill the data. In the\nconcern of memory, wtzmo only index kmers tailed with C/A, halve the memory.\nwtzmo -k 16, means the compressed kmer size is set to 16. wtzmo -K 500, the\nupper K, means discarding high frequency (\u003e500) kmer.\n\nWhen query one read, wtzmo split the query into kmers. Then query each kmer\nagainst the hashtable and seed array to find which [read_id,strand] cover\nenough region on query sequence. Because seeds in a kmer are sorted by\nread_id+strand, a process like merge-sort is used to calculate the covered size\nfor each existed read_id+strand. Here, wtzmo -A 500, means a heap records the\ntop 500 best covered reads. wtzmo -d 300 means a candidate must cover at least\n300 bp uncompressed region on query. Suppose seed1 cover 1-21, seed2 cover\n3-22, the sum is 1-22.\n\nAs we have candidates for one query, we can just use banded-SW to align them to\nget the final alignments. However, it will spend too much time.\n\nTo speed it up, wtzmo build a similar seed index on the query sequence,\nexcepting the kmer_size is smaller, let call it zmer, wtzmo -z 10 is about\nthis. In this zmer-index, each zmer has its offset and strand. Filter high\nfrequency zmers by wtzmo -Z 100. Then match one candidate's zmers on\nzmer-index. For now, we got pairs of matched seeds between candidate and query,\nthe next job is search the co-linear matched zmers.\n\nwtzmo search the synteny within one window size specified by wtzmo -y 800. The\nreason for why not search the synteny for whole overlapped region, is high rate\nof INDELs make the distance between two nearby zmer high variable. wtzmo -R 200\nhas the similar meaning with wtzmo -d, but is it used in a window of 800bp.\n\nWhen wtzmo matched lots of pairs of zmer-windows, it search coliner pairs in DP\nmanner. The final synteny of zmer-windows is called seed map. If a seed map\ncovers too small part of the query sequence, it will be discarded. It is\nspecified by wtzmo -r 300. Here, 300 means the solid zmer regions inside a\nwindow, not the whole window. A candidate success in this step, will go to the\nalignment procedure.\n\nThe alignment is splited into four parts.\n\n1. matched zmer. just recover the compression, and add gaps.\n\n2. within zmer window. Global alignment between two adject matched zmer.\n\n3. Between two adject windows. Use banded global SW algorithm, the band size\n   iteratively increase from wtzmo -w 50 to wtzmo -W 3200, until the alignment\n   score is positive.\n\n4. Extending of two ends. Local alignment. Its band width is specified by wtzmo\n   -e 800.\n\nwtzmo -T -50 means if the best score is bigger than the score at ending by less\nthan 50, will take the global alignment as result.\n\nThe order of alignment for candidates is sorted by their zmers covered size on\nquery. When wtzmo success to perform wtzmo -B 100 alignments, it will give up\nto try more candidates. Use wtzmo -m 0.6 and wtmzo -s 200 to tell which\nalignment is successful.\n\nThe order of alignment for candidates is also affected by estimation of\nrepetitive region. wtzmo calculates the zmer-window coverage on query, and\ngives every position a weight, which is negatively correlated with coverage.\nThis weighting strategy avoids lots of useless computation. For example, a\nquery has a repeat in the middle, will recruit floods of false candidates that\ncontain the similar/same repeat. In this case, wtzmo will decrease the weight\nof those flase candidates, and preffer to perform alignment on other\ncandidates. wtzmo -q 100 will set the weight of a position to zero when its\ncoverage reach to 100.\n\nTo get accurate CIGAR of alignment, add wtzmo -n. Refinement will be performed\nbased on the original alignment with flexible band width for each position.\nThat is wtzmo use global alignment on the alignment but with danymatic band\nwidth. The basic band width is the same with wtzmo -w 50. When come to a gap of\nsize N, the bandwidth around it will increase N - d, d is the distance from\ngap, the bandwidth increase by N inside the gap. wtzmo will trim the large\nbandwidth into reasonable value based on DP alignment process.\n\nTo examine what happens during the alignment, use wtzmo -v, or -vv, -vvv,\n-vvvv. The more -v will give more detailed information, catch them on the\nSTDERR.\n\nIf you have the read mask file, use wtzmo -b wt.cyc.clp to trim the reads\ninside wtzmo. It will save the disk storage. wtzmo -J 1000 will filter all\nreads have length less than 1000 bp. If you don't want to alignment a pair of\nreads, add them in a file like 'already_aligned.pairs' and use wtzmo -L\nalready_aligned.pairs.\n\nwtzmo is designed as a overlapper for de novo assembly. In the context of\nString Graph, if one read is contained by another, it won't contribute in\ngraph. By default, when wtzmo find a candidate contained by query read, it will\nmark the candidate as contained, and refuse to use it as query. Please note\nthat, the order of query sequence is sorted by read length DSC. But this\ncontained read can be used as candiate in other query. That is, we will know\nwhich reads contain this shorter read, which will be informative in consensus\ncalling.\n\nFor huge genome, wtzmo can be run in low memory by `wtzmo -G \u003cN\u003e`. N is the\nnumber of parts of kmer-index. Most of memory is spent on kmer-index. Thus,\nsplit the kmer-index into N parts will reduce the memory into about 1/N. wtzmo\nwill record the candidates in additional memory and return to normal flow after\nquery all kmer-index parts.\n\nTo invoke wtzmo parallelly on multiple nodes, use `-P \u003ctotal_nodes\u003e -p\n\u003cindex_of_node\u003e`, e.g. -P 60 -p 0 on the first node of 60 nodes. It will take\nthe same memory on each node as only on one node. Example:\n```sh\nwtzmo -t 32 -i wt.fa -o wt.zmo.ovl\n```\n\nOuput format: tab-delimited\n\n|Col|Description              |\n|--:|:------------------------|\n|1  |qry_name                 |\n|2  |qry_strand: +/-          |\n|3  |qry_length               |\n|4  |qry_beg: 0-based         |\n|5  |qry_end: exclusive       |\n|6  |sbj_name                 |\n|7  |sbj_strand               |\n|8  |sbj_length               |\n|9  |sbj_beg                  |\n|10 |sbj_end                  |\n|11 |score                    |\n|12 |idenity: 0.0 - 1.0       |\n|13 |n_mat: number of matches |\n|14 |n_mis: mismatch          |\n|15 |n_ins: insertion         |\n|16 |n_del: deletion          |\n|17 |cigar: CIGAR in SAM      |\n\nwtobt\n-----\n\nTrim reads base on overlaps. wtobt takes overlaps as input, trims high error\nending and chimeria. It try to retain max part of one read. First, it find a\nmax continous region that covered by other aligned reads. Then, it detect spurs\nwhich one read get partial alignment on it, and counting how many reads cross\nthe spurs (as m) and how many reads get partial alignments (as n). If m no less\nthan half of the average depth at the spur, reject a chimeria. If m is bigger\nthan half of m, reject. otherwise, a chimra is detected. wtobt will retain the\nlarger part.\n```sh\nwtobt -i wt.fa -j wt.zmo.ovl -o wt.zmo.obt -c 2\n```\nwtobt output a read mask file with lines like: read_name offset length.\n\nwtgbo\n-----\n\nSMARTdenovo cannot find all of pairwise alignments as nearly all of other\naligners. It may miss key overlaps on graph. To rescue overlaps that are\nimportant to assembly graph, wtgbo scans pairs of reads that might have\noverlaps in: a) two reads have overlaps with the same other read, b) two reads\nare connected by at most N steps, N is defined as 2. wtgbo try to align those\npairs in the same manner as wtzmo's candidates and query.\n\nAfter generate new valid overlaps, wtgbo build a newer best overlap graph, and\ninfer new potential overlaps, until none new valid overlaps or max iterations.\n```sh\nwtgbo -t 32 -i wt.fa -j wt.zmo.ovl -o wt.zmo.gbo\n```\nwtgbo combine the algorithm of wtzmo and wtlay, its parameters like that of\nwtzmo and wtlay. Its output is the same with wtzmo.\n\nwtclp\n-----\n\nThe goal of wtclp is to maximize the total length of valid overlaps by trimming\nreads or discarding reads (wtclp -F).\n\nwtclp puts one read as reference, and tiles all reads having overlaps regards\nof spurs. A function call_legal_overlaps_wtclp is used to calculate the length\nof valid overlaps, all operations aim to maxmize the result of this function.\n\nIf -F is not specified, wtclp first clip high error ends by calculating\ncoverage. The threshold is set by wtclp -c 2. In discarding mode (wtclp -F), a\nread is discarded or kept as whole.\n\nwtclp detects all structure errors as chimera. Two algorithms are used: a)\ndepth depended, like wtobt; b) graph based. For (b, if one read connecting two\nsubgraphs by itself, it is thought to be structure error. wtclp checks whether\nthere is an alternative path formed by valid overlaps of tiled reads.\n\nIn NGS de novo assembly, people often plot kmer-distribution to estimate the\ngenome size and other features. TGS has very high error rate, it is not\npossible to get genome size as usual. wtclp and wtobt use a new strategy to\nestimate the genome size. As we get the total length of reads, if we know the\naverage depth, we can compute the genome size. The average depth is figured out\nin the similar way as finding peak in NGS kmer-plot.\n\nwtclp has the same output as wtobt. The reason of having both wtobt and wtclp\nis to save the disk storage. Will discuss it in following section.\n\nwtext\n-----\n\nwtext is used in one kind of SMARTdenovo pipelines. It takes overlaps from\nwtzmo and reads mask file from wtobt as input, and curates the overlaps. After\ntrimming reads, some alignments may be wrong, due to coordinate. wtext will try\nto extend local alignment to a global alignment by wtext -T -100, like -T in\nwtzmo.\n\nwtext requires cigar of alignment, so that we need to keep cigars in the\nalignment files. Cigars take most of the disk usage of alignment files, and\nbecome hard in very big dataset. To save the disk usage, I introduced wtclp -F\nmode. It either discards whole read or keeps it as orginal, and needn't to\ninvoke wtext to curate alignments.\n\nwtlay\n-----\n\nRoughly, wtlay implements BOG to generate layout of reads. It may take another\nsize of this document to discribe it. Just list some parameters.\n\n * `wtlay -w 100`. If an overlap is not end-to-end, but leaving N bp unaligned,\n   and the N is no greater than 100, wtlay trust it as true overlap.\n\n * `wtlay -c 1`. Given a edge/overlap E between two node/reads A and B, the\n   coverage of E is computed as how many edges draw out the same path as E. If\n   the coverage of E is less than 1, will mask E as unreliable.\n\n * `wtlay -r 0.95`. which overlap is the best overlap? As high INDEL rate,\n   wtlay doesn't trust the longest one on faith.  wtlay says the best overlap\n   should have alignment score no less than 0.95 * \u003cmax score of its same\n   strand\u003e.\n\n * `wtlay -q 0.4`. I feel uneasy about it. wtlay doesn't merge bubbles, but cut\n   up one path instead, which will leave islands.  To avoid to output them,\n   wtlay filtered an unitig haing more than 40% of its length aligned on\n   another untig.  It may bring the assembly size down.\n\n * `wtlay -Q gCwgBgRURg`. Having funs!\n\nThe prefix of output files is specified by `wtlay -o \u003cwt.lay\u003e`. \n\nwt.lay.utg is a fasta file, contains uncorrected sequences of unitigs.\n\nwt.lay is a layout file, contains all the information needed by consensus caller.\n```\n\u003eutg1\nY/N\tread_name\tstrand\toffset\tlength\tsequence\n...\n```\nwhere:\n\n * Y/N: whether this read is used to build backbone or not. Reads in backbone\n   should be not contained by others.\n\n * strand: for human read\n\n * offset: the offset to previous Y-starting read.\n\n * length: for huamn read\n\n * sequence: trimmed if need, reversed if need, the direct-in-use sequence used\n   in calling consensus without any addtional operation.\n\n`wt.lay.\u003cN\u003e.dot` is the graphviz source file.\n\nwtcns\n-----\n\nwtcns implemented the DAGCon algorithm described in HGAP paper. Alignment\nalgorithm is integrated into wtcns, thus doesn't need other alignment tool.\nwtcns takes the layout file as input, and outputs consensus sequences of fasta\nformat.\n\nThe consensus sequences from wtcns are much accurater than PacBio reads, may\nreach to 99.7%, but still cannot fit the need of genome assembly. If you have\nother tools (e.g. Quiver) to improve the consensus sequences, please reduce the\nnumber of iterations to save time, by wtcns -n 1 or -n 2.\n\nwtcorr\n------\n\nCorrecting long noisy reads on DBG from short accurate reads using k-mer\nmoving. The DBG contains a smaller kmer (e.g. k=25) for k-mer moving, and a\nbigger kmer (e.g. k=41) to verify the path of kmer moving. The bigger kmers are\nstored in counting-bloom-filter to save the memory.\n\nBuilding the DBG:\n```\nwtcorr -t 32 -i read_1.fa -i read_2.fa -k 25 -K 41 -w sr.k25K41.dbg\n```\nand correcting:\n```\nwtcorr -t 32 -r sr.k25K41.dbg -c 5 -0 3 -1 2 -L -o corrected.fa raw.fa\n```\n\nwtcorr is very slow in large genome. It doesn't trust any matched kmer, and try\nto start graph alignment from any possible matched kmer. Given a matched kmer\nbetween PacBio read and DBG, it performs forward and backword kmer moving, and\nmerges the alignments. By default, it is local alignment mode. It will give\nmore than one corrected fragments. There may be more than hundreds of matched\nkmers, there would be many kmers are random-matched (depends on the genome size\nand PacBio errors). So that, when come to a kmer contained in prevois local\nalignment, it may loss the true. I have worked out a algorithm which can start\nalignments from all matched kmers, and use dynamic programming to avoid repeat\ncalculation. However, it is not implemented by now.\n\nWhether error correction helps or not? Correcting pacbio reads using short\nreads or itselves, MUST answer this question: Can we distinguish repetitive\nsequences simplely? Here, repetitive is defined by calculation in the\nbackground of high sequencing error, espically INDELs. For less repetitive\ngenomes, error correction will works well and simplify the process of assembly.\nHowever, TGS is expected to solve complicated genomes, which always have flood\nof repeats, will error correction still works well? In SMARTnodeovo, I choose\nto assemble un-corrected long reads.\n\nwtcyc\n-----\n\nAlign read against its reverse complementary sequence, to detect reads which\nmay miss its adpter. wtcyc generate a read mask file to discribe which part of\nreads can be used in further analysis. User needn't process the raw reads file,\ndownstream programs can recongize this mask file.\n```sh\nwtcyc -t 32 -i wt.fa -o wt.cyc.clp -a wt.cyc.info\n```\n\nwt.cyc.clp is the resulting mask file. wt.cyc.info is the alignments.\n\nwtcyc is obselete, because it may clip the real DNA palindrome (exists in large\ngenome), and cause breakpionts on genome. I classfy this kind of structure\nerror as chimeric, and solve it in wtclp.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fruanjue%2Fsmartdenovo","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fruanjue%2Fsmartdenovo","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fruanjue%2Fsmartdenovo/lists"}