{"id":17125705,"url":"https://github.com/daniel-liu-c0deb0t/block-aligner","last_synced_at":"2025-04-06T06:13:33.951Z","repository":{"id":40702606,"uuid":"310771920","full_name":"Daniel-Liu-c0deb0t/block-aligner","owner":"Daniel-Liu-c0deb0t","description":"SIMD-accelerated library for computing global and X-drop affine gap penalty sequence-to-sequence or sequence-to-profile alignments using an adaptive block-based algorithm.","archived":false,"fork":false,"pushed_at":"2024-06-23T05:21:50.000Z","size":2794,"stargazers_count":134,"open_issues_count":13,"forks_count":7,"subscribers_count":6,"default_branch":"main","last_synced_at":"2025-03-30T05:07:47.525Z","etag":null,"topics":["algorithms","alignment","avx2","bioinformatics","neon","rust","simd","wasm","webassembly"],"latest_commit_sha":null,"homepage":"https://crates.io/crates/block_aligner","language":"Jupyter Notebook","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/Daniel-Liu-c0deb0t.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2020-11-07T05:08:02.000Z","updated_at":"2025-03-26T14:00:18.000Z","dependencies_parsed_at":"2023-10-17T05:01:09.465Z","dependency_job_id":"465537b3-b7e8-4c41-a036-d08b436ce10c","html_url":"https://github.com/Daniel-Liu-c0deb0t/block-aligner","commit_stats":{"total_commits":314,"total_committers":2,"mean_commits":157.0,"dds":0.006369426751592355,"last_synced_commit":"6c569b640f2b2c62aa0ee7a36ca44859e37b5398"},"previous_names":[],"tags_count":1,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Daniel-Liu-c0deb0t%2Fblock-aligner","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Daniel-Liu-c0deb0t%2Fblock-aligner/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Daniel-Liu-c0deb0t%2Fblock-aligner/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Daniel-Liu-c0deb0t%2Fblock-aligner/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/Daniel-Liu-c0deb0t","download_url":"https://codeload.github.com/Daniel-Liu-c0deb0t/block-aligner/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":247441059,"owners_count":20939239,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["algorithms","alignment","avx2","bioinformatics","neon","rust","simd","wasm","webassembly"],"created_at":"2024-10-14T18:45:35.133Z","updated_at":"2025-04-06T06:13:33.926Z","avatar_url":"https://github.com/Daniel-Liu-c0deb0t.png","language":"Jupyter Notebook","funding_links":[],"categories":[],"sub_categories":[],"readme":"# block aligner\n[![CI](https://github.com/Daniel-Liu-c0deb0t/block-aligner/actions/workflows/ci.yaml/badge.svg)](https://github.com/Daniel-Liu-c0deb0t/block-aligner/actions/workflows/ci.yaml)\n[![License](https://img.shields.io/github/license/Daniel-Liu-c0deb0t/block-aligner)](LICENSE)\n[![Crates.io](https://img.shields.io/crates/v/block-aligner)](https://crates.io/crates/block_aligner)\n[![Docs.rs](https://docs.rs/block-aligner/badge.svg)](https://docs.rs/block-aligner)\n\nSIMD-accelerated library for computing global and X-drop affine gap penalty sequence-to-sequence or\nsequence-to-profile alignments using an adaptive block-based algorithm.\n\nSee the Bioinformatics paper [here](https://doi.org/10.1093/bioinformatics/btad487) for more info on the\nalgorithm and how it compares with other algorithms.\n\nIf you found Block Aligner useful, please cite:\n```\n@article{liu2023block,\n  title={{Block Aligner}: an adaptive {SIMD}-accelerated aligner for sequences and position-specific scoring matrices},\n  author={Liu, Daniel and Steinegger, Martin},\n  journal={Bioinformatics},\n  volume={39},\n  number={8},\n  pages={btad487},\n  year={2023},\n  publisher={Oxford University Press}\n}\n```\n\n\u003cp align = \"center\"\u003e\n\u003cimg src = \"vis/block_img1.png\" width = \"300px\"\u003e\n\u003c/p\u003e\n\n## Example\n```rust\nuse block_aligner::{cigar::*, scan_block::*, scores::*};\n\nlet min_block_size = 32;\nlet max_block_size = 256;\n\n// A gap of length n will cost: open + extend * (n - 1)\nlet gaps = Gaps { open: -2, extend: -1 };\n\n// Note that PaddedBytes, Block, and Cigar can be initialized with sequence length\n// and block size upper bounds and be reused later for shorter sequences, to avoid\n// repeated allocations.\nlet r = PaddedBytes::from_bytes::\u003cNucMatrix\u003e(b\"TTAAAAAAATTTTTTTTTTTT\", max_block_size);\nlet q = PaddedBytes::from_bytes::\u003cNucMatrix\u003e(b\"TTTTTTTTAAAAAAATTTTTTTTT\", max_block_size);\n\n// Align with traceback, but no X-drop threshold (global alignment).\nlet mut a = Block::\u003ctrue, false\u003e::new(q.len(), r.len(), max_block_size);\na.align(\u0026q, \u0026r, \u0026NW1, gaps, min_block_size..=max_block_size, 0);\nlet res = a.res();\n\nassert_eq!(res, AlignResult { score: 7, query_idx: 24, reference_idx: 21 });\n\nlet mut cigar = Cigar::new(res.query_idx, res.reference_idx);\n// Compute traceback and resolve =/X (matches/mismatches).\na.trace().cigar_eq(\u0026q, \u0026r, res.query_idx, res.reference_idx, \u0026mut cigar);\n\nassert_eq!(cigar.to_string(), \"2=6I16=3D\");\n```\nSee the [docs](https://docs.rs/block-aligner) for detailed API information.\n\nHere is a diagram showing alignment types that are supported in Block Aligner:\n\n\u003cp align = \"center\"\u003e\n\u003cimg src = \"block_aligner_modes.png\" width = \"400px\"\u003e\n\u003c/p\u003e\n\n## Algorithm\nBlock aligner provides a new efficient way to compute pairwise alignments on proteins, DNA sequences,\nand byte strings with dynamic programming.\nBlock aligner also supports aligning sequences to profiles, which are position-specific\nscoring matrices and position-specific gap open costs.\n\nIt works by calculating scores in a small square block that is shifted down or right in a greedy\nmanner, based on the scores at the edges of the block.\nThis dynamic approach results in a much smaller calculated block area compared to previous approaches,\nthough at the expense of some accuracy.\nThe block can also go back to a previous best checkpoint and grow larger, to handle difficult regions\nwith large gaps.\nThe block size can also dynamically shrink when it detects that a large block is not needed.\nBoth block growing and shrinking are based on heuristics.\n\nBy trading off some accuracy for speed, block aligner is able to efficiently handle a variety of scoring matrices and\nadapt to sequences of varying sequence identities. In practice, it is still very accurate on a variety of protein and\nnucleotide sequences.\n\nBlock aligner is designed to exploit SIMD parallelism on modern CPUs.\nCurrently, SSE2 (128-bit vectors), AVX2 (256-bit vectors), Neon (128-bit vectors), and WASM SIMD (128-bit vectors) are supported.\nFor score calculations, 16-bit score values (lanes) and 32-bit per block offsets are used.\n\nBlock aligner behaves similarly to an (adaptive) banded aligner when the minimum and maximum block size is set to\nthe same value.\n\n## Tuning block sizes\n\nFor long, noisy Nanopore reads, a min block size of ~1% sequence length and a max block size\nof ~10% sequence length performs well (tested with reads up to ~50kbps).\nFor proteins, a min block size of 32 and a max block size of 256 performs well.\nUsing a minimum block size that is at least 32 is recommended for most applications.\nUsing a maximum block size greater than `2^14 = 16384` is not recommended.\nThe library contains a `percent_len` function that computes a percentage of the sequence length with these recommendations.\nIf the alignment scores are saturating (score too large), then use a smaller block size.\nLet me know how block aligner performs on your data!\n\n## Install\nThis library can be used on both stable and nightly Rust channels.\nThe nightly channel is needed for running tests and benchmarks. Additionally, the tests\nand benchmarks need to run on Linux or MacOS.\n\nTo use this as a crate in your Rust project, add the following to your `Cargo.toml`:\n```\n[dependencies]\nblock-aligner = { version = \"0.5\", features = [\"simd_avx2\"] }\n```\nUse the `simd_sse2`, `simd_neon`, or `simd_wasm` feature flag for x86 SSE2, ARM Neon, or WASM SIMD support, respectively.\nIt is your responsibility to ensure the correct feature to be enabled and supported by the\nplatform that runs the code because this library does not automatically detect the supported\nSIMD instruction set. More information on specifying different features for different platforms\nwith the same dependency [here](https://doc.rust-lang.org/cargo/reference/specifying-dependencies.html#platform-specific-dependencies).\nHere's a simple example:\n```\n[target.'cfg(target_arch = \"x86_64\")'.dependencies]\nblock-aligner = { version = \"0.5\", features = [\"simd_avx2\"] }\n[target.'cfg(target_arch = \"aarch64\")'.dependencies]\nblock-aligner = { version = \"0.5\", features = [\"simd_neon\"] }\n```\n\nFor developing, testing, or using the C API, you should clone this repo\nand use Rust nightly. In general, when building, you need to specify the\ncorrect feature flags through the command line.\n\nFor x86 AVX2:\n```\ncargo build --features simd_avx2 --release\n```\n\nFor x86 SSE2:\n```\ncargo build --features simd_sse2 --release\n```\n\nFor ARM Neon:\n```\ncargo build --target=aarch64-unknown-linux-gnu --features simd_neon --release\n```\n\nFor WASM SIMD:\n```\ncargo build --target=wasm32-wasi --features simd_wasm --release\n```\n\nTo run WASM programs, you will need [`wasmtime`](https://github.com/bytecodealliance/wasmtime)\ninstalled and on your `$PATH`.\n\n## C API\nThere are C bindings for block aligner. More information on how to use them is located in\nthe [C readme](c/README.md).\nSee the `3di` branch for an example of using block aligner to do local alignment in C,\nalong with block aligner modifications to support aligning with amino acid 3D interaction (3Di) information.\n\n## Improving Block Aligner\nDuring alignment, three decisions need to be made at each step (using heuristics):\n* Whether to grow the block size\n* Whether to shrink the block size\n* Whether to shift right or down\n\nBlock aligner uses simple greedy heuristics that are cheap to evaluate for making these decisions.\nThere is probably a lot of room to improve here! Maybe seeds? Neural network models?\n\nTo try your ideas, take a look at the code after the comment `// TODO: better heuristics?` in `src/scan_block.rs`\n(depending on your changes, you may need to modify other parts of the code too). Let me know if you\nare working on new ideas!\n\n**Most of the instructions below are for benchmarking and testing block aligner.**\n\n## Data\nSome Illumina/Nanopore (DNA), Uniclust30 (protein), and SCOP (protein profile) data are used in some tests and benchmarks.\nYou will need to download them by following the instructions in the [data readme](data/README.md).\n\n## Test\nRun `scripts/test_avx2.sh` or `scripts/test_wasm.sh` to run tests.\nCI will run these tests when commits are pushed to this repo.\nMore testing and evaluating scripts are available in the `scripts` directory.\n\nFor debugging, there exists a `debug` feature flag that prints out a lot of\nuseful info about the internal state of the aligner while it runs.\nThere is another feature flag, `debug_size`, that prints the sizes of blocks after they grow.\nTo manually inspect alignments, run `scripts/debug_avx2.sh` with two sequences as arguments.\n\n## Docs\nRun `scripts/doc_avx2.sh` or `scripts/doc_wasm.sh` to build the docs locally.\n\n## Benchmark\nRun `scripts/bench_avx2.sh` or `scripts/bench_wasm.sh` for basic benchmarks.\nSee the `scripts` directory for runnable benchmark scripts on real data.\nMost of the actual implementations of the benchmarks are in the `examples` directory.\n\n## Data analysis and visualizations\nUse the Jupyter notebook in the `vis/` directory to gather data and plot them. An easier way\nto run the whole notebook is to run the `vis/run_vis.sh` script. This reproduces the\nexperiments in the manuscript.\n\n## Profiling with MacOS Instruments\nUse\n```\nbrew install cargo-instruments\nRUSTFLAGS=\"-g\" cargo instruments --example profile --release --features simd_avx2 --open\n```\n\n## Analyzing performance with LLVM-MCA\nUse\n```\nscripts/build_ir_asm.sh\n```\nto generate assembly output and run LLVM-MCA.\n\n## Viewing the assembly\nUse either `scripts/build_ir_asm.sh`, `objdump -d` on a binary (avoids recompiling code in\nsome cases), or a more advanced tool like Ghidra (has a decompiler, too).\n\n## Compare (relatively unused)\nEdits were made to [Hajime Suzuki](https://github.com/ocxtal)'s adaptive banding benchmark code\nand difference recurrence benchmark code. These edits are available [here](https://github.com/Daniel-Liu-c0deb0t/adaptivebandbench)\nand [here](https://github.com/Daniel-Liu-c0deb0t/diff-bench-paper), respectively.\nGo to those repos, then follow the instructions for installing and running the code.\n\nIf you run the scripts in those repos for comparing scores produced by different algorithms,\nyou should get `.tsv` generated files. Then, in this repo's directory, run\n```\nscripts/compare_avx2.sh /path/to/file.tsv 50\n```\nto get the comparisons. The X-drop threshold is specified after the path.\n\n## Old ideas and history\nSee the [ideas](ideas.md) file.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdaniel-liu-c0deb0t%2Fblock-aligner","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fdaniel-liu-c0deb0t%2Fblock-aligner","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdaniel-liu-c0deb0t%2Fblock-aligner/lists"}