{"id":46460818,"url":"https://github.com/mcwdsi/bam2tensor","last_synced_at":"2026-05-12T02:06:06.819Z","repository":{"id":212528140,"uuid":"730830533","full_name":"mcwdsi/bam2tensor","owner":"mcwdsi","description":"Convert methylation data to sparse objects for machine learning.","archived":false,"fork":false,"pushed_at":"2026-04-18T11:44:09.000Z","size":19413,"stargazers_count":1,"open_issues_count":0,"forks_count":0,"subscribers_count":2,"default_branch":"main","last_synced_at":"2026-04-18T13:21:56.803Z","etag":null,"topics":["machine-learning","methylation"],"latest_commit_sha":null,"homepage":"https://mcwdsi.github.io/bam2tensor/","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/mcwdsi.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":"SECURITY.md","support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2023-12-12T19:09:41.000Z","updated_at":"2026-04-18T11:42:45.000Z","dependencies_parsed_at":"2024-01-16T23:34:02.342Z","dependency_job_id":"97f4e2bb-3c4d-4754-b921-cf6920426948","html_url":"https://github.com/mcwdsi/bam2tensor","commit_stats":null,"previous_names":["mcwdsi/bam2tensor"],"tags_count":18,"template":false,"template_full_name":null,"purl":"pkg:github/mcwdsi/bam2tensor","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mcwdsi%2Fbam2tensor","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mcwdsi%2Fbam2tensor/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mcwdsi%2Fbam2tensor/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mcwdsi%2Fbam2tensor/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/mcwdsi","download_url":"https://codeload.github.com/mcwdsi/bam2tensor/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mcwdsi%2Fbam2tensor/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":32920399,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-11T17:09:15.040Z","status":"online","status_checked_at":"2026-05-12T02:00:06.338Z","response_time":102,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["machine-learning","methylation"],"created_at":"2026-03-06T03:10:01.004Z","updated_at":"2026-05-12T02:06:06.810Z","avatar_url":"https://github.com/mcwdsi.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# bam2tensor\n\n**Author:** [Nick Semenkovich](https://nick.semenkovich.com/) (semenko@alum.mit.edu)\n\n[![PyPI](https://img.shields.io/pypi/v/bam2tensor.svg)][pypi status]\n[![Status](https://img.shields.io/pypi/status/bam2tensor.svg)][pypi status]\n[![Python Version](https://img.shields.io/pypi/pyversions/bam2tensor)][pypi status]\n[![License](https://img.shields.io/pypi/l/bam2tensor)][license]\n\n[![Documentation](https://github.com/mcwdsi/bam2tensor/actions/workflows/docs.yml/badge.svg)][documentation]\n[![Tests](https://github.com/mcwdsi/bam2tensor/actions/workflows/tests.yml/badge.svg)][tests]\n[![Coverage](https://img.shields.io/endpoint?url=https://raw.githubusercontent.com/mcwdsi/bam2tensor/python-coverage-comment-action-data/endpoint.json)][coverage]\n\n[![pre-commit](https://img.shields.io/badge/pre--commit-enabled-brightgreen?logo=pre-commit\u0026logoColor=white)][pre-commit]\n[![Black](https://img.shields.io/badge/code%20style-black-000000.svg)][black]\n[![Ruff](https://img.shields.io/endpoint?url=https://raw.githubusercontent.com/astral-sh/ruff/main/assets/badge/v2.json)](https://github.com/astral-sh/ruff)\n\n[pypi status]: https://pypi.org/project/bam2tensor/\n[documentation]: https://mcwdsi.github.io/bam2tensor\n[tests]: https://github.com/mcwdsi/bam2tensor/actions?workflow=Tests\n[coverage]: https://htmlpreview.github.io/?https://github.com/mcwdsi/bam2tensor/blob/python-coverage-comment-action-data/htmlcov/index.html\n[pre-commit]: https://github.com/pre-commit/pre-commit\n[black]: https://github.com/psf/black\n\n**bam2tensor** is a Python package for converting bisulfite-sequencing (BS-seq) and enzymatic methylation sequencing (EM-seq) `.bam` files to sparse tensor representations of DNA methylation data. It extracts read-level methylation states from CpG sites and outputs efficient sparse COO matrices as `.npz` files, ready for deep learning pipelines.\n\n![bam2tensor overview figure](https://raw.githubusercontent.com/mcwdsi/bam2tensor/main/docs/nano-banana-overview-shrunk.png)\n\n## Table of Contents\n\n- [Features](#features)\n- [Requirements](#requirements)\n- [Installation](#installation)\n- [Quick Start](#quick-start)\n- [Usage](#usage)\n  - [Basic Usage](#basic-usage)\n  - [Auto-Download Reference Genomes](#auto-download-reference-genomes)\n  - [Processing Multiple BAM Files](#processing-multiple-bam-files)\n  - [Custom Output Directory](#custom-output-directory)\n  - [Using a Custom Genome](#using-a-custom-genome)\n  - [Command-Line Options](#command-line-options)\n- [Filtering Conversion Errors](#filtering-conversion-errors)\n- [Inspecting Output Files](#inspecting-output-files)\n- [Output Data Structure](#output-data-structure)\n  - [Per-Read Fragment Length (TLEN)](#per-read-fragment-length-tlen)\n  - [Embedded Metadata](#embedded-metadata)\n  - [Loading Output Files](#loading-output-files)\n  - [Converting to Dense Arrays](#converting-to-dense-arrays)\n  - [Working with Genomic Coordinates](#working-with-genomic-coordinates)\n- [Supported Aligners](#supported-aligners)\n- [Performance Tips](#performance-tips)\n- [API Reference](#api-reference)\n- [Contributing](#contributing)\n- [AI Coding Agents](#ai-coding-agents)\n- [License](#license)\n- [Credits](#credits)\n\n## Features\n\n- **BAM Parsing**: Efficiently parses `.bam` files using [pysam](https://github.com/pysam-developers/pysam)\n- **Complete CpG Extraction**: Extracts methylation data from all CpG sites genome-wide\n- **Multi-Genome Support**: Works with any reference genome (GRCh38/hg38, T2T-CHM13, mm10, etc.)\n- **Sparse Storage**: Stores data in sparse COO matrix format for memory-efficient loading\n- **NumPy/SciPy Integration**: Exports to `.npz` files compatible with NumPy and SciPy\n- **Efficient Algorithm**: Linear-scan algorithm ensures minimal memory usage with no read duplication\n- **Batch Processing**: Process multiple BAM files with directory recursion\n- **Caching**: CpG site indexing is cached to accelerate repeated runs on the same genome\n- **Quality Filtering**: Configurable mapping quality thresholds\n- **Per-Read Fragment Length**: Stores BAM TLEN (template length) alongside the methylation tensor for joint fragment-methylation analysis\n- **Conversion-Error Filters**: Optional per-read filters for incomplete bisulfite/EM-seq conversion (ported from `nebiolabs/mark-nonconverted-reads`) and EM-seq fragment-level over-conversion (Loyfer et al. 2026)\n\n## Requirements\n\n- Python 3.10 or higher\n- A reference genome FASTA file (must match the genome used for alignment)\n- Indexed BAM files (`.bam` with corresponding `.bam.bai` index files)\n\n### Dependencies\n\nCore dependencies are automatically installed:\n- `pysam` - BAM file handling\n- `biopython` - FASTA parsing\n- `scipy` - Sparse matrix operations\n- `numpy` - Numerical operations\n- `click` - Command-line interface\n- `tqdm` - Progress bars\n\n## Installation\n\n### From PyPI (Recommended)\n\n```bash\npip install bam2tensor\n```\n\n### From Source\n\n```bash\ngit clone https://github.com/mcwdsi/bam2tensor.git\ncd bam2tensor\npip install .\n```\n\n### Development Installation\n\n```bash\ngit clone https://github.com/mcwdsi/bam2tensor.git\ncd bam2tensor\nuv sync\n```\n\n## Quick Start\n\n```bash\n# Basic usage with a single BAM file\nbam2tensor \\\n    --input-path sample.bam \\\n    --reference-fasta GRCh38.fa \\\n    --genome-name hg38\n\n# This creates: sample.methylation.npz\n```\n\n## Usage\n\n### Basic Usage\n\nProcess a single bisulfite-sequencing or EM-seq BAM file:\n\n```bash\nbam2tensor \\\n    --input-path /path/to/aligned_reads.bam \\\n    --reference-fasta /path/to/reference.fa \\\n    --genome-name hg38\n```\n\nThis will:\n1. Parse the reference FASTA to identify all CpG sites (cached for future runs)\n2. Extract methylation states from each read in the BAM file\n3. Output a sparse matrix to `aligned_reads.methylation.npz`\n\n### Auto-Download Reference Genomes\n\nDon't have a local reference FASTA? bam2tensor can download and cache common reference genomes automatically:\n\n```bash\n# Download hg38 and process a BAM file in one command\nbam2tensor \\\n    --input-path sample.bam \\\n    --download-reference hg38\n\n# List available genomes\nbam2tensor --input-path sample.bam --list-genomes\n```\n\nAvailable genomes: `hg38`, `hg19`, `mm10`, `T2T-CHM13`. Downloaded references are cached in `~/.cache/bam2tensor/` for future runs.\n\n### Processing Multiple BAM Files\n\nProcess all BAM files in a directory recursively:\n\n```bash\nbam2tensor \\\n    --input-path /path/to/bam_directory/ \\\n    --reference-fasta /path/to/reference.fa \\\n    --genome-name hg38 \\\n    --verbose\n```\n\nEach BAM file will generate a corresponding `.methylation.npz` file in the same location.\n\n### Custom Output Directory\n\nWrite output files to a specific directory instead of next to the input BAMs:\n\n```bash\nbam2tensor \\\n    --input-path /shared/bams/ \\\n    --reference-fasta /shared/ref/GRCh38.fa \\\n    --genome-name hg38 \\\n    --output-dir /scratch/methylation_output/\n```\n\nThis is useful on HPC clusters where input directories may be read-only or on slow shared storage.\n\n### Using a Custom Genome\n\nFor non-human genomes or custom chromosome sets:\n\n```bash\n# Mouse genome (mm10)\nbam2tensor \\\n    --input-path mouse_sample.bam \\\n    --reference-fasta mm10.fa \\\n    --genome-name mm10 \\\n    --expected-chromosomes \"chr1,chr2,chr3,chr4,chr5,chr6,chr7,chr8,chr9,chr10,chr11,chr12,chr13,chr14,chr15,chr16,chr17,chr18,chr19,chrX,chrY\"\n\n# T2T-CHM13 human genome\nbam2tensor \\\n    --input-path sample.bam \\\n    --reference-fasta chm13v2.0.fa \\\n    --genome-name T2T-CHM13 \\\n    --expected-chromosomes \"chr1,chr2,chr3,chr4,chr5,chr6,chr7,chr8,chr9,chr10,chr11,chr12,chr13,chr14,chr15,chr16,chr17,chr18,chr19,chr20,chr21,chr22,chrX,chrY\"\n```\n\n### Command-Line Options\n\n```\nUsage: bam2tensor [OPTIONS]\n\n  Extract read-level methylation data from an aligned bisulfite-seq or EM-seq\n  .bam file and export the data as a SciPy sparse matrix.\n\nOptions:\n  --version                       Show the version and exit.\n  --input-path PATH               Input .bam file OR directory to recursively\n                                  process.  [required]\n  --genome-name TEXT              A custom string referring to your genome\n                                  name, used to save a cache file (e.g. hg38,\n                                  hg39-no-alt, etc.).\n  --expected-chromosomes TEXT     A comma-separated list of chromosomes to\n                                  expect in the .fa genome. Defaults to hg38\n                                  chromosomes.\n  --reference-fasta FILE          Reference genome fasta file (critical to\n                                  determine CpG sites).\n  --quality-limit INTEGER         Quality filter for aligned reads (default =\n                                  20)\n  --filter-non-converted          Drop reads with \u003e= --non-converted-threshold\n                                  retained non-CpG cytosines, the signature of\n                                  incomplete bisulfite/EM-seq conversion (port\n                                  of nebiolabs/mark-nonconverted-reads).\n                                  Default: off.\n  --non-converted-threshold INTEGER\n                                  Minimum count of retained non-CpG cytosines\n                                  to drop a read (default = 3, matches NEB\n                                  mark-nonconverted-reads).\n  --filter-em-overconversion      Drop EM-seq reads whose covered CpGs are all\n                                  called unmethylated and cover at least --em-\n                                  overconversion-min-cpgs sites (heuristic for\n                                  the fragment-level over-conversion artifact\n                                  described in Loyfer et al. bioRxiv\n                                  2026.03.24.713040). Default: off.\n  --em-overconversion-min-cpgs INTEGER\n                                  Minimum covered CpG count required before\n                                  the EM over-conversion filter will drop a\n                                  read (default = 3).\n  --verbose                       Verbose output.\n  --skip-cache                    De-novo generate CpG sites (slow).\n  --debug                         Debug mode (extensive validity checking +\n                                  debug messages).\n  --overwrite                     Overwrite output file if it exists.\n  --output-dir DIRECTORY          Output directory for .methylation.npz files.\n                                  Defaults to same directory as the input BAM.\n  --download-reference [t2t-chm13|hg19|hg38|mm10]\n                                  Download and cache a known reference genome\n                                  (e.g. hg38, hg19, mm10, T2T-CHM13). Use\n                                  --list-genomes to see options.\n  --list-genomes                  List available reference genomes for\n                                  download and exit.\n  --help                          Show this message and exit.\n```\n\n#### Option Details\n\n| Option | Description |\n|--------|-------------|\n| `--input-path` | Path to a single `.bam` file or a directory. If a directory is provided, all `.bam` files are processed recursively. |\n| `--genome-name` | An identifier for your reference genome (e.g., `hg38`, `mm10`). Used to name the cache file for CpG site positions. |\n| `--expected-chromosomes` | Comma-separated list of chromosome names to process. Chromosomes not in this list are skipped. Defaults to human autosomes + sex chromosomes. |\n| `--reference-fasta` | Path to the reference genome FASTA file. Must match the genome used for alignment. |\n| `--quality-limit` | Minimum mapping quality score (MAPQ) for reads to be included. Default is 20. |\n| `--filter-non-converted` | Drop reads with retained non-CpG cytosines above `--non-converted-threshold` (incomplete conversion). See [Filtering Conversion Errors](#filtering-conversion-errors). |\n| `--non-converted-threshold` | Threshold for the non-converted filter. Default is 3. |\n| `--filter-em-overconversion` | Drop EM-seq reads whose covered CpGs are all unmethylated and cover ≥ `--em-overconversion-min-cpgs` sites. See [Filtering Conversion Errors](#filtering-conversion-errors). |\n| `--em-overconversion-min-cpgs` | Minimum covered CpG count before the EM over-conversion filter will drop a read. Default is 3. |\n| `--verbose` | Enable detailed progress output including per-chromosome progress bars. |\n| `--skip-cache` | Force regeneration of CpG site cache. Useful if you've modified the reference or chromosome list. |\n| `--debug` | Enable extensive validation and debug output. Slower but useful for troubleshooting. |\n| `--overwrite` | Overwrite existing `.methylation.npz` files. Without this flag, existing outputs are skipped. |\n| `--output-dir` | Write `.methylation.npz` files to this directory instead of next to the input BAMs. Created automatically if it doesn't exist. |\n| `--download-reference` | Download and cache a known reference genome. Choices: `hg38`, `hg19`, `mm10`, `T2T-CHM13`. Replaces `--reference-fasta`. |\n| `--list-genomes` | List available reference genomes for `--download-reference` and exit. |\n\n## Filtering Conversion Errors\n\nBisulfite and EM-seq library preparation can produce two kinds of per-read conversion errors that bias downstream methylation calls. bam2tensor provides two opt-in filters to drop affected reads at extraction time. Both are **default-off**, apply per read, and are recorded in the output `metadata.json` so downstream consumers know which filters were applied.\n\n### `--filter-non-converted` — incomplete conversion\n\nPorts the logic of [nebiolabs/mark-nonconverted-reads](https://github.com/nebiolabs/mark-nonconverted-reads). A read is dropped if it carries at least `--non-converted-threshold` (default 3) retained non-CpG cytosines, a signature of incomplete bisulfite or EM-seq conversion.\n\n- **Bismark BAMs**: counted directly from the `XM` tag's uppercase `H`/`X`/`U` characters (retained cytosines in CHH/CHG/unknown contexts).\n- **Biscuit / bwameth / gem3 BAMs**: counted by comparing the read to the reference via the `MD` tag (using pysam's `get_aligned_pairs(with_seq=True)`). SNPs — where the read's retained `C` sits over a reference base that isn't `C` — are excluded from the count, matching NEB's reference-validation step. No separate FASTA reload is required.\n\n### `--filter-em-overconversion` — EM-seq fragment-level over-conversion\n\nA heuristic inspired by [Loyfer et al. (bioRxiv 2026.03.24.713040)](https://www.biorxiv.org/content/10.64898/2026.03.24.713040v1). That paper shows EM-seq reproducibly produces ~1–2.5% of multi-CpG fragments that appear fully unmethylated across every covered CpG — a fragment-level artifact absent from WGBS and Oxford Nanopore. This filter drops any read whose covered CpGs are **all** called unmethylated *and* cover at least `--em-overconversion-min-cpgs` sites (default 3, the regime where the EM-seq artifact is clearly separable from WGBS in Loyfer et al. Fig. 1C).\n\nThe filter is a blunt instrument: it will also drop genuinely fully-unmethylated biological fragments at unmethylated markers. Enable it only when your downstream application (e.g., cfDNA deconvolution at constitutively methylated loci) can tolerate that trade-off.\n\n### Usage\n\n```bash\nbam2tensor \\\n    --input-path sample.bam \\\n    --reference-fasta GRCh38.fa \\\n    --genome-name hg38 \\\n    --filter-non-converted \\\n    --filter-em-overconversion\n```\n\nFilter parameters and enabled state are written to the output `metadata.json`:\n\n```json\n{\n    \"filters\": {\n        \"non_converted_reads\": {\"enabled\": true, \"threshold\": 3},\n        \"em_overconversion\": {\"enabled\": true, \"min_cpgs\": 3}\n    }\n}\n```\n\n### Reproducibility note\n\nThe two filters differ in whether they can be replayed downstream without the source BAM:\n\n- **`--filter-em-overconversion` is reproducible from the `.npz` alone.** The heuristic is a pure function of each row's CpG state values. A downstream consumer who receives an unfiltered `.npz` can replay the filter at analysis time:\n\n  ```python\n  import scipy.sparse\n  mat = scipy.sparse.load_npz(\"sample.methylation.npz\").tocsr()\n  min_cpgs = 3\n  kept_rows = []\n  for i in range(mat.shape[0]):\n      row = mat.getrow(i).toarray().ravel()\n      covered = row[(row == 0) | (row == 1)]  # drop -1 no-data\n      is_overconv = len(covered) \u003e= min_cpgs and (covered == 0).all()\n      if not is_overconv:\n          kept_rows.append(i)\n  ```\n\n- **`--filter-non-converted` is *not* reproducible from the `.npz` alone.** It relies on retained non-CpG cytosines (or Bismark's `H`/`X`/`U`), which are never written to the matrix. If you need this filter, apply it at extraction time (or re-run bam2tensor against the original BAM).\n\n## Inspecting Output Files\n\nUse `bam2tensor-inspect` to view a summary of any `.methylation.npz` file without writing Python:\n\n```bash\n$ bam2tensor-inspect sample.methylation.npz\nsample.methylation.npz\n  Genome:          hg38\n  Chromosomes:     24 (chr1, chr2, ... chrX, chrY)\n  Reads:           1,423,891\n  CpG sites:       28,217,448\n  Data points:     12,847,322 (sparsity: 99.97%)\n  Fragment len:    median 167, mean 182, range [50, 600]\n  Filters:         non-converted (\u003e= 3 non-CpG Cs)\n                   EM over-conversion (all-unmethylated, \u003e= 3 CpGs)\n  CpG index CRC32: a1b2c3d4\n  bam2tensor:      v2.5\n  File size:       14.2 MB\n```\n\nWhen no filters were applied, the line reads `Filters:         none`. Files produced by bam2tensor versions older than v2.5 omit the line entirely.\n\nYou can pass multiple files at once:\n\n```bash\n$ bam2tensor-inspect *.methylation.npz\n```\n\nThis works on files produced by older versions of bam2tensor too (metadata fields will be omitted).\n\n## Output Data Structure\n\nbam2tensor generates one `.methylation.npz` file per input BAM file. Each file contains a SciPy sparse [COO matrix] (`scipy.sparse.coo_matrix`) with the following structure:\n\n| Dimension | Represents |\n|-----------|------------|\n| **Rows** | Unique sequencing reads (primary alignments that pass quality and flag filters, numbered sequentially as encountered across chromosomes) |\n| **Columns** | CpG sites from the reference genome, ordered by genomic position across all chromosomes (chr1, chr2, ..., chrX, chrY). Column `i` maps to the `i`-th CpG dinucleotide in the reference FASTA. |\n\nThe **column dimension is determined entirely by the reference genome**: it equals the total number of CpG sites across all `--expected-chromosomes`. For example, hg38 with default chromosomes has ~28 million CpG columns. To map column indices back to genomic coordinates (e.g., column 12345 → chr1:29503), use the `GenomeMethylationEmbedding` class with the same reference FASTA and chromosome list (see [Working with Genomic Coordinates](#working-with-genomic-coordinates) below).\n\n### Methylation State Values\n\n| Value | Meaning |\n|-------|---------|\n| `1` | Methylated (cytosine preserved as C after bisulfite/enzymatic conversion) |\n| `0` | Unmethylated (cytosine converted to T by bisulfite/enzymatic treatment) |\n| `-1` | No data (indel, SNV, or other non-C/T base at a CpG position) |\n\nNote: The matrix uses SciPy's COO sparse format, which explicitly stores all non-zero values. Unmethylated sites (value `0`) **are** stored as explicit entries. Positions not covered by a read are simply absent from the matrix (implicit zero, which is distinct from the explicit `0` = unmethylated).\n\n### Per-Read Fragment Length (TLEN)\n\nEach `.methylation.npz` file includes a `tlen.npy` entry inside the ZIP archive containing the signed BAM template length (TLEN) for every read in the matrix. This enables joint fragment-length and methylation analysis without re-processing the BAM.\n\n- One `int32` value per read (row), in the same order as the sparse matrix rows\n- Signed: positive for the leftmost read in a pair, negative for the rightmost\n- Zero for single-end reads or reads with unmapped mates\n- Use `abs(tlen)` to get fragment lengths\n\n```python\nfrom bam2tensor.metadata import read_npz_tlen\nimport numpy as np\n\ntlen = read_npz_tlen(\"sample.methylation.npz\")\nif tlen is not None:\n    frag_lengths = np.abs(tlen)\n    nonzero = frag_lengths[frag_lengths \u003e 0]\n    print(f\"Median fragment length: {np.median(nonzero):.0f}\")\n    print(f\"Mean fragment length: {np.mean(nonzero):.0f}\")\n```\n\n### Embedded Metadata\n\nEach `.methylation.npz` file includes a `metadata.json` entry inside the ZIP archive with provenance information:\n\n| Field | Description |\n|-------|-------------|\n| `bam2tensor_version` | Version of bam2tensor that produced the file |\n| `genome_name` | Genome identifier (e.g., `hg38`, `mm10`) |\n| `expected_chromosomes` | List of chromosomes included in the column mapping |\n| `total_cpg_sites` | Total number of CpG columns in the matrix |\n| `cpg_index_crc32` | CRC32 checksum of the CpG site positions (verifies identical column semantics) |\n| `filters` | Nested dict recording which opt-in conversion-error filters were applied (`non_converted_reads`, `em_overconversion`) and their parameters. See [Filtering Conversion Errors](#filtering-conversion-errors). Added in v2.5. |\n\nThis metadata is ignored by `scipy.sparse.load_npz`, so existing code continues to work. To read it:\n\n```python\nfrom bam2tensor.metadata import read_npz_metadata\n\nmeta = read_npz_metadata(\"sample.methylation.npz\")\nif meta is not None:\n    print(f\"Genome: {meta['genome_name']}\")\n    print(f\"CpG sites: {meta['total_cpg_sites']:,}\")\n    print(f\"CpG index CRC32: {meta['cpg_index_crc32']}\")\n```\n\nThe `cpg_index_crc32` field uniquely identifies the column mapping. Two files with the same CRC32 have identical column semantics (same chromosomes, same CpG positions, same order) and their matrices can be directly stacked or compared. The metadata is also accessible without bam2tensor installed, since `.npz` files are ZIP archives:\n\n```bash\nunzip -p sample.methylation.npz metadata.json | python -m json.tool\n```\n\n### Loading Output Files\n\n```python\nimport scipy.sparse\nimport numpy as np\n\n# Load the sparse matrix\nmethylation_matrix = scipy.sparse.load_npz(\"sample.methylation.npz\")\n\nprint(f\"Matrix shape: {methylation_matrix.shape}\")\nprint(f\"Number of reads: {methylation_matrix.shape[0]}\")\nprint(f\"Number of CpG sites: {methylation_matrix.shape[1]}\")\nprint(f\"Non-zero entries: {methylation_matrix.nnz}\")\nprint(f\"Sparsity: {1 - methylation_matrix.nnz / np.prod(methylation_matrix.shape):.4%}\")\n```\n\n### Converting to Dense Arrays\n\nFor small regions or when dense operations are needed:\n\n```python\n# Convert entire matrix to dense (warning: may use significant memory)\ndense_matrix = methylation_matrix.toarray()\n\n# Convert to CSR format for efficient row slicing\ncsr_matrix = methylation_matrix.tocsr()\n\n# Get methylation data for reads 0-99\nsubset = csr_matrix[0:100, :].toarray()\n\n# Convert to CSC format for efficient column slicing\ncsc_matrix = methylation_matrix.tocsc()\n\n# Get data for CpG sites 1000-1099\ncpg_subset = csc_matrix[:, 1000:1100].toarray()\n```\n\n### Working with Genomic Coordinates\n\nTo map between matrix column indices and genomic coordinates, use the `GenomeMethylationEmbedding` class:\n\n```python\nfrom bam2tensor.embedding import GenomeMethylationEmbedding\n\n# Load or recreate the embedding used during extraction\nembedding = GenomeMethylationEmbedding(\n    genome_name=\"hg38\",\n    expected_chromosomes=[\"chr\" + str(i) for i in range(1, 23)] + [\"chrX\", \"chrY\"],\n    fasta_source=\"/path/to/GRCh38.fa\",\n)\n\n# Convert matrix column index to genomic position\nchrom, pos = embedding.embedding_to_genomic_position(12345)\nprint(f\"Column 12345 corresponds to {chrom}:{pos}\")\n\n# Convert genomic position to matrix column index\ncol_idx = embedding.genomic_position_to_embedding(\"chr1\", 10525)\nprint(f\"chr1:10525 is at column {col_idx}\")\n\n# Get total number of CpG sites\nprint(f\"Total CpG sites: {embedding.total_cpg_sites:,}\")\n```\n\n### Example: Analyzing Methylation Patterns\n\n```python\nimport scipy.sparse\nimport numpy as np\n\n# Load the data\nmatrix = scipy.sparse.load_npz(\"sample.methylation.npz\")\ncsr = matrix.tocsr()\n\n# Calculate per-CpG methylation rates (excluding -1 values)\nmethylation_rates = []\nfor cpg_idx in range(matrix.shape[1]):\n    col_data = csr.getcol(cpg_idx).toarray().flatten()\n    # Filter out -1 (no data) and positions with no coverage\n    valid_data = col_data[(col_data \u003e= 0)]\n    if len(valid_data) \u003e 0:\n        rate = np.mean(valid_data)\n    else:\n        rate = np.nan\n    methylation_rates.append(rate)\n\nmethylation_rates = np.array(methylation_rates)\nprint(f\"Mean methylation rate: {np.nanmean(methylation_rates):.2%}\")\nprint(f\"CpG sites with coverage: {np.sum(~np.isnan(methylation_rates)):,}\")\n```\n\n### Example: Integration with PyTorch\n\n```python\nimport torch\nimport scipy.sparse\nimport numpy as np\n\n# Load sparse matrix\nmatrix = scipy.sparse.load_npz(\"sample.methylation.npz\")\n\n# Convert to PyTorch sparse tensor\ncoo = matrix.tocoo()\nindices = torch.LongTensor(np.vstack((coo.row, coo.col)))\nvalues = torch.FloatTensor(coo.data)\nshape = torch.Size(coo.shape)\n\nsparse_tensor = torch.sparse_coo_tensor(indices, values, shape)\nprint(f\"PyTorch sparse tensor shape: {sparse_tensor.shape}\")\n\n# For models that need dense input (specific region)\nregion_start, region_end = 0, 1000\ndense_region = matrix.tocsc()[:, region_start:region_end].toarray()\ndense_tensor = torch.FloatTensor(dense_region)\n```\n\n## Supported Aligners\n\nbam2tensor supports BAM files from bisulfite-aware and EM-seq-aware aligners that include strand information tags:\n\n| Aligner | Tag | Values |\n|---------|-----|--------|\n| [Bismark](https://github.com/FelixKrueger/Bismark) | `XM` | `Z` (methylated CpG), `z` (unmethylated CpG) |\n| [Biscuit](https://huishenlab.github.io/biscuit/) | `YD` | `f` (forward/OT/CTOT), `r` (reverse/OB/CTOB) |\n| [bwameth](https://github.com/brentp/bwameth) | `YD` | `f` (forward/OT/CTOT), `r` (reverse/OB/CTOB) |\n| [gem3](https://github.com/gemtools/gemtools) / [Blueprint](http://www.blueprint-epigenome.eu/) | `XB` | `C` (forward), `G` (reverse) |\n\nThese tags indicate which strand the original bisulfite/EM-seq-converted DNA came from, which is essential for correctly interpreting C/T as methylated/unmethylated.\n\n**Note:** EM-seq (enzymatic methyl-seq) data produces the same C-to-T conversion pattern as bisulfite sequencing and is fully supported when aligned with any of the above tools.\n\n## Performance Tips\n\n1. **Use the cache**: The first run on a new genome builds a CpG site index, which is cached. Subsequent runs are much faster.\n\n2. **Process in parallel**: bam2tensor processes one BAM at a time, but you can run multiple instances in parallel on different BAM files:\n   ```bash\n   # Using GNU parallel\n   find /data/bams -name \"*.bam\" | parallel -j 4 \\\n       bam2tensor --input-path {} --reference-fasta ref.fa --genome-name hg38\n   ```\n\n3. **Ensure BAM files are indexed**: Each BAM file requires a corresponding `.bam.bai` index file. Create with:\n   ```bash\n   samtools index sample.bam\n   ```\n\n4. **Use SSDs**: Both reading BAM files and writing output benefit from fast storage.\n\n5. **Memory considerations**: Memory usage scales with the number of CpG sites (columns) rather than reads. For human genomes (~28M CpG sites), expect moderate memory usage.\n\n## API Reference\n\n### `bam2tensor.embedding.GenomeMethylationEmbedding`\n\nMain class for managing CpG site positions and coordinate conversions.\n\n```python\nGenomeMethylationEmbedding(\n    genome_name: str,           # Identifier for caching\n    expected_chromosomes: list, # List of chromosome names to process\n    fasta_source: str,          # Path to reference FASTA\n    skip_cache: bool = False,   # Force regeneration of cache\n    verbose: bool = False       # Enable verbose output\n)\n```\n\n**Key Methods:**\n- `embedding_to_genomic_position(embedding: int) -\u003e tuple[str, int]` - Convert column index to (chromosome, position)\n- `genomic_position_to_embedding(chrom: str, pos: int) -\u003e int` - Convert genomic position to column index\n\n**Key Attributes:**\n- `total_cpg_sites: int` - Total number of CpG sites across all chromosomes\n- `cpg_sites_dict: dict[str, list[int]]` - Dictionary mapping chromosome names to lists of CpG positions\n\n### `bam2tensor.functions.extract_methylation_data_from_bam`\n\nCore function for extracting methylation data from a BAM file.\n\n```python\nextract_methylation_data_from_bam(\n    input_bam: str,                                    # Path to BAM file\n    genome_methylation_embedding: GenomeMethylationEmbedding,  # Embedding object\n    quality_limit: int = 20,                           # Minimum MAPQ\n    filter_non_converted: bool = False,                # Drop reads with retained non-CpG Cs\n    non_converted_threshold: int = 3,                  # Threshold for the above filter\n    filter_em_overconversion: bool = False,            # Drop EM-seq fragment-level over-conversion reads\n    em_overconversion_min_cpgs: int = 3,               # Min CpGs before applying the above filter\n    verbose: bool = False,                             # Enable verbose output\n    debug: bool = False                                # Enable debug output\n) -\u003e ExtractionResult\n```\n\n**Returns:** An `ExtractionResult` named tuple with two fields:\n- `matrix`: A SciPy COO sparse matrix with shape (n_reads, n_cpg_sites)\n- `tlen`: A 1-D numpy `int32` array of shape (n_reads,) containing the signed template length (BAM TLEN field) for each read\n\n### `bam2tensor.metadata.read_npz_tlen`\n\nRead per-read template lengths from a `.methylation.npz` file.\n\n```python\nread_npz_tlen(npz_path: str) -\u003e np.ndarray | None\n```\n\n**Returns:** The per-read template-length array, or `None` if the file was produced by an older version of bam2tensor.\n\n## Contributing\n\nContributions are welcome! Please see the [Contributor Guide] for guidelines on:\n- Setting up a development environment\n- Running tests\n- Code style requirements\n- Submitting pull requests\n\n### Development Commands\n\n```bash\n# Install development dependencies\nuv sync\n\n# Run all checks (linting, type checking, tests)\nnox\n\n# Run specific checks\nnox --session=tests       # Run pytest\nnox --session=mypy        # Type checking\nnox --session=pre-commit  # Linting\n\n# Format code\nuv run black src tests\nuv run ruff check --fix src tests\n```\n\n## AI Coding Agents\n\nThis project includes a `CLAUDE.md` file with detailed guidance for AI coding agents (Claude Code, Copilot, Cursor, etc.). If you're using an AI assistant to contribute, please ensure it follows the project conventions described there.\n\nKey points for AI agents:\n- Run `nox` before committing to validate all checks pass\n- Follow Google-style docstrings validated by darglint (long strictness)\n- Use `uv sync` for dependency management\n- See `CLAUDE.md` for complete guidelines\n\n## License\n\nDistributed under the terms of the [MIT license][license], bam2tensor is free and open source software.\n\n## Issues\n\nIf you encounter any problems, please [file an issue] with:\n- A description of the problem\n- Steps to reproduce\n- Your Python version and operating system\n- Relevant error messages or logs\n\n## Credits\n\n**Created and maintained by [Nick Semenkovich](https://nick.semenkovich.com/) ([@semenko](https://github.com/semenko))**, as part of the Medical College of Wisconsin's [Data Science Institute].\n\nThis project was generated from [Statistics Norway]'s [SSB PyPI Template].\n\n[Nick Semenkovich (@semenko)]: https://nick.semenkovich.com/\n[Data Science Institute]: https://www.mcw.edu/departments/data-science-institute\n[statistics norway]: https://www.ssb.no/en\n[pypi]: https://pypi.org/\n[ssb pypi template]: https://github.com/statisticsnorway/ssb-pypitemplate\n[file an issue]: https://github.com/mcwdsi/bam2tensor/issues\n[pip]: https://pip.pypa.io/\n[COO matrix]: https://docs.scipy.org/doc/scipy/reference/generated/scipy.sparse.coo_matrix.html\n[license]: https://github.com/mcwdsi/bam2tensor/blob/main/LICENSE\n[contributor guide]: https://github.com/mcwdsi/bam2tensor/blob/main/CONTRIBUTING.md\n[reference guide]: https://mcwdsi.github.io/bam2tensor/reference.html\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmcwdsi%2Fbam2tensor","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fmcwdsi%2Fbam2tensor","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmcwdsi%2Fbam2tensor/lists"}