{"id":16618582,"url":"https://github.com/brentp/spacepile","last_synced_at":"2025-10-29T19:31:41.139Z","repository":{"id":65801027,"uuid":"579126701","full_name":"brentp/spacepile","owner":"brentp","description":"convert reads from repeated measures of same piece of DNA into spaced matricies for deep learners.","archived":false,"fork":false,"pushed_at":"2023-04-21T18:44:21.000Z","size":32,"stargazers_count":14,"open_issues_count":2,"forks_count":0,"subscribers_count":3,"default_branch":"main","last_synced_at":"2025-02-02T04:41:12.761Z","etag":null,"topics":["deep-learning","genomics","sequence-alignment"],"latest_commit_sha":null,"homepage":"","language":"Rust","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/brentp.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2022-12-16T18:19:37.000Z","updated_at":"2024-06-05T07:11:38.000Z","dependencies_parsed_at":"2024-10-12T02:20:50.252Z","dependency_job_id":"d45f9655-60da-4994-b972-e3ecc926b581","html_url":"https://github.com/brentp/spacepile","commit_stats":null,"previous_names":[],"tags_count":3,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/brentp%2Fspacepile","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/brentp%2Fspacepile/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/brentp%2Fspacepile/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/brentp%2Fspacepile/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/brentp","download_url":"https://codeload.github.com/brentp/spacepile/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":238882444,"owners_count":19546519,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["deep-learning","genomics","sequence-alignment"],"created_at":"2024-10-12T02:20:44.851Z","updated_at":"2025-10-29T19:31:40.751Z","avatar_url":"https://github.com/brentp.png","language":"Rust","funding_links":[],"categories":[],"sub_categories":[],"readme":"# spacepile (WIP)\n\n# proposed usage\n\n```Python\nimport spacepile\n\n# spacing will be filled by `space` call.\nspacing = np.zeros((len(cigs), max_width), dtype=np.uint32)\nspacepile.space(spacing, cigs, posns)\n\n# now with label (truth)\nspacing = np.zeros((len(cigs) + 1, max_width), dtype=np.uint32)\nspacepile.space(spacing, cigs, posns, label_cigs, label_posn)\n# how to return mapping from spacing column to label? after removing label_only insertions?\n\n\n```\n\n`spacepile` efficiently creates (multi) alignments from replicate measurements of the same piece of DNA as\nfrom **duplex** sequencing, single molecule consensus, pac-bio **CCS**, or nanopore [**cyclomics**](https://cyclomics.com/).\n`spacepile` accepts alignments (cigars) of grouped sequences and _spaces_ them into matricies that can then be sent to any deep-learner.\n\nNote that this is done nicely in [deepconsensus](https://github.com/google/deepconsensus) which is a major source\nof inspiration for this work, but that is specific to PacBio and (likely) less efficient at extraction.\n\n`spacepile` **will** also provide means to facilitate training (agnostic to the learner) by providing tools to\nsample and augment reads and to separate training from test datasets.\n\nCurrently `spacepile` can be used by downloading and `pip install`ing the python wheel from the releases.\n\n```Python\nimport spacepile\nprint(spacepile)\nimport numpy as np\nimport pysam\n\n\n# make matricies of at most this width. usually this will be ~ 10% longer than read-length\nmax_width = 9\n# NOTE, these would likely be drawn from a bam file that is sorted by fragment (an option in samtools sort) or grouped by UMI\n# these are available as aln.cigartuples in pysam.\n# here we show 3 reads for an example.\ncigs = [\n        [(pysam.CMATCH, 4)],                                     # read 1. 4M     ACTG\n        [(pysam.CMATCH, 2), (pysam.CDEL, 2), (pysam.CMATCH, 1)], # read 2. 2M2D1M  CTC\n        [(pysam.CMATCH, 2), (pysam.CINS, 3), (pysam.CMATCH, 2)], # read 3. 2M3I2M ACGGGTG\n        ]\n\nposns = [0, 1, 0] # the 2nd read starts 1 base after the other reads.\n\n# idxs will be filled by `space` call.\nidxs = np.zeros((len(cigs), max_width), dtype=np.uint32)\nspacepile.space(idxs, cigs, posns)\n\n#  now we have the indexes into the original sequence or base-qualities or IPDs. etc.\nSPACE = np.iinfo(np.uint32).max - 1\nEND = np.iinfo(np.uint32).max\nassert np.array_equal(idxs,\n                [[  0,    1,SPACE,SPACE,SPACE,    2,    3,END  ,END],\n                 [END,    0,SPACE,SPACE,SPACE,    1,SPACE,SPACE,  2],\n                 [  0,    1,    2,    3,    4,    5,    6,END  ,END]])\n\n\n# sequences likely retrieved from pysam's aln.query_sequence or aln.query_alignment_sequence\nraw_sequences = [\"ACTG\", \"CTC\", \"ACGGGTG\"]\nbase_qs =       [[ 40, 50, 60, 70], [60, 60, 60], [45, 45, 45, 55, 55, 55, 60]]\n\nsequences = np.zeros((len(cigs), max_width), dtype=np.int16)\nbqs = np.zeros_like(sequences)\nfor i, s in enumerate(raw_sequences):\n    sequences[i, :len(s)] = np.array(list(s), dtype='U1').view(np.int32) # Q: is there a better way to do this in numpy?\n    bqs[i, :len(s)] = np.array(base_qs[i], dtype=np.int32)\n\n\n# NOTE we will put sequences and bqs in the same output matrix.\nmat = np.zeros((2 * len(cigs), max_width), dtype=np.int16)\nspacepile.translate(idxs, sequences, mat[:len(cigs)])\n\n# NOTE how the columns generally match\nassert np.array_equal(mat[:len(cigs)],\n                   [[65,67,-2,-2,-2,84,71,-1,-1],\n                    [-1,67,-2,-2,-2,84,-2,-2,67],\n                    [65,67,71,71,71,84,71,-1,-1]])\n\n\nspacepile.translate(idxs, bqs, mat[len(cigs):])\nassert np.array_equal(mat[len(cigs):],\n\t\t\t[[40,50,-2,-2,-2,60,70,-1,-1],\n\t\t\t [-1,60,-2,-2,-2,60,-2,-2,60],\n\t\t\t [45,45,45,55,55,55,60,-1,-1]])\n\n# now repeat for many sequences and send many mats to a learner.\n```\n\n# Research\n\n## steps\n\n### training\n\n#### read selection\n\nNeed enough unique reads from across genome to train. Given duplex-seq is often deep,\nwe want to sample wisely so we're not training on the same position 10,000 times.\n\n#### calibration\n\n[Paper covering this topic](https://arxiv.org/abs/1706.04599)\n\nhttps://scikit-learn.org/stable/modules/calibration.html\nhttps://arxiv.org/abs/2210.00045 ( from google 9/30/2022)\n\nNOTE: this one seems best. Just get pytorch model into sklearn-like class.\nhttps://github.com/dirichletcal/dirichlet_python\n\nSee this python module: https://github.com/classifier-calibration/PyCalib\n\nThe output from the model is not calibrated -- the error rates do not match the probabilities.\nThis is a known problem.\nThere is some work on this here:\nhttps://github.com/gpleiss/temperature_scaling (needs logits, not softmax)\n\nhttps://github.com/dirichletcal/experiments_dnn\n\nBut this may require further work because of the positional info where the ends of reads\noften have lower quality.\n\n### spacing\n\nThis implements the multi-alignment spacing which takes a set of reads and returns\nsomething that is sent to the network. Currently this is implemented in python.\nA rust version is in progress.\n\nThis involves clustering (by tag) and selection to max_depth so we can have a fixed\nnetwork size.\n\n### consensus\n\nTake a trained model and a bam and write a new fastq with well-calibrated base-qualities.\n\n## build notes\n\n```\ndocker run --rm -v $(pwd):/io ghcr.io/pyo3/maturin build --release\npip install target/wheels/spacepile-0.1.0-cp37-abi3-linux_x86_64.whl\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fbrentp%2Fspacepile","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fbrentp%2Fspacepile","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fbrentp%2Fspacepile/lists"}