{"id":18583359,"url":"https://github.com/alexpreynolds/cull-exemplars","last_synced_at":"2025-07-15T04:33:52.001Z","repository":{"id":75267193,"uuid":"426387977","full_name":"alexpreynolds/cull-exemplars","owner":"alexpreynolds","description":"Filter lower-scoring, neighboring genomic intervals within a specified distance","archived":false,"fork":false,"pushed_at":"2022-02-25T00:13:39.000Z","size":60843,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":2,"default_branch":"main","last_synced_at":"2025-05-16T05:08:23.990Z","etag":null,"topics":["bed","bedops","bioinformatics","interval","set","set-operations"],"latest_commit_sha":null,"homepage":"","language":"Jupyter Notebook","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/alexpreynolds.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2021-11-09T21:11:25.000Z","updated_at":"2022-01-28T18:01:44.000Z","dependencies_parsed_at":"2023-06-05T23:15:24.036Z","dependency_job_id":null,"html_url":"https://github.com/alexpreynolds/cull-exemplars","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/alexpreynolds/cull-exemplars","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/alexpreynolds%2Fcull-exemplars","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/alexpreynolds%2Fcull-exemplars/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/alexpreynolds%2Fcull-exemplars/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/alexpreynolds%2Fcull-exemplars/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/alexpreynolds","download_url":"https://codeload.github.com/alexpreynolds/cull-exemplars/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/alexpreynolds%2Fcull-exemplars/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":265402833,"owners_count":23759237,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["bed","bedops","bioinformatics","interval","set","set-operations"],"created_at":"2024-11-07T00:22:06.875Z","updated_at":"2025-07-15T04:33:51.976Z","avatar_url":"https://github.com/alexpreynolds.png","language":"Jupyter Notebook","funding_links":[],"categories":[],"sub_categories":[],"readme":"# cull-exemplars\nFilter lower-scoring, neighboring genomic intervals within a specified distance\n\n## Overview\n\nWe apply two methods to filter intervals and meet the following criteria:\n\n1. Find a good, potentially optimal distribution of scored intervals over the input set, which should be some specified distance away from each other (i.e., \"mutually compatible\").\n\n2. We would like both methods to prioritize high-scoring intervals.\n\n3. We would like 250k such elements out of the full superset, which meet the first two criteria.\n\n## Data\n\n### Input\n\nWe start with `data/intervals.txt.gz`, which contains ~4.5M intervals within `hg19` space.\n\nEach interval is a 200nt \"bin\" with a priority score in the fifth column (other columns are allowed and are ignored during processing).\n\nNot all of `hg19` is covered. All starting bins have a positive, non-zero score. We take advantage of this property to fill in gaps in `hg19` space with \"placeholder\" or \"sentinel\" bins, each with a zero score. This property will be exploited to know when the priority queue method is \"exhausted\", i.e., when there are no more valid bins to filter.\n\n### Output\n\nThe output from each of the two methods (discussed below) contains a filtered subset of the input bins, which meet the desired criteria.\n\n### Targets\n\nRunning the `assembly` and `prep` targets sets up files required for running the two filtering methods (`priority_queue` and `wis`).\n\n## Methods\n\n### Priority queue\n\nIn the `priority_queue` and `priority_queue_max_hits` targets, we put padded bins into a [priority queue](https://en.wikipedia.org/wiki/Priority_queue), ordered by score. \n\nWe pop the highest-scoring element off the queue, and keep it if it does not overlap any other popped elements. If it does, we still keep it, but all lower-scored overlaps within 2kb are marked as rejected.\n\nWe pop the next highest-scoring element and ask if it has already been rejected. If it hasn't, we keep it and we again mark any lower-scored overlaps as rejected, if there are any. If it was previously rejected, we skip it and keep popping and testing, until there are no elements left in the queue to pop-and-test, or until we hit some preselected number of intervals (250k).\n\nIntervals we keep are written to standard output.\n\nIn the `priority_queue_max_hits` target, we specify a larger `k` value of 4453117 (~4.5M). This is the number of intervals we started with and therefore the maximum number we can get back from any method. In the optimal case, all intervals would be 2kb or more away from each other, but in reality there are overlaps and so some fraction of these will get filtered. In this case, we will therefore eventually pop an element off the queue that has a score of zero. This is an artificial \"placeholder\" or sentinel bin, so we cease testing at this point and write whatever intervals we have found to standard output.\n\n### Weighted-interval scheduling\n\nIn the `wis` target, we use a [dynamic programming](https://en.wikipedia.org/wiki/Interval_scheduling#Weighted) approach to trace a path of high-scoring elements which do not overlap one another within 2kb.\n\nThis locally optimal path contains elements which do not overlap within 2kb. If we have more than 250k such elements in this path, we use a priority queue to pull out the top 250k such elements.\n\nIntervals retrieved via this path are written to standard output.\n\n### Validation\n\nThe `priority_queue_validate_no_overlaps` target tests that the outputted bins from the `priority_queue` method are not within 2kb of each other. Likewise, the `wis_validate_no_overlaps` target tests that the bins from the `wis` method are not within 2kb of each other.\n\n## Results\n\nHere is a summary of the baseline score distribution of the input elements. This is obtained via the `intervals_summary` target:\n\n```\n    Min.  1st Qu.   Median     Mean  3rd Qu.     Max. \n  0.3819   0.9664   2.2576   4.5391   5.3447 102.5106 \n```\n\nThe filtered subsets should meet or beat these values. \n\nHere is the score profile of `priority_queue` output, via the `priority_queue_summary` target:\n\n```\n   Min. 1st Qu.  Median    Mean 3rd Qu.    Max. \n  1.962   3.566   6.455  10.689  13.111 102.511 \n```\n\nAnd likewise for the output from the `wis` method, via the `wis_summary` target:\n\n```\n   Min. 1st Qu.  Median    Mean 3rd Qu.    Max. \n 0.3943  1.3949  3.9383  8.4296 10.4911 99.7358 \n```\n\nIn addition to giving a better score result, the `priority_queue` method ran much faster on this data than the `wis` method.\n\n### Priority queue (max-hits)\n\nIn the case of the `priority_queue_max_hits` target output, we get 395649 elements before we exhaust the queue. \n\nThese elements have the following score profile:\n\n```\n   Min. 1st Qu.  Median    Mean 3rd Qu.    Max. \n  0.382   1.206   3.236   7.119   8.480 102.511 \n```\n\nExhausting the queue performs worse than the `wis` method, score-wise, but this method returns more elements (~400k) as opposed to `wis`, which can return at most only ~251k elements when it is run to completion.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Falexpreynolds%2Fcull-exemplars","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Falexpreynolds%2Fcull-exemplars","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Falexpreynolds%2Fcull-exemplars/lists"}