{"id":16755251,"url":"https://github.com/luizirber/2017-recomb","last_synced_at":"2025-04-05T07:09:37.719Z","repository":{"id":66476502,"uuid":"89943716","full_name":"luizirber/2017-recomb","owner":"luizirber","description":"Poster presented at RECOMB 2017","archived":false,"fork":false,"pushed_at":"2018-06-26T15:40:53.000Z","size":2786,"stargazers_count":7,"open_issues_count":0,"forks_count":3,"subscribers_count":2,"default_branch":"master","last_synced_at":"2025-04-02T12:43:04.112Z","etag":null,"topics":["data-sharing","databases","decentralized","minhash","persistent-data-structures","sequence-bloom-trees","similarity","sourmash"],"latest_commit_sha":null,"homepage":"","language":null,"has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"bsd-3-clause","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/luizirber.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null}},"created_at":"2017-05-01T17:02:32.000Z","updated_at":"2023-03-02T16:45:26.000Z","dependencies_parsed_at":"2024-01-18T03:46:48.781Z","dependency_job_id":null,"html_url":"https://github.com/luizirber/2017-recomb","commit_stats":null,"previous_names":[],"tags_count":1,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/luizirber%2F2017-recomb","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/luizirber%2F2017-recomb/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/luizirber%2F2017-recomb/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/luizirber%2F2017-recomb/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/luizirber","download_url":"https://codeload.github.com/luizirber/2017-recomb/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":247299843,"owners_count":20916192,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["data-sharing","databases","decentralized","minhash","persistent-data-structures","sequence-bloom-trees","similarity","sourmash"],"created_at":"2024-10-13T03:07:10.736Z","updated_at":"2025-04-05T07:09:37.690Z","avatar_url":"https://github.com/luizirber.png","language":null,"funding_links":[],"categories":["文章"],"sub_categories":[],"readme":"# Decentralized indexes for public genomic data\n\n[Luiz Carlos Irber Júnior](https://github.com/luizirber), [C. Titus Brown](https://github.com/ctb), [Tim Head](https://github.com/betatim)\n\n- Department of Population Health and Reproduction, University of California, Davis, USA\n- Head's Wild Tree Tech, Switzerland\n\nPoster presented at [RECOMB 2017][1].\n\n## Abstract\n\nMinHashes can be used to estimate the similarity of two or more datasets.\nExpanding on the work pioneered by mash and extended in our library Sourmash,\nwe calculated signatures for 412 thousand microbial reads datasets on the Sequence Read Archive.\nTo be able to efficiently search for matches of these signatures in the RefSeq microbial genomes database we developed a new data structure based on Sequence Bloom Trees adapted for searching MinHash signatures (named SBTMH) and made it available to whoever wants to use it.\n\nWe explore how to encode the SBTMH structure as objects in a MerkleDAG and store it in IPFS (InterPlanetary File System),\na decentralized system for data sharing, and how to load and spread the SBTMH indexes as well as the signatures calculated.\nThe SBTMH behaves like a persistent data structure,\nwhere updates and new nodes can share parts of the structure of previous versions of the SBTMH.\nWhile this property can be used to avoid duplicating data,\nin this case it is important because it allows common nodes in trees to be shared,\nleading to increased availability and facilitating sharing and remixing of indexes and signatures.\n\nThis design can be extended to change how databases and archives (like the SRA) are offered and implemented,\nsince users can collaborate by choosing to share subsets of the archive and spread the network bandwidth.\nMore importantly,\nit avoids the central point of failure,\nwhile still allowing for curation and quality assurance of the data.\nWe present a prototype showing how this can be achieved.\n\n## Table of Contents\n\n- [Introduction](01.intro.md)\n- [Saving a SBTMH in IPFS](02.saving.md)\n- [Sharing datasets in IPFS](03.sharing.md)\n- [Future work](04.future.md)\n- [References](#references)\n- Appendices\n  - [Submitted abstract](abstract.md)\n  - [The final poster](poster/poster.pdf)\n\n## References\n\n- Benet, Juan. 2014. “IPFS - Content Addressed, Versioned, P2P File System.” arXiv:1407.3561 [Cs], July. http://arxiv.org/abs/1407.3561.\n- Broder, Andrei Z. 1997. “On the Resemblance and Containment of Documents.” - In Compression and Complexity of Sequences 1997. Proceedings, 21–29. IEEE. http://ieeexplore.ieee.org/abstract/document/666900/.\n- GB Editorial Team. 2011. “Closure of the NCBI SRA and Implications for the Long-Term Future of Genomics Data Storage.” Genome Biology 12: 402. doi:10.1186/gb-2011-12-3-402. https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3129670/\n- Driscoll, James R., Neil Sarnak, Daniel D. Sleator, and Robert E. Tarjan. 1989. “Making Data Structures Persistent.” Journal of Computer and System Sciences 38 (1): 86–124. https://dx.doi.org/10.1016/0022-0000(89)90034-2\n- Ondov, Brian D., Todd J. Treangen, Páll Melsted, Adam B. Mallonee, Nicholas H. Bergman, Sergey Koren, and Adam M. Phillippy. 2016. “Mash: Fast Genome and Metagenome Distance Estimation Using MinHash.” Genome Biology 17: 132. https://dx.doi.org/10.1186/s13059-016-0997-x\n- Solomon, Brad, and Carl Kingsford. 2016. “Fast Search of Thousands of Short-Read Sequencing Experiments.” Nature Biotechnology 34 (3): 300–302. https://dx.doi.org/10.1038/nbt.3442\n- Titus Brown, C., and Luiz Irber. 2016. “sourmash: A Library for MinHash Sketching of DNA.” The Journal of Open Source Software 1 (5). https://dx.doi.org/10.21105/joss.00027\n\n\n[1]: http://cb.csail.mit.edu/cb/recomb2017/\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fluizirber%2F2017-recomb","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fluizirber%2F2017-recomb","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fluizirber%2F2017-recomb/lists"}