{"id":26573246,"url":"https://github.com/linsalrob/searchsra","last_synced_at":"2025-03-23T00:39:55.832Z","repository":{"id":146768200,"uuid":"90091634","full_name":"linsalrob/SearchSRA","owner":"linsalrob","description":"Tools to search through the Sequence Read Archive using XSEDE's Jetstream","archived":false,"fork":false,"pushed_at":"2021-05-17T07:14:06.000Z","size":17,"stargazers_count":2,"open_issues_count":2,"forks_count":3,"subscribers_count":2,"default_branch":"master","last_synced_at":"2023-10-20T21:17:26.300Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"http://edwards.sdsu.edu/research/SRA","language":"Shell","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/linsalrob.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null}},"created_at":"2017-05-03T00:38:43.000Z","updated_at":"2023-10-20T21:17:27.728Z","dependencies_parsed_at":null,"dependency_job_id":"79b2f275-585e-4244-ba4a-8e3c1888de3a","html_url":"https://github.com/linsalrob/SearchSRA","commit_stats":null,"previous_names":[],"tags_count":1,"template":null,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/linsalrob%2FSearchSRA","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/linsalrob%2FSearchSRA/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/linsalrob%2FSearchSRA/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/linsalrob%2FSearchSRA/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/linsalrob","download_url":"https://codeload.github.com/linsalrob/SearchSRA/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":245040189,"owners_count":20551299,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2025-03-23T00:39:55.154Z","updated_at":"2025-03-23T00:39:55.813Z","avatar_url":"https://github.com/linsalrob.png","language":"Shell","funding_links":[],"categories":[],"sub_categories":[],"readme":"[![Edwards Lab](https://img.shields.io/badge/Bioinformatics-EdwardsLab-03A9F4)](https://edwards.sdsu.edu/research)\n# Searching the SRA\n\nWe're using the [Jetstream cloud](https://use.jetstream-cloud.org/) to search the [Sequence Read Archive](https://www.ncbi.nlm.nih.gov/sra/). \n\nThis repository contains some of the scripts that we use and introductions to get started. It is not a comprehensive repository as we are still in a pre-alpha development mode with the code. I'm looking forward to deleting this line at some point in the future.\n\n## Citing this repository\n\nYou can access this repository at [Zeonodo](http://www.zenodo.org/) with DOI:10.5281/zenodo.1043562 [![DOI](https://www.zenodo.org/badge/90091634.svg)](https://www.zenodo.org/badge/latestdoi/90091634)\n\n\n## Getting Started\n\nBefore you can start using this code you will need an [XSEDE](https://www.xsede.org/web/site/for-users/getting-started) account and an allocation that allows you to use JetStream. If you are interested in going down this road, shoot [Rob](https://edwards.sdsu.edu/research/) an email and he'll help you get going. \n\nDon't clone this repository until you get the allocation set up!\n\n\n## Preparing the data\n\nBefore you start, you will need:\n\n1. A fasta file with the sequences that you want to search. Mine is called `genomes.fasta`\n2. A list of SRA IDs that you want to search against. This easiest way to get this is to grep for WGS in [SRA_Metagenome_Types.tsv](https://raw.githubusercontent.com/linsalrob/partie/master/SRA_Metagenome_Types.tsv) from [PARTIE](https://github.com/linsalrob/partie). Currently this list is ~67,429 SRA files. Note that for this code, we just want the ID, and not the WGS part, so you may need to use: `grep WGS SRA_Metagenome_Types.tsv | cut -f 1` to get just the ids.\n3. A website where you can house the data.\n4. The setup.sh, compare2sra.sh, and search_sra.sh from here.\n\nWe start by splitting the SRA ids file into as many files as you have Jetstream instances. For example, if you have 67,429 lines in your file and 10 instances, I would use: `split -l 6743 -d sra.ids`. This will give you 10 files x00, x01, x02, ... x09. \n\nPut those files and the fasta file to search on a website somewhere and make sure that they are publicly available.\n\nFor example, [here is an example data set I searched](https://edwards.sdsu.edu/redwards/SRA/genome_search/). \n\nThe reason that we do this is that each instance will download a file and process it.\n\nPlease edit the search_sra.sh script and add that URL at line 10.\n\n## Getting started searching the SRA\n\nNow we have the data we need to get our jetstream instances ready. \n\nFirst, you need one or more Jetstram instances. I usually fire up 10 or 12 instances. On my to do list is work with the Jetstream API to burst a bunch of images at once, but I typically do this using the web interface above.\n\nCreate a list of your IPs separated by spaces. I also reserve one IP to test everything out. Once it runs for that IP everything should be good to go. I have also had issues where I complete break the test machine, and so I just delete it and fire up a new machine and everything works (this is usually while testing the setup.sh script!).\n\n## The process to search SRA\n\nTo search the SRA we are going to:\n0. copy the scripts over\n1. install the software\n2. run the search in the background using screen\n\nWe can test the code on a single instance. Change the IP address to the actual IP address.\n\n```\nIP=149.165.0.0 \nscp compare2sra.sh search_sra.sh setup.sh $IP:\nssh $IP \"~/setup.sh\"\nssh $IP \"~/search_sra.sh genomes.fasta x00\"\n```\n\nNote that in the last line: genomes.fasta is the name of my fasta file (1, above). x00 is the name of the file from the split command.\n\nIf that completes successfully we can run the code for the other nodes. I have a file that I call `ips.txt` which has one IP address per line.\n\n```\nC=0; \nfor IP in $(cat ips.txt)\ndo\n\tC=$((C+1));\n\tscp compare2sra.sh search_sra.sh setup.sh $IP:\n\tssh $IP \"~/setup.sh\"\n\tssh $IP \"~/search_sra.sh x0$C\"\ndone\n```\n\n# Get the results.\n\nOnce this is run, we just need to get the results. This code will access all the machines and synchronize the results into the search folder. You can run this as often as you like.\n\n```\nfor IP in $(cat ips.txt)\ndo \n\tOUT=$(rsync -avz $IP:search/ search/);\n\tCOUNT=$(echo $OUT| sed -e 's/bam /bam\\n/g' | wc -l)\n\techo $IP  \" :: \"  $((COUNT-1)) \" FILES\"\ndone\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Flinsalrob%2Fsearchsra","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Flinsalrob%2Fsearchsra","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Flinsalrob%2Fsearchsra/lists"}