{"id":26573272,"url":"https://github.com/linsalrob/sra_metadata","last_synced_at":"2025-06-21T23:38:32.820Z","repository":{"id":65492833,"uuid":"201698336","full_name":"linsalrob/SRA_Metadata","owner":"linsalrob","description":"Get, parse, and extract information from the SRA metadata files","archived":false,"fork":false,"pushed_at":"2022-09-20T04:24:01.000Z","size":539,"stargazers_count":38,"open_issues_count":2,"forks_count":5,"subscribers_count":2,"default_branch":"master","last_synced_at":"2025-04-08T21:32:41.562Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/linsalrob.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2019-08-11T00:35:39.000Z","updated_at":"2025-01-21T19:57:19.000Z","dependencies_parsed_at":"2023-01-26T02:31:54.169Z","dependency_job_id":null,"html_url":"https://github.com/linsalrob/SRA_Metadata","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/linsalrob/SRA_Metadata","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/linsalrob%2FSRA_Metadata","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/linsalrob%2FSRA_Metadata/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/linsalrob%2FSRA_Metadata/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/linsalrob%2FSRA_Metadata/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/linsalrob","download_url":"https://codeload.github.com/linsalrob/SRA_Metadata/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/linsalrob%2FSRA_Metadata/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":261211728,"owners_count":23125546,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2025-03-23T00:39:59.821Z","updated_at":"2025-06-21T23:38:27.801Z","avatar_url":"https://github.com/linsalrob.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"[![Edwards Lab](https://img.shields.io/badge/Bioinformatics-EdwardsLab-03A9F4)](https://edwards.flinders.edu.au)\n# SRA_Metadata\nGet, parse, and extract information from the SRA metadata files\n\n## About the SRA metadata\n\nThe SRA contains over 1.5 million samples, and each sample contains lots of runs. The metadata is really key to understanding that data, but the metadata is difficult to organize and understand. Here we collate the metadata information available from the SRA to make it easier to search and find things.\n\n\n## See also\n\nYou might also look at our [collection of blog posts](https://edwards.flinders.edu.au/sra) about the SRA that explain the organization of the SRA data, and provide alternate mechanisms to download the data, and so on.\n\n# Downloading the SRA metadata\n\nThere are several components to the SRA data that we are going to download.\n\n## SRA_Accessions.tab\n\nThis tab separated file can be downloaded directly from the NCBI: [ftp://ftp.ncbi.nlm.nih.gov/sra/reports/Metadata/SRA_Accessions.tab](ftp://ftp.ncbi.nlm.nih.gov/sra/reports/Metadata/SRA_Accessions.tab). \n\nThis file lists all the submissions to the SRA, and lists every accession number associated with each submission. It tells you the status of the datasets.\n\nIt contains the following columns:\n\n* Accession\n* Submission\n* Status\n* Updated\n* Published\n* Received\n* Type\n* Center\n* Visibility\n* Alias\n* Experiment\n* Sample\n* Study\n* Loaded\n* Spots\n* Bases\n* Md5sum\n* BioSample\n* BioProject\n* ReplacedBy\n\nThe key columns here are Accession, Submission, and Status.\n\nThe data in this file is replicated. A single submission may occur mutliple times, represented once for each of the accessions associated with it.\n\nAt the time of writing there were 27,838,771 entries (lines) in that file. However, there are only 1,413,223 unique submission IDs. \n\nFrom those 1,413,223 unique submission IDs, the `Status` field reports\n\n* 1,290,528 live\n* 161,652 suppressed\n* 92,103 unpublished\n* 10 withdrawn\n\n(These numbers don't quite add up because there are some projects where the project maybe be live, but the runs or other parts of the data release may be suppressed or unpublished.)\n\n# XML Metadata\n\nThe XML metadata is available for download from [ftp://ftp.ncbi.nlm.nih.gov/sra/reports/Metadata/](ftp://ftp.ncbi.nlm.nih.gov/sra/reports/Metadata/). There are daily files, and then once per month, or so, there is a complete release. \n\nFor example, this file was downloaded:\n\n```bash\ncurl -o NCBI_SRA_Metadata_Full.tgz ftp://ftp.ncbi.nlm.nih.gov/sra/reports/Metadata/NCBI_SRA_Metadata_Full_20180205.tar.gz\n```\n\nWhen you extract these files, you will get 1,000,000+ directories! Each directory is a single submission, and contains several files describing the data. I extract these using a command like: \n\n```bash\nmkdir xml\ntar -C xml/ -zxf NCBI_SRA_Metadata_Full.tar.gz\n```\n\nThere are several [XML Schema Definition files](https://trace.ncbi.nlm.nih.gov/Traces/sra/sra.cgi?view=xml_schemas) that define the data sets. Currently, there are `.xsd` files for:\n\n1. SRA Common\n2. SRA Submission\n3. SRA Study\n4. SRA Sample\n5. SRA Experiment\n6. SRA Run\n7. SRA Analysis\n\n# Converting the XML files to JSON\n\nWe batch process the XML files and convert them to JSON, using [a Python script](xml2json/xml_dir2json_random.py). This code uses the XML Schema Definition files to validate the XML files, and then dumps a single file per submission in JSON format.\n\nThis version chooses a file at random from the XML directory, checks to see if it has already been processed, and if not, it processes it. This allows us to run the code in parallel (using the awesome [GNU parallel](https://www.gnu.org/software/parallel/) and process lots of XML files all at once. For example, to process this code using 30 different processors, we can do:\n\n```bash\necho \"xml_dir2json_random.py -s $HOME/SRA/SRAdb/XML/Schemas/ -d xml -o json -m srr_sra_ids.tsv\" \u003e ./run_xml.sh\nseq 1 30 | parallel ./run_xml.sh {}\n```\n\nThis command creates a directory called `json` with three subdirectories, one each for `SRA`, `ERA`, `DRA`. Within those three directories, there are directories for each run, starting with the first three numbers. We use this structure because (a) it mirrors the structure at NCBI and elsewhere, and (b) breaking up the files into multiple subdirectories is much better for your filesystem. There are over 1,000,000 files, and so it takes commands like `ls` a long time to read the [inodes](http://www.grymoire.com/Unix/Inodes.html). By splitting the files out, we can more readily access and process them.\n\n\n\u003e *Tip:* If you have an SRA ID such as `SRR=SRA889255` you can access the appropriate file with, for example, `ls json/${SRR:0:3}/${SRR:0:6}/$SRR.json`.\n\nThis command also creates an *id mapping* file called `srr_sra_ids.tsv` that has two columns, the SRA submission ID (or ERA/DBA ID) and the SRA Run ID. The most common association we are looking for is from SRR -\u003e SRA. For example, we usually know the SRR IDs associated with a sequence run, and would like to explore the metadata associated with that run. Alternatively, we know a sample we would like to get the DNA sequences associated with. This mapping provides that connection, and you can quickly look for either a run or a submission using `grep`.\n\nIn addition, we create a file called `XML_validation_errors.txt` that reports any improper XML data that does not match the XML Schema Defintions. \n\nWe now have a directory with all the metadata as json objects that you can analyze in different ways. \n\n# JSON\n\nWe have some [JSON](json/) parsing code to help you explore the data. Before you begin, however, take a look at the [json_examples](json_examples/) data directory. These are ten samples chosen completely at random from the August 2019 metadata to demonstrate the organization of the metadata there.\n\nI also recommend using [jq](https://stedolan.github.io/jq/) for processing the data on the command line.\n\nHere are a couple of examples from our [partie](https://github.com/linsalrob/partie) analysis of SRA datasets.\n\nFirst , find all the submissions that are metagenomes or microbiomes using grep. You could also do this with the XML files, there is nothing specific about this grep and json.\n\n```bash\negrep -rli 'metagenome|microbiome' json | perl -pe 's#json/##; s#.json##' \u003e metagenomes.txt \n```\n\nWe now have a file, called `metagenomes.txt` that has one SRA submission per line where somewhere in the file it has the words `metagenome` or `microbiome`.\n\nNow we can use [jq](https://stedolan.github.io/jq/) to extract just the run identifiers from these files:\n\n```bash\ncat metagenomes.txt | xargs -i jq -r \"try .RUN[].IDENTIFIERS.PRIMARY_ID\" json/{}.json \u003e metagenome_runs.txt\n```\n\nIn this command, we cat the file of IDs, and for each file, we use `jq` to parse the json data. We look for any `RUN` and from that pull the `IDENTIFIERS` entry, and then the `PRIMARY_ID` for that run. This prints out one `PRIMARY_ID` per line. The `try` in that command is a jq option that is basic error handling. We could add both a `try` and a `catch`, and use that to report on any JSON files that do not have a RUN associated with them, however, at the moment we don't care about those ... we just ignore them!\n\nI don't know how to succinctly parse the XML to get this information (though you could probably do it with `grep`).\n\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Flinsalrob%2Fsra_metadata","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Flinsalrob%2Fsra_metadata","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Flinsalrob%2Fsra_metadata/lists"}