https://github.com/luizirber/2017-jgi-download
Downloading files from JGI using the API
https://github.com/luizirber/2017-jgi-download
Last synced: 8 months ago
JSON representation
Downloading files from JGI using the API
- Host: GitHub
- URL: https://github.com/luizirber/2017-jgi-download
- Owner: luizirber
- Created: 2017-03-27T23:14:10.000Z (over 8 years ago)
- Default Branch: master
- Last Pushed: 2018-02-22T20:52:44.000Z (almost 8 years ago)
- Last Synced: 2025-03-15T14:56:43.913Z (9 months ago)
- Language: Python
- Size: 1.95 KB
- Stars: 0
- Watchers: 2
- Forks: 1
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
Awesome Lists containing this project
README
# Downloading files from JGI
This is an example for downloading assembled (and unmasked) fungi genomes from [MycoCosm][0].
It uses the [API][1] to list files,
and [Snakemake][2] to manage the interaction with the API.
## Steps
- Set up JGI login and password in the Snakefile
- `pip install snakemake lxml`
- `snakemake -k -j32` to use 32 threads to download files. `-k` means "don't stop on errors",
which is useful because some datasets might be private.
## TODO
- need to run twice: one for generating `urls.txt`,
and another for the proper download
(Comment out `url_mapping` in the rule `all` in the first run).
Need to figure out how to make snakemake manage that (probably [dynamic rules][3]?)
[0]: http://genome.jgi.doe.gov/programs/fungi/index.jsf
[1]: http://genome.jgi.doe.gov/help/download.jsf#api
[2]: https://snakemake.readthedocs.io/en/stable/
[3]: http://snakemake.readthedocs.io/en/stable/snakefiles/rules.html#dynamic-files