https://github.com/esgf/esgf-download
ESGF data transfer and replication tool
https://github.com/esgf/esgf-download
Last synced: 4 months ago
JSON representation
ESGF data transfer and replication tool
- Host: GitHub
- URL: https://github.com/esgf/esgf-download
- Owner: ESGF
- License: bsd-3-clause
- Created: 2023-03-14T14:10:16.000Z (about 3 years ago)
- Default Branch: main
- Last Pushed: 2025-12-01T17:32:18.000Z (6 months ago)
- Last Synced: 2025-12-04T06:53:29.011Z (6 months ago)
- Language: Python
- Homepage: https://esgf.github.io/esgf-download/
- Size: 2.07 MB
- Stars: 20
- Watchers: 7
- Forks: 8
- Open Issues: 20
-
Metadata Files:
- Readme: README.md
- License: LICENSE
- Citation: CITATION.cff
Awesome Lists containing this project
README
# esgpull - ESGF data management utility
[](https://github.com/astral-sh/uv)
`esgpull` is a tool that simplifies usage of the [ESGF Search API](https://esgf.github.io/esg-search/ESGF_Search_RESTful_API.html) for data discovery, and manages procedures related to downloading and storing files from ESGF.
```py
from esgpull import Esgpull, Query
query = Query()
query.selection.project = "CMIP6"
query.options.distrib = True # default=False
esg = Esgpull()
nb_datasets = esg.context.hits(query, file=False)[0]
nb_files = esg.context.hits(query, file=True)[0]
datasets = esg.context.datasets(query, max_hits=5)
print(f"Number of CMIP6 datasets: {nb_datasets}")
print(f"Number of CMIP6 files: {nb_files}")
for dataset in datasets:
print(dataset)
```
## Features
- Command-line interface
- HTTP download (async multi-file)
## Installation
`esgpull` is distributed via PyPI:
```shell
pip install esgpull
esgpull --help
```
For isolated installation, [`uv`](https://github.com/astral-sh/uv) or
[`pipx`](https://github.com/pypa/pipx) are recommended:
```shell
# with uv
uv tool install esgpull
esgpull --help
# alternatively, uvx enables running without explicit installation (comes with uv)
uvx esgpull --help
```
```shell
# with pipx
pipx install esgpull
esgpull --help
```
## Usage
```console
Usage: esgpull [OPTIONS] COMMAND [ARGS]...
esgpull is a management utility for files and datasets from ESGF.
Options:
-V, --version Show the version and exit.
-h, --help Show this message and exit.
Commands:
add Add queries to the database
config View/modify config
convert Convert synda selection files to esgpull queries
download Asynchronously download files linked to queries
login OpenID authentication and certificates renewal
remove Remove queries from the database
retry Re-queue failed and cancelled downloads
search Search datasets and files on ESGF
self Manage esgpull installations / import synda database
show View query tree
status View file queue status
track Track queries
untrack Untrack queries
update Fetch files, link files <-> queries, send files to download...
```
## Useful links
* [ESGF Webinar: An Introduction to esgpull, A Replacement for Synda](https://www.youtube.com/watch?v=xv2RVMd1iCA)
## Contributions
You can use the common github workflow (through pull requests and issues) to contribute.