{"id":22752426,"url":"https://github.com/perrette/iisd-cdstoolbox","last_synced_at":"2025-07-09T05:04:50.268Z","repository":{"id":144977931,"uuid":"252131361","full_name":"perrette/iisd-cdstoolbox","owner":"perrette","description":"Code related to the CDS Toolbox and CDS API","archived":false,"fork":false,"pushed_at":"2023-02-18T08:51:08.000Z","size":480,"stargazers_count":9,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"master","last_synced_at":"2025-04-14T14:15:24.420Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/perrette.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2020-04-01T09:34:10.000Z","updated_at":"2024-07-02T11:52:50.000Z","dependencies_parsed_at":null,"dependency_job_id":"500597bf-3149-4fda-b574-b4b77e51553b","html_url":"https://github.com/perrette/iisd-cdstoolbox","commit_stats":null,"previous_names":[],"tags_count":9,"template":false,"template_full_name":null,"purl":"pkg:github/perrette/iisd-cdstoolbox","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/perrette%2Fiisd-cdstoolbox","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/perrette%2Fiisd-cdstoolbox/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/perrette%2Fiisd-cdstoolbox/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/perrette%2Fiisd-cdstoolbox/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/perrette","download_url":"https://codeload.github.com/perrette/iisd-cdstoolbox/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/perrette%2Fiisd-cdstoolbox/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":264396623,"owners_count":23601541,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-12-11T05:11:43.335Z","updated_at":"2025-07-09T05:04:50.262Z","avatar_url":"https://github.com/perrette.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# iisd-cdstoolbox\n\nCode related to offline processing of CDS Toolbox and CDS API data for the C3S_428h_IISD-EU project.\n\n## How does this code relate to the CDS API ?\n\nThis code builds on the powerful CDS API but focuses on local impact analysis specific for the C3S_428h_IISD-EU project. It makes it easier to retrieve a timeseries for a specific location or region, and save the result to a CSV file (a simpler format than netCDF for most climate adaptation practitioners). Additionally, the code combines variables across multiple datasets, aggregate them into asset classes (such as all energy-related variables) and perform actions such as bias correction (use of ERA5 and CMIP5).\n\n## Download this code\n\nThe easy way is to download the zipped archive:\n- latest (development): https://github.com/perrette/iisd-cdstoolbox/archive/master.zip\n- or check stable releases with description of changes: https://github.com/perrette/iisd-cdstoolbox/releases (see assets at the bottom of each release to download a zip version)\n\nThe hacky way is to use git (only useful during development, for frequent updates, to avoid having to download and extract the archive every time):\n- First time: `git clone https://github.com/perrette/iisd-cdstoolbox.git`\n- Subsequent updates: `git pull` from inside the repository \n\n## Installation steps\n\n- Download the code (see above) and inside the folder.\n- Install Python 3, ideally Anaconda Python which comes with pre-installed packages\n- Install the CDS API key: https://cds.climate.copernicus.eu/api-how-to\n- Install the CDS API client: `pip install cdsapi`\n- Install other [dependencies](requirements.txt): `conda install --file requirements.txt` or `pip install -r requirements.txt`\n- _Optional_ dependency for coastlines on plots: `conda install -c conda-forge cartopy` or see [docs](https://scitools.org.uk/cartopy/docs/latest/installing.html)\n- _Optional_ dependency: CDO (might be needed later, experimental): `conda install -c conda-forge python-cdo`\n\n\nTroubleshooting:\n- If install fails, you may need to go through the dependencies in `requirements.txt` one by one and try either `pip install` or `conda install` or other methods specific to that dependency.\n- In the examples that follow, if you have both python2 and python3 installed, you might need to replace `python` with `python3`.\n\n## cds api\n\nDownload indicators associated with one asset class.\n\n**Examples of use**:\n\n    python download.py --asset energy --location Welkenraedt \n    \nThe corresponding csv timeseries will be stored in `indicators/welkenraedt/energy`. Note that raw downloaded data from the CDS API (regional tiles in netcdf format, and csv for the required lon/lat, without any correction) are stored under `download/` and can be re-used across multiple indicators.\n\nThe `indicators` folder is organized by location, asset class, simulation set and indicator name. The aim is to provide multiple sets for Savi simulation. For instance, `era5` for past simulations, and various `cmip5` versions for future simulations, that may vary with model and experiment. For instance the above command creates the folder structure (here a subset of all variables is shown):\n\n\tindicators/\n\t  welkenraedt/\n\t    energy/\n\t\t  era5/\n\t\t\t2m_temperature.csv\n\t\t\tprecipitation.csv\n\t\t\t...\n\t\t  cmip5-ipsl_cm5a_mr-rcp_8_5/\n\t\t\t2m_temperature.csv\n\t\t\tprecipitation.csv\n\t\t\t...\n\t\t  ...\n\nwith two simulation sets `era5` and `cmip5-ipsl_cm5a_mr-rcp_8_5`. It is possible to specify other models and experiment via `--model`  and `--experiment` parameters, to add futher simulation sets and thus test how the choice of climate models and experiment affect the result of Savi simulations.\n\nCompared to raw CDS API, some variables are renamed and scaled so that units match and are the same across simulation sets.\nFor instance, temperature was adjusted from Kelvin to degree Celsius, and precipitation was renamed and units-adjusted into mm per month from original (mean_total_precipitation_rate (mm/s) in ERA5, and mean_precipitation_flux (mm/s) in CMIP5). Additionally, CMIP5 data is corrected so that climatological mean matches with ERA5 data (climatology computed over 1979-2019 by default). \n\nAdditionally to the files shown in the example folder listing above, figures can also be created for rapid control of the data, either for interactive viewing (`--view-timeseries` and `--view-region`) or or saved as PNG files (`--png-timeseries` and `--png-region`), e.g.\n\n\tpython download.py --asset energy --location Welkenraedt --png-timeseries --png-region\n\nSingle indicators can be downloaded via:\n\n\tpython download.py --indicator 2m_temperature --location Welkenraedt\n\nThe choices available for `--indicator` , `--asset` and `--location` area defined in the following configuration files, respectively:\n\n- controls which indicators are available, how they are renamed and unit-adjusted: [indicators.yml](indicators.yml) (see [sub-section](#indicator-definition) below)\n- controls the indicator list in each asset class: [assets.yml](assets.yml)\n- controls the list of locations available: [locations.yml](locations.yml)\n\nFull documentation, including fine-grained controls, is provided in the command-line help:\n\n    python download.py --help\n\n\nVisit the CDS Datasets download pages, for more information about available variables, models and scenarios:\n- ERA5: https://cds.climate.copernicus.eu/cdsapp#!/dataset/reanalysis-era5-single-levels-monthly-means?tab=form\n- CMIP5: `https://cds.climate.copernicus.eu/cdsapp#!/dataset/projections-cmip5-monthly-single-levels?tab=form`\nIn particular, clicking on \"Show API request\" provides information about spelling of the parameters, e.g. that \"2m temperature\" is spelled `2m_temperature` and \"RCP 8.5\" is spelled `rcp_8_5`.\n\n\n\n### Indicator definition\n\nThis section is intended for users who wish to extend the list of indicators currently defined in [indicators.yml](indicators.yml).\nIt can be safely ignored for users who are only interested in using the existing indicators.\n\nLet's see how `10m_wind_speed` is defined:\n\n\t- name: 10m_wind_speed\n\t  units: m / s\n\t  description: Wind speed magnitude at 10 m\n\nThe fields `name` and `units` define the indicator. Description is optional, just to provide some context.\nIt is possible to provide `scale` and `offset` fieds to correct the data as `(data + offset) * scale`.\nHere for `2m temperature`:\n\n\t- name: 2m_temperature\n\t  units: degrees Celsius\n\t  description: 2-m air temperature\n      offset: -273.15  # Kelvin to degrees C\n\n`#` denotes a comment to provide some context. \nSome indicators have different names in ERA5 and CMIP5, and possibly different units. \nThat can be dealt with by providing `era5` and `cmip5` fields, which have precedence over the top-level fields. \nHere the `evaporation` definition:\n\n\t- name: evaporation\n\t  units: mm per month\n\t  era5:\n\t    name: mean_evaporation_rate  # different name in ERA5\n\t    scale: -2592000  # change sign and convert from mm/s to mm / month\n\t  cmip5:\n\t    scale: 2592000  # mm/s to mm / month\n\nIn that case both scaling and name depend on the dataset. In CMIP5 which variable name is identical to our indicator name, the `name` field can be omitted.\nIn ERA5, evaporation is negative (downwards fluxes are counted positively), whereas it is counted positively in ERA5.\n\nIndicators composed of several CDS variables can be defined via `compose` and `expression` fields.\nLet's look at `100m_wind_speed`:\n\n\t- name: 100m_wind_speed\n\t  units: m / s\n\t  description: Wind speed magnitude at 100 m\n\t  era5:\n\t    compose:\n\t      - 100m_u_component_of_wind\n\t      - 100m_v_component_of_wind\n\t    expression: (_100m_u_component_of_wind**2 + _100m_v_component_of_wind**2)**0.5\n\t  cmip5:\n\t    name: 10m_wind_speed\n\t    scale: 1.6  # average scaling from 10m to 100m, based on one test location (approximate!)\n\nIn ERA5, vector components of 100m wind speed are provided. \nOur indicator is therefore a composition of these two variables, defined by the `expression` field, which is evaluated as a python expression.\nNote that variables that start with a digit are not licit in python and must be prefixed with an underscore `_` in the `expression` field (only there).\n\nFor complex expressions, it is possible to provide a `mapping` field to store intermediate variables, for readability. This is used for the `relative_humidity` indicator:\n\n\t- name: relative_humidity\n\t  units: '%'\n\t  era5:\n\t    compose:\n\t      - 2m_temperature\n\t      - 2m_dewpoint_temperature\n\t    expression: 100*(exp((17.625*TD)/(243.04+TD))/exp((17.625*T)/(243.04+T)))\n\t    mapping: {T: _2m_temperature - 273.15, TD: _2m_dewpoint_temperature - 273.15}\n\t  cmip5:\n\t    name: near_surface_relative_humidity\n\nwhere `T` and `TD` are provided as intermediary variables, to be used in `expression`.\n\nERA5-hourly dataset can be retrieved via `frequency: hourly` field, and subsequently aggregated to monthly indicators \nthanks to pre-defined functions `daily_max`, `daily_min`, `daily_mean`, `monthly_mean`, `yearly_mean`. For instance:\n\n\t- name: maximum_monthly_temperature\n\t  units: degrees Celsius\n\t  offset: -273.15\n\t  cmip5:\n\t    name: maximum_2m_temperature_in_the_last_24_hours\n\t  era5:\n\t    name: 2m_temperature\n\t    frequency: hourly\n\t    transform: \n\t      - daily_max\n\t      - monthly_mean\n\nThis variable is available directly for CMIP5, but not in ERA5. It is calculated from `2m_temperature` from ERA5 `hourly` dataset, and subsequently aggregated.\nNote the ERA5-hourly dataset takes significantly longer to retrieve than ERA5 monthly. Consider using in combination with `--year 2000` to retrieve a single year of the ERA5 dataset.\n\nCurrently CMIP5 daily is not supported. \n\n## netcdf to csv\n\nConvert netcdf timeseries files downloaded from the CDS Toolbox pages into csv files (note : this does not work for netcdf files downloaded via the cds api):\n\n    python netcdf_to_csv.py data/*nc\n\nHelp:\n\n    python netcdf_to_csv.py --help\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fperrette%2Fiisd-cdstoolbox","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fperrette%2Fiisd-cdstoolbox","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fperrette%2Fiisd-cdstoolbox/lists"}