https://github.com/ncar/cesm-lens-aws
Examples of analysis of CESM LENS data publicly available on Amazon S3 (us-west-2 region) using xarray and dask
https://github.com/ncar/cesm-lens-aws
aws binder cesm-lens dask intake pangeo python xarray zarr
Last synced: 7 months ago
JSON representation
Examples of analysis of CESM LENS data publicly available on Amazon S3 (us-west-2 region) using xarray and dask
- Host: GitHub
- URL: https://github.com/ncar/cesm-lens-aws
- Owner: NCAR
- License: bsd-3-clause
- Created: 2019-08-27T20:54:20.000Z (about 6 years ago)
- Default Branch: main
- Last Pushed: 2024-02-28T19:37:22.000Z (over 1 year ago)
- Last Synced: 2025-03-25T01:51:08.104Z (7 months ago)
- Topics: aws, binder, cesm-lens, dask, intake, pangeo, python, xarray, zarr
- Language: Jupyter Notebook
- Homepage: https://doi.org/10.26024/wt24-5j82
- Size: 20.8 MB
- Stars: 45
- Watchers: 13
- Forks: 23
- Open Issues: 9
-
Metadata Files:
- Readme: README.md
- License: LICENSE
Awesome Lists containing this project
README
# CESM LENS on AWS
- [CESM LENS on AWS](#cesm-lens-on-aws)
- [Re-create notebooks with Pangeo Binder](#re-create-notebooks-with-pangeo-binder)
- [ESM catalog](#esm-catalog)
- [Requirements](#requirements)
- [Examples](#examples)
- [Reference Documentation](#reference-documentation)
- [Source Code for CESM LENS on AWS Site](#source-code-for-cesm-lens-on-aws-site)
Examples of analysis of [CESM LENS data](https://registry.opendata.aws/ncar-cesm-lens/) publicly available on Amazon S3 (us-west-2 region) using xarray and dask.
## Re-create notebooks with Pangeo Binder
Try these notebooks on Pangeo Binder. Note that
the session is ephemeral. **Your home directory will not persist, so remember to download your notebooks if you made changes that you need to use at a later time!**
[](https://aws-uswest2-binder.pangeo.io/v2/gh/NCAR/cesm-lens-aws/binder-config?urlpath=git-pull?repo=https://github.com/NCAR/cesm-lens-aws%26amp%3Bbranch=main%26amp%3Burlpath=lab/tree/cesm-lens-aws/%3Fautodecode)
## ESM catalog
The main catalog URL is:
https://raw.githubusercontent.com/NCAR/cesm-lens-aws/main/intake-catalogs/aws-cesm1-le.json
This catalog is an [ESM collection](https://github.com/NCAR/esm-collection-spec) catalog. The data is stored in [Zarr](https://github.com/zarr-developers/zarr) format and meant to be opened with [Xarray](http://xarray.pydata.org/en/latest/).
## Requirements
Using this catalog requires the following package versions:
- [Intake-esm](https://github.com/intake/intake-esm) >= `v2020.11.4`
## Examples
To open the catalog and load a data set from Python, you can run the following code:
```python
In [1]: import intake
In [2]: col = intake.open_esm_datastore("https://raw.githubusercontent.com/NCAR/cesm-lens-aws/main/intake-catalogs/aws-cesm1-le.json")
In [3]: col
Out[3]:
In [4]: col.df.head()
Out[4]:
component frequency experiment ... dim_per_tstep start end
0 atm daily CTRL ... 2.0 0402-01-01 12:00:00 2200-12-31 12:00:00
1 atm daily CTRL ... 2.0 0402-01-01 12:00:00 2200-12-31 12:00:00
2 atm daily CTRL ... 2.0 0402-01-01 12:00:00 2200-12-31 12:00:00
3 atm daily CTRL ... 2.0 0402-01-01 12:00:00 2200-12-31 12:00:00
4 atm daily CTRL ... 2.0 0402-01-01 12:00:00 2200-12-31 12:00:00
[5 rows x 9 columns]
In [5]: col_subset = col.search(experiment="RCP85", frequency="monthly", variable=["hi", "aice"])
In [6]: dsets = col_subset.to_dataset_dict(zarr_kwargs={"consolidated": True}, storage_options={"anon": True})
--> The keys in the returned dictionary of datasets are constructed as follows:
'component.experiment.frequency'
|████████████████████████████████████████████████████████████████████████████████████████████████████| 100.00% [2/2 00:00<00:00]
In [7]: dsets.keys()
Out[7]: dict_keys(['ice_sh.RCP85.monthly', 'ice_nh.RCP85.monthly'])
In [8]: ds = dsets['ice_sh.RCP85.monthly']
In [9]: ds
Out[9]:
Dimensions: (d2: 2, member_id: 40, ni: 320, nj: 76, time: 1140)
Coordinates:
* member_id (member_id) int64 1 2 3 4 5 6 7 8 ... 34 35 101 102 103 104 105
* time (time) object 2006-01-16 12:00:00 ... 2100-12-16 12:00:00
time_bounds (time, d2) object dask.array
Dimensions without coordinates: d2, ni, nj
Data variables:
aice (member_id, time, nj, ni) float32 dask.array
hi (member_id, time, nj, ni) float32 dask.array
Attributes:
comment3: seconds elapsed into model date: 0
conventions: CF-1.0
nco_openmp_thread_number: 1
source: sea ice model: Community Ice Code (CICE)
NCO: 4.3.4
contents: Diagnostic and Prognostic Variables
comment2: File written on model date 20060201
comment: All years have exactly 365 days
intake_esm_dataset_key: ice_sh.RCP85.monthly
```
## Reference Documentation
- For details about intake-esm API, see the [reference documentation](https://intake-esm.readthedocs.io/en/latest)
- [CESM LENS on AWS Site](https://doi.org/10.26024/wt24-5j82)
## Source Code for CESM LENS on AWS Site
The source code for [https://doi.org/10.26024/wt24-5j82](https://doi.org/10.26024/wt24-5j82) resides in the [site directory](./site) of this repository.
The site is built with [sphinx](https://www.sphinx-doc.org/).
To build the site locally, please use [conda](https://docs.conda.io/) to set up a build environment with all dependencies.
First, make a local clone of this source repository on your machine. For example:
```bash
git clone https://github.com/NCAR/cesm-lens-aws
```
Set up your a conda environment:
```bash
conda env create -f site/environment.yml
conda activate cesm-lens-aws-site
bash site/install-extension.sh
```
You can then build the site with:
```bash
make live
```