{"id":26056236,"url":"https://github.com/ncar/cesm-lens-aws","last_synced_at":"2025-04-11T03:50:23.278Z","repository":{"id":50218863,"uuid":"204788835","full_name":"NCAR/cesm-lens-aws","owner":"NCAR","description":"Examples of analysis of CESM LENS data publicly available on Amazon S3 (us-west-2 region) using xarray and dask","archived":false,"fork":false,"pushed_at":"2024-02-28T19:37:22.000Z","size":21767,"stargazers_count":45,"open_issues_count":9,"forks_count":23,"subscribers_count":13,"default_branch":"main","last_synced_at":"2025-03-25T01:51:08.104Z","etag":null,"topics":["aws","binder","cesm-lens","dask","intake","pangeo","python","xarray","zarr"],"latest_commit_sha":null,"homepage":"https://doi.org/10.26024/wt24-5j82","language":"Jupyter Notebook","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"bsd-3-clause","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/NCAR.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null}},"created_at":"2019-08-27T20:54:20.000Z","updated_at":"2025-03-04T13:49:39.000Z","dependencies_parsed_at":"2022-09-03T23:11:40.326Z","dependency_job_id":"691e7121-5b81-460a-9a45-b3dfccc20e1f","html_url":"https://github.com/NCAR/cesm-lens-aws","commit_stats":{"total_commits":113,"total_committers":7,"mean_commits":"16.142857142857142","dds":"0.49557522123893805","last_synced_commit":"182f206a73f7a64a1a07af02443afb4994384abf"},"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/NCAR%2Fcesm-lens-aws","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/NCAR%2Fcesm-lens-aws/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/NCAR%2Fcesm-lens-aws/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/NCAR%2Fcesm-lens-aws/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/NCAR","download_url":"https://codeload.github.com/NCAR/cesm-lens-aws/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248339262,"owners_count":21087214,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["aws","binder","cesm-lens","dask","intake","pangeo","python","xarray","zarr"],"created_at":"2025-03-08T10:53:03.884Z","updated_at":"2025-04-11T03:50:23.258Z","avatar_url":"https://github.com/NCAR.png","language":"Jupyter Notebook","funding_links":[],"categories":[],"sub_categories":[],"readme":"![GitHub Workflow Status (branch)](https://img.shields.io/github/workflow/status/NCAR/cesm-lens-aws/deploy-site/main?logo=github\u0026style=for-the-badge)\n\n# CESM LENS on AWS\n\n- [CESM LENS on AWS](#cesm-lens-on-aws)\n  - [Re-create notebooks with Pangeo Binder](#re-create-notebooks-with-pangeo-binder)\n  - [ESM catalog](#esm-catalog)\n  - [Requirements](#requirements)\n  - [Examples](#examples)\n  - [Reference Documentation](#reference-documentation)\n  - [Source Code for CESM LENS on AWS Site](#source-code-for-cesm-lens-on-aws-site)\n\nExamples of analysis of [CESM LENS data](https://registry.opendata.aws/ncar-cesm-lens/) publicly available on Amazon S3 (us-west-2 region) using xarray and dask.\n\n## Re-create notebooks with Pangeo Binder\n\nTry these notebooks on Pangeo Binder. Note that\nthe session is ephemeral. **Your home directory will not persist, so remember to download your notebooks if you made changes that you need to use at a later time!**\n\n[![badge](https://img.shields.io/badge/Launch%20Pangeo%20Binder-%20AWS%20us--west--2-F5A252.svg?logo=data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAFkAAABZCAMAAABi1XidAAAB8lBMVEX///9XmsrmZYH1olJXmsr1olJXmsrmZYH1olJXmsr1olJXmsrmZYH1olL1olJXmsr1olJXmsrmZYH1olL1olJXmsrmZYH1olJXmsr1olL1olJXmsrmZYH1olL1olJXmsrmZYH1olL1olL0nFf1olJXmsrmZYH1olJXmsq8dZb1olJXmsrmZYH1olJXmspXmspXmsr1olL1olJXmsrmZYH1olJXmsr1olL1olJXmsrmZYH1olL1olLeaIVXmsrmZYH1olL1olL1olJXmsrmZYH1olLna31Xmsr1olJXmsr1olJXmsrmZYH1olLqoVr1olJXmsr1olJXmsrmZYH1olL1olKkfaPobXvviGabgadXmsqThKuofKHmZ4Dobnr1olJXmsr1olJXmspXmsr1olJXmsrfZ4TuhWn1olL1olJXmsqBi7X1olJXmspZmslbmMhbmsdemsVfl8ZgmsNim8Jpk8F0m7R4m7F5nLB6jbh7jbiDirOEibOGnKaMhq+PnaCVg6qWg6qegKaff6WhnpKofKGtnomxeZy3noG6dZi+n3vCcpPDcpPGn3bLb4/Mb47UbIrVa4rYoGjdaIbeaIXhoWHmZYHobXvpcHjqdHXreHLroVrsfG/uhGnuh2bwj2Hxk17yl1vzmljzm1j0nlX1olL3AJXWAAAAbXRSTlMAEBAQHx8gICAuLjAwMDw9PUBAQEpQUFBXV1hgYGBkcHBwcXl8gICAgoiIkJCQlJicnJ2goKCmqK+wsLC4usDAwMjP0NDQ1NbW3Nzg4ODi5+3v8PDw8/T09PX29vb39/f5+fr7+/z8/Pz9/v7+zczCxgAABC5JREFUeAHN1ul3k0UUBvCb1CTVpmpaitAGSLSpSuKCLWpbTKNJFGlcSMAFF63iUmRccNG6gLbuxkXU66JAUef/9LSpmXnyLr3T5AO/rzl5zj137p136BISy44fKJXuGN/d19PUfYeO67Znqtf2KH33Id1psXoFdW30sPZ1sMvs2D060AHqws4FHeJojLZqnw53cmfvg+XR8mC0OEjuxrXEkX5ydeVJLVIlV0e10PXk5k7dYeHu7Cj1j+49uKg7uLU61tGLw1lq27ugQYlclHC4bgv7VQ+TAyj5Zc/UjsPvs1sd5cWryWObtvWT2EPa4rtnWW3JkpjggEpbOsPr7F7EyNewtpBIslA7p43HCsnwooXTEc3UmPmCNn5lrqTJxy6nRmcavGZVt/3Da2pD5NHvsOHJCrdc1G2r3DITpU7yic7w/7Rxnjc0kt5GC4djiv2Sz3Fb2iEZg41/ddsFDoyuYrIkmFehz0HR2thPgQqMyQYb2OtB0WxsZ3BeG3+wpRb1vzl2UYBog8FfGhttFKjtAclnZYrRo9ryG9uG/FZQU4AEg8ZE9LjGMzTmqKXPLnlWVnIlQQTvxJf8ip7VgjZjyVPrjw1te5otM7RmP7xm+sK2Gv9I8Gi++BRbEkR9EBw8zRUcKxwp73xkaLiqQb+kGduJTNHG72zcW9LoJgqQxpP3/Tj//c3yB0tqzaml05/+orHLksVO+95kX7/7qgJvnjlrfr2Ggsyx0eoy9uPzN5SPd86aXggOsEKW2Prz7du3VID3/tzs/sSRs2w7ovVHKtjrX2pd7ZMlTxAYfBAL9jiDwfLkq55Tm7ifhMlTGPyCAs7RFRhn47JnlcB9RM5T97ASuZXIcVNuUDIndpDbdsfrqsOppeXl5Y+XVKdjFCTh+zGaVuj0d9zy05PPK3QzBamxdwtTCrzyg/2Rvf2EstUjordGwa/kx9mSJLr8mLLtCW8HHGJc2R5hS219IiF6PnTusOqcMl57gm0Z8kanKMAQg0qSyuZfn7zItsbGyO9QlnxY0eCuD1XL2ys/MsrQhltE7Ug0uFOzufJFE2PxBo/YAx8XPPdDwWN0MrDRYIZF0mSMKCNHgaIVFoBbNoLJ7tEQDKxGF0kcLQimojCZopv0OkNOyWCCg9XMVAi7ARJzQdM2QUh0gmBozjc3Skg6dSBRqDGYSUOu66Zg+I2fNZs/M3/f/Grl/XnyF1Gw3VKCez0PN5IUfFLqvgUN4C0qNqYs5YhPL+aVZYDE4IpUk57oSFnJm4FyCqqOE0jhY2SMyLFoo56zyo6becOS5UVDdj7Vih0zp+tcMhwRpBeLyqtIjlJKAIZSbI8SGSF3k0pA3mR5tHuwPFoa7N7reoq2bqCsAk1HqCu5uvI1n6JuRXI+S1Mco54YmYTwcn6Aeic+kssXi8XpXC4V3t7/ADuTNKaQJdScAAAAAElFTkSuQmCC)](https://aws-uswest2-binder.pangeo.io/v2/gh/NCAR/cesm-lens-aws/binder-config?urlpath=git-pull?repo=https://github.com/NCAR/cesm-lens-aws%26amp%3Bbranch=main%26amp%3Burlpath=lab/tree/cesm-lens-aws/%3Fautodecode)\n\n## ESM catalog\n\nThe main catalog URL is:\n\nhttps://raw.githubusercontent.com/NCAR/cesm-lens-aws/main/intake-catalogs/aws-cesm1-le.json\n\nThis catalog is an [ESM collection](https://github.com/NCAR/esm-collection-spec) catalog. The data is stored in [Zarr](https://github.com/zarr-developers/zarr) format and meant to be opened with [Xarray](http://xarray.pydata.org/en/latest/).\n\n## Requirements\n\nUsing this catalog requires the following package versions:\n\n- [Intake-esm](https://github.com/intake/intake-esm) \u003e= `v2020.11.4`\n\n## Examples\n\nTo open the catalog and load a data set from Python, you can run the following code:\n\n```python\n\nIn [1]: import intake\n\nIn [2]: col = intake.open_esm_datastore(\"https://raw.githubusercontent.com/NCAR/cesm-lens-aws/main/intake-catalogs/aws-cesm1-le.json\")\n\nIn [3]: col\nOut[3]: \u003caws-cesm1-le catalog with 55 dataset(s) from 391 asset(s)\u003e\n\nIn [4]: col.df.head()\nOut[4]:\n  component frequency experiment  ... dim_per_tstep                start                  end\n0       atm     daily       CTRL  ...           2.0  0402-01-01 12:00:00  2200-12-31 12:00:00\n1       atm     daily       CTRL  ...           2.0  0402-01-01 12:00:00  2200-12-31 12:00:00\n2       atm     daily       CTRL  ...           2.0  0402-01-01 12:00:00  2200-12-31 12:00:00\n3       atm     daily       CTRL  ...           2.0  0402-01-01 12:00:00  2200-12-31 12:00:00\n4       atm     daily       CTRL  ...           2.0  0402-01-01 12:00:00  2200-12-31 12:00:00\n\n[5 rows x 9 columns]\n\nIn [5]: col_subset = col.search(experiment=\"RCP85\", frequency=\"monthly\", variable=[\"hi\", \"aice\"])\n\nIn [6]: dsets = col_subset.to_dataset_dict(zarr_kwargs={\"consolidated\": True}, storage_options={\"anon\": True})\n\n--\u003e The keys in the returned dictionary of datasets are constructed as follows:\n        'component.experiment.frequency'\n |████████████████████████████████████████████████████████████████████████████████████████████████████| 100.00% [2/2 00:00\u003c00:00]\nIn [7]: dsets.keys()\nOut[7]: dict_keys(['ice_sh.RCP85.monthly', 'ice_nh.RCP85.monthly'])\n\nIn [8]: ds = dsets['ice_sh.RCP85.monthly']\n\nIn [9]: ds\nOut[9]:\n\u003cxarray.Dataset\u003e\nDimensions:      (d2: 2, member_id: 40, ni: 320, nj: 76, time: 1140)\nCoordinates:\n  * member_id    (member_id) int64 1 2 3 4 5 6 7 8 ... 34 35 101 102 103 104 105\n  * time         (time) object 2006-01-16 12:00:00 ... 2100-12-16 12:00:00\n    time_bounds  (time, d2) object dask.array\u003cchunksize=(1140, 2), meta=np.ndarray\u003e\nDimensions without coordinates: d2, ni, nj\nData variables:\n    aice         (member_id, time, nj, ni) float32 dask.array\u003cchunksize=(1, 1140, 76, 320), meta=np.ndarray\u003e\n    hi           (member_id, time, nj, ni) float32 dask.array\u003cchunksize=(1, 1140, 76, 320), meta=np.ndarray\u003e\nAttributes:\n    comment3:                  seconds elapsed into model date:      0\n    conventions:               CF-1.0\n    nco_openmp_thread_number:  1\n    source:                    sea ice model: Community Ice Code (CICE)\n    NCO:                       4.3.4\n    contents:                  Diagnostic and Prognostic Variables\n    comment2:                  File written on model date 20060201\n    comment:                   All years have exactly 365 days\n    intake_esm_dataset_key:    ice_sh.RCP85.monthly\n```\n\n## Reference Documentation\n\n- For details about intake-esm API, see the [reference documentation](https://intake-esm.readthedocs.io/en/latest)\n- [CESM LENS on AWS Site](https://doi.org/10.26024/wt24-5j82)\n\n## Source Code for CESM LENS on AWS Site\n\nThe source code for [https://doi.org/10.26024/wt24-5j82](https://doi.org/10.26024/wt24-5j82) resides in the [site directory](./site) of this repository.\n\nThe site is built with [sphinx](https://www.sphinx-doc.org/).\n\nTo build the site locally, please use [conda](https://docs.conda.io/) to set up a build environment with all dependencies.\n\nFirst, make a local clone of this source repository on your machine. For example:\n\n```bash\ngit clone https://github.com/NCAR/cesm-lens-aws\n```\n\nSet up your a conda environment:\n\n```bash\nconda env create -f site/environment.yml\nconda activate cesm-lens-aws-site\nbash site/install-extension.sh\n```\n\nYou can then build the site with:\n\n```bash\nmake live\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fncar%2Fcesm-lens-aws","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fncar%2Fcesm-lens-aws","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fncar%2Fcesm-lens-aws/lists"}