{"id":13948902,"url":"https://github.com/coecms/clef","last_synced_at":"2025-07-20T11:30:24.638Z","repository":{"id":49125480,"uuid":"105484358","full_name":"coecms/clef","owner":"coecms","description":null,"archived":false,"fork":false,"pushed_at":"2023-08-15T23:17:35.000Z","size":1008,"stargazers_count":8,"open_issues_count":15,"forks_count":3,"subscribers_count":5,"default_branch":"master","last_synced_at":"2025-06-22T18:51:56.356Z","etag":null,"topics":["climate-data","cmip","python3"],"latest_commit_sha":null,"homepage":"https://clef.readthedocs.io","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/coecms.png","metadata":{"files":{"readme":"README.rst","changelog":null,"contributing":null,"funding":null,"license":"LICENSE-2.0.txt","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null}},"created_at":"2017-10-02T00:19:31.000Z","updated_at":"2025-01-18T22:12:18.000Z","dependencies_parsed_at":"2023-09-21T19:47:13.139Z","dependency_job_id":null,"html_url":"https://github.com/coecms/clef","commit_stats":{"total_commits":401,"total_committers":6,"mean_commits":66.83333333333333,"dds":0.4688279301745636,"last_synced_commit":"d7dd2983641e52a176e772a36372248ebb5812fe"},"previous_names":[],"tags_count":43,"template":false,"template_full_name":null,"purl":"pkg:github/coecms/clef","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/coecms%2Fclef","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/coecms%2Fclef/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/coecms%2Fclef/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/coecms%2Fclef/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/coecms","download_url":"https://codeload.github.com/coecms/clef/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/coecms%2Fclef/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":261945351,"owners_count":23234234,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["climate-data","cmip","python3"],"created_at":"2024-08-08T05:01:33.930Z","updated_at":"2025-07-20T11:30:24.235Z","avatar_url":"https://github.com/coecms.png","language":"Python","funding_links":[],"categories":["Climate Change"],"sub_categories":["Climate Data Access and Visualization"],"readme":"=============================\n𝄞 `clef \u003chttps://clef.readthedocs.io/en/stable\u003e`_\n=============================\n\nCleF - Climate Finder - Dataset search tool developed by the `CLEX \u003chttps://climateextremes.org.au\u003e`_ `CMS \u003chttps://climate-cms.org\u003e`_ team, powered by `ESGF \u003chttps://esgf-node.llnl.gov/\u003e`_ and the `NCI \u003chttps://nci.org.au\u003e`_ clef.nci.org.au local database\n\n.. image:: https://readthedocs.org/projects/clef/badge/?version=latest\n  :target: https://clef.readthedocs.io/en/stable/\n.. image:: https://circleci.com/gh/coecms/clef/tree/master.svg?style=shield\n  :target: https://circleci.com/gh/coecms/clef/tree/master\n.. image:: https://img.shields.io/conda/v/coecms/clef.svg\n  :target: https://anaconda.org/coecms/clef\n.. image:: https://zenodo.org/badge/DOI/10.5281/zenodo.8251030.svg\n   :target: https://doi.org/10.5281/zenodo.8251030\n\n.. content-marker-for-sphinx\n\nClef searches the Earth System Grid Federation datasets stored at the Australian National Computational Infrastructure, both data published on the NCI\nESGF node as well as files that are locally replicated from other ESGF nodes.\n\nCurrently it searches for the following datasets:\n\n- **CMIP5**  raijin projects: rr3, where NCI is the primary publisher and al33 for replicas \n- **CMIP6**  raijin projects: 0i10 for replicas \n- **CORDEX**  raijin projects: rr3, where NCI is the primary publisher and al33 for replicas \n\nThe search returns both the path of data that is already available at NCI as well as information on data that\nis on external ESGF nodes but not yet available locally.\n\n-------\nInstall\n-------\n\nCleF is pre-installed into a Conda environment at NCI. Load it with::\n\n    module use /g/data3/hh5/public/modules\n    module load conda/analysis3-unstable\n\nNB You need to be a member of hh5 to load the modules\n\nWe are constantly adding new features, the development version is available in a separate environment::\n    module use /g/data3/hh5/public/modules\n    module load conda\n    source activate clef-test\n\nYou can install it to your own environment with::\n\n    conda install -c coecms -c conda-forge clef\n\nBut note that the clef.nci.org.au database necessary for running ``clef`` can only be accessed\nfrom NCI systems\n\n---\nUse\n---\n\nclef cmip5\n~~~~~\n\nFind CMIP5 files matching the constraints::\n\n    clef cmip5 --model BCC-CSM1.1 --variable tas --experiment historical --table day\n\nYou can filter CMIP5 by the following terms:\n \n * ensemble/member\n * experiment\n * experiment-family\n * model\n * table/cmor_table\n * realm\n * frequency\n * variable\n * cf-standard-name\n * institution\n\nSee ``clef cmip5 --help`` for all available filters and their aliases\n\n   ``--latest`` will check the latest versions of the datasets on the ESGF\nwebsite, and will only return matching files\n\nIt will return a path for all the files available locally at NCI and a dataset-id for the ones that haven't been downloaded yet.\n\nYou can use the flags ``--local`` and ``--missing`` to return respectively only the local paths or the missing dataset-id::\n\n    clef --local cmip5 --model MPI-ESM-LR --variable tas --table day\n    clef --missing cmip5 --model MPI-ESM-LR --variable tas --table day\n\nNB these flags come immediately after the command \"clef\" and before the sub-command \"cmip5\" or \"cmip6\". They are also clearly mutually exclusive.\nYou can repeat arguments more than once:: \n\n    clef --missing cmip5 --model MPI-ESM-LR -v tas -v tasmax -t day -t Amon\n\nclef cmip6\n~~~~~\n\nYou can filter CMIP6 by the following terms:\n \n * activity\n * experiment\n * source_type \n * model\n * member\n * table\n * grid\n * resolution\n * realm\n * frequency\n * variable\n * version\n * sub_experiment\n * variant_label\n * institution\n * cf_standard_name\n\nSee ``clef cmip6 --help`` for all available filters\n\nclef cordex\n~~~~~\n\nYou can filter CORDEX by the following terms:\n \n * experiment\n * domain\n * driving_model\n * rcm_name (model)\n * rcm_version\n * ensemble\n * table\n * time_frequency\n * variable\n * version\n * experiment_family\n * institute\n * cf_standard_name\n\nSee ``clef cordex --help`` for all available filters\n\n-------\n-------\nDevelop\n-------\n\nDevelopment install::\n\n    conda env create -f conda/dev-environment.yml\n    source activate clef-dev\n    pip install -e '.[dev]'\n\nThe `dev-environment.yml` file is for speeding up installs and installing\npackages unavailable on pypi, `requirements.txt` is the source of truth for\ndependencies.\n\nTo work on the database tables you may need to start up a test database.\n\nYou can start a test database either with Docker::\n\n    docker-compose up # (In a separate terminal)\n    psql -h localhost -U postgres -f db/nci.sql\n    psql -h localhost -U postgres -f db/tables.sql\n    # ... do testing\n    docker-compose rm\n\nOr with Vagrant::\n\n    vagrant up\n    # ... do testing\n    vagrant destroy\n\nRun tests with py.test (they will default to using the test database)::\n\n    py.test\n\nor connect to the production database with::\n\n    py.test --db=postgresql://clef.nci.org.au/postgres\n\nBuild the documentation using Sphinx::\n\n    python setup.py build_sphinx\n    firefox docs/_build/index.html\n\nNew releases are packaged and uploaded to anaconda.org by CircleCI when a new\nGithub release is made\n\nDocumentation is available on ReadTheDocs, both for `stable\n\u003chttps://clef.readthedocs.io/en/stable/\u003e`_ and `latest\n\u003chttps://clef.readthedocs.io/en/latest/\u003e`_ versions.\n\nDisclaimer\n----------\nCleF can only return datasets which are listed in the ESGF database system for remote results and on the NCI clef database for local results. This means that potentially some of the datasets might not be returned in the following cases:\n * One or more of the ESGF nodes are offline: this can affect clef returning results for the models which are hostedworks which are offline. It is usually easy to verify if this is the case since a query on the browser should show a reduced list of models. In such cases using the *--local* flag will use a query method completely independent and will return at least what is available locally.\n * The NCI ESGF node is offline then nothing will be returned by the default or remote queries, again using *--local* should work. \n * The checksums stored in the ESGF database are different from the actual file checksums. CleF uses the checksums to match the files available remotely if even one file does not match it will flag the dataset as missing. Using the *--local* flag should still return the datasets regardless because it doesn't compare them to what is available remotely. \n * A dataset has been recently donwloaded (up to a week before) and hasn't yet been added to the NCI clef database. In such case it might not show locally even if it has been downloaded. The NCI clef database is updated weekly so we cannot guarantee for clef to find data which is more recent than that. NCI also provides us with a list of datasets recently queued or downloaded. The default query will show this data as \"queued\" or \"downloaded\", rather than missing. While this list aims to cover the gap in between database updates, we have no control on its frequency and it might not capture all the data.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcoecms%2Fclef","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fcoecms%2Fclef","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcoecms%2Fclef/lists"}