Data

Tools

Indexes

Applications

Experiments

Awesome

An open API service indexing awesome lists of open source software.

https://github.com/ecmwf-projects/gecko

Build catalogue entries for the DSS
https://github.com/ecmwf-projects/gecko

cads

Last synced: 25 days ago
JSON representation

Build catalogue entries for the DSS

Awesome Lists containing this project

README 

          
# gecko



![LOGO](gecko-logo.png)



Package for constructing CADS catalogue entries. The software is used as a command line interface

which has been documented via --help commands.



## Installation



1. If you are working on ATOS, you are encourage to use the gecko module which will ensure you are using the

   latest stable version of gecko:



   ```

   module load gecko

   ```



1. If you are working externally to ATOS, you can pip install directly from the github repository using the

   command below. Replace `` with the version you you want to install.

   You can find the latest version tag by viewing the releases in the right hand pane of the

   github page.



   ```

   pip install "gecko@git+https://github.com/ecmwf-projects/gecko@" --upgrade

   ```



1. Should you have difficulties with installation you should install gecko in a clean python environment.

   conda is recommended for this:



   ```

   # If working on ATOS:

   module load conda



   conda create -y -n GECKO -c conda-forge python=3.10

   conda activate GECKO

   ```



Given the difficulties in supporting many different operating systems,

the only officially supported installation is via the gecko module, or installation with

a dedicated conda environment.



## Getting started



```

gecko --help

```



This gives a full list of commands available in gecko, each command has it's own `--help` option with details on how to use that method. The **Using gecko to generate catalogue entries** section provides short summary of the commands you will use in operation, but you are encouraged to use the `--help` to get a full description as that will be updated with the software.



### Setup your gecko configuration



```

gecko update-geckorc

```



This command will create a `.geckorc` file in your current working directory which can be used to configure your gecko working envionment. It will fetch any environment variables you had set up for previous versions of gecko.



When gecko runs it will use the configuration found in first `.geckorc` file found from these locations:



- `./.geckorc`  (current working directory, i.e. at the catalogue entry level)

- `../.geckorc` (e.g. at the repository level)

- `../../.geckorc` (e.g. at the group of respositories level)

- `$HOME/.geckorc`  (e.g. global)



This should give you the flexibility to create specialised gecko configurations for various projects you may be working on.



Here is an example .geckorc file and the variables that can be configured



```

# This dictionary describes the configurartions for the VMs where admin commands (e.g. update-website) are

#  executed. You will need your public ssh key added to the VM to make use of these admin commands.

CLUSTER_CONFIGRATIONS:

   cci2:

      hostname: 136.156.137.225

      namespaces:

      - dev

      - test

      - preprod

      - prod

      username: gecko

# The default cluster to interact with

DEFAULT_CLUSTER: cci2



# Local instance of the cads-forms-json. gecko will clone the repo to this location if it does not already

#  exist. If this variable is not set, when required gecko will clone the json-repo at the top directory

#  level of the yaml repo you are working on.

DEFAULT_JSON_REPO_LOCAL: /path/to/local/cads-forms-json



# Remote address of the json repo. Note this is where the form configurations in the format required by the

#  CADS architecture are pushed so that they can be harvested by the catalogue-manager.

DEFAULT_JSON_REPO_REMOTE: git@github.com:ecmwf-projects/cads-forms-json.git



```



## Using gecko to generate catalogue entries



For examples of how to configure catalogue entries please refer the [cads-forms-demo](https://git.ecmwf.int/projects/CDS/repos/cads-forms-demo/browse) repository.



### Building



```

gecko build-form [PATH_TO_ENTRY]

```



Build the json files required for the web-form and adaptor:

`form.json`, `mapping.json`, `adaptor.json`, `constraints.json`



______________________________________________________________________



```

gecko build-content [PATH_TO_ENTRY]

```



Build the json files required for metadata and the web entry layout:

`layout.json`, `metadata.json`, `variables.json`



______________________________________________________________________



```

gecko build [PATH_TO_ENTRY]

```



Build all the json files for an entry (form + content):

`form.json`, `mapping.json`, `adaptor.json`, `constraints.json`, `layout.json`, `metadata.json`, `variables.json`



______________________________________________________________________



```

gecko build-ready [READY_FILE]

```



Build all forms listed in a `READY` file (default `./READY`). Each line in the file should be a path to

a catalogue entry directory, relative to the location of the `READY` file.



______________________________________________________________________



### Publishing



```

gecko publish [PATH_TO_ENTRY]

```



Publish the json content created by the build. This pushes the json files to the git repository

used operationally by the CADS infrastructure.



______________________________________________________________________



```

gecko publish-ready [READY_FILE]

```



Publish all forms listed in a `READY` file (default `./READY`). Same format as `build-ready`.



______________________________________________________________________



```

gecko unpublish [PATH_TO_ENTRY]

```



Remove (unpublish) a catalogue entry from the instance. Removes the content from the json repo

so that it can be repopulated at a later date.



______________________________________________________________________



```

gecko update-local-json [PATH_TO_ENTRY]

```



Update your local json content. Recommended to run this prior to using `gecko build --update`,

or when debugging configs or making hand edits to the json files.



______________________________________________________________________



```

gecko update-website

```



Trigger the catalogue manager to update the website with the latest published changes. You will need

your public ssh key added to the image-factory VM — contact the CDS core team regarding this.



______________________________________________________________________



### Configuration



```

gecko create-geckorc

```



Create a `.geckorc` file in the current directory using the currently active configuration values.

In most cases this file should be moved to `~/.geckorc`. Use `--update-existing` (`-u`) to update

the geckorc file that is currently in use (useful when upgrading gecko).



______________________________________________________________________



### Security



```

gecko encrypt VALUE

gecko encrypt --decrypt VALUE

```



Encrypt a plaintext value using the Fernet key configured in your `.geckorc`

(`DSS_ADAPTOR_ENCRYPTION_KEY`). The resulting token can be embedded directly in adaptor configs

(e.g. as a password or API key) and will be decrypted server-side by the adaptor using the

`ADAPTOR_DECRYPTION_KEY` environment variable. Use `--decrypt` / `-d` to reverse the operation.



```

# Encrypt a value

gecko encrypt "my-secret-password"

# gAAAAABqAv4ZBbRF1B...



# Decrypt a previously encrypted token

gecko encrypt --decrypt "gAAAAABqAv4ZBbRF1B..."

# my-secret-password

```



Encrypted credentials can be used in manifest configurations that require HTTP basic auth:



```yaml

manifest: https://example.com/list.txt

auth:

  username: myuser

  password: gAAAAABqAv4ZBbRF1B...  # output of `gecko encrypt`

```



The `{AUTH:USERNAME}` and `{AUTH:PASSWORD}` placeholders are also substituted into

`manifest_plugin` shell commands, so credentials never need to appear in plain text in config files.



______________________________________________________________________



### Discovery and debugging



```

gecko block-help [BLOCKS ...]

```



Print help and descriptions for content-management blocks. Pass one or more block names to filter

the output, or omit arguments to print help for all available blocks.



______________________________________________________________________



```

gecko standard-layout TAB

```



Print the standard layout configuration YAML for a given tab (e.g. `overview`, `download`).

Useful as a starting point when authoring layout configuration files.



______________________________________________________________________



```

gecko version

```



Print the installed gecko version.



______________________________________________________________________



### EUMETSAT / JupyterHub helpers



```

gecko populate-eumdac-yamls EUM_COLLECTION_ID [PATH_TO_ENTRY]

```



Populate the initial YAML configuration files for a EUMETSAT dataset based on a template and

available EUM metadata. `EUM_COLLECTION_ID` is the EUM collection ID,

e.g. `EO:EUM:DAT:METOP:AMSUL1`. Use `--netcdf-file` / `-n` to provide a sample NetCDF file

for richer metadata extraction.



______________________________________________________________________



```

gecko create-jupyterhub-links-file [PATH_TO_ENTRY]

```



Create a `jupyterhub_links.yaml` file for a catalogue entry (default output:

`content/jupyterhub_links.yaml`). Use `--output-file` / `-o` to specify a different output path.



______________________________________________________________________



## Workflow for developers/contributors



For best experience create a new conda environment (e.g. DEVELOP) with Python 3.10:



```

conda create -n DEVELOP -c conda-forge python=3.10

conda activate DEVELOP

```



Before pushing to GitHub, run the following commands:



1. Update conda environment: `make conda-env-update`

1. Install this package: `pip install -e .`

1. Sync with the latest [template](https://github.com/ecmwf-projects/cookiecutter-conda-package) (optional): `make template-update`

1. Run quality assurance checks: `make qa`

1. Run tests: `make unit-tests`

1. Build the documentation (see [Sphinx tutorial](https://www.sphinx-doc.org/en/master/tutorial/)): `make docs-build`



## License



```

Copyright 2022, ECMWF.



Licensed under the Apache License, Version 2.0 (the "License");

you may not use this file except in compliance with the License.

You may obtain a copy of the License at



    http://www.apache.org/licenses/LICENSE-2.0



Unless required by applicable law or agreed to in writing, software

distributed under the License is distributed on an "AS IS" BASIS,

WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.

See the License for the specific language governing permissions and

limitations under the License.

```