{"id":21683753,"url":"https://github.com/open-eo/openeo-test-suite","last_synced_at":"2025-05-12T13:21:39.348Z","repository":{"id":196554755,"uuid":"696381489","full_name":"Open-EO/openeo-test-suite","owner":"Open-EO","description":"Test suite for validation of openEO back-ends against the openEO API specification","archived":false,"fork":false,"pushed_at":"2024-12-18T12:56:22.000Z","size":335,"stargazers_count":0,"open_issues_count":9,"forks_count":4,"subscribers_count":5,"default_branch":"main","last_synced_at":"2025-03-31T22:33:02.619Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/Open-EO.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":null,"funding":null,"license":"LICENSE.txt","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2023-09-25T16:21:08.000Z","updated_at":"2024-12-18T12:56:26.000Z","dependencies_parsed_at":null,"dependency_job_id":"db8a7b36-ade6-4423-84dc-190099d7034d","html_url":"https://github.com/Open-EO/openeo-test-suite","commit_stats":null,"previous_names":["open-eo/openeo-test-suite"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Open-EO%2Fopeneo-test-suite","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Open-EO%2Fopeneo-test-suite/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Open-EO%2Fopeneo-test-suite/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Open-EO%2Fopeneo-test-suite/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/Open-EO","download_url":"https://codeload.github.com/Open-EO/openeo-test-suite/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":253745198,"owners_count":21957319,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-25T16:13:22.714Z","updated_at":"2025-05-12T13:21:39.326Z","avatar_url":"https://github.com/Open-EO.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"\n# openEO Test Suite\n\nTest suite for validation of openEO back-ends against the openEO API and related specifications.\n\nLocation: https://github.com/Open-EO/openeo-test-suite\n\n## Project structure and modular design\n\nThe openEO test suite was developed in separate work packages,\nwhich is roughly reflected in the project structure.\nThe separate modules allow to run the test suite in a more fine-grained way,\nfocussing on a specific API aspect to test or verify\n(e.g. just collection metadata validation or individual process testing).\n\n\n\u003e **Note**\n\u003e In the following overview includes some invocation examples that are given as basic reference.\n\u003e Make sure to also check the more [detailed documentation of the test run options](#run-options) further in the docs.\n\n\n- **WP1 General test suite framework** (lead implementation partner: VITO)\n  - Main location: [`src/openeo_test_suite/lib`](./src/openeo_test_suite/lib)\n  - Provides various reusable utilities and helpers to power the openEO test suite,\n    and defines a pytest plugin to properly hook into the various phases of the pytest framework.\n  - This module also defines some \"internal\" tests that are just meant to test these utilities in isolation,\n    but it is not part of the openEO test suite itself:\n    ```bash\n    pytest src/openeo_test_suite/lib/internal-tests\n    ```\n- **WP2 Validation of collection metadata** (lead implementation partner: EURAC)\n  - Main location: [`src/openeo_test_suite/tests/collections`](./src/openeo_test_suite/tests/collections)\n  - Defines tests to validate openEO collection metadata against specs like\n    the [openEO API](https://openeo.org/) and [STAC](https://stacspec.org/en).\n  - Usage example of just running these tests against a desired openEO backend URL:\n    ```bash\n    pytest src/openeo_test_suite/tests/collections \\\n        -U https://openeo.example \\\n        --html=reports/collection-metadata.html\n    ```\n- **WP3 Validation of process metadata** (lead implementation partner: EODC)\n  - Main location: [`src/openeo_test_suite/tests/processes/metadata`](./src/openeo_test_suite/tests/processes/metadata)\n  - Defines tests to validate openEO process metadata against specs\n    defined in the [openeo-processes](https://github.com/Open-EO/openeo-processes) project\n    - Functional tests concern actual values and behavior of processes (like parameters and return values),\n      failures in these tests should be looked into and fixed.\n    - Non-functional tests concern descriptions and other metadata of processes that have no impact on the actual behavior of the process,\n      failures in these tests should be taken as warnings, but don't necessarily need to be fixed. These can be skipped by adding\n      `-m \"not optional\"` to the pytest command.\n  - Usage example for running these tests against a desired openEO backend URL:\n    ```bash\n    pytest src/openeo_test_suite/tests/processes/metadata \\\n        -U https://openeo.example \\\n        -m \"not optional\" \\\n        --html=reports/process-metadata.html\n    ```\n- **WP4 General openEO API compliance validation** (lead implementation partner: EODC)\n  - Main location: [`src/openeo_test_suite/tests/general`](./src/openeo_test_suite/tests/general)\n  - Provides tests to validate the general openEO API compliance of a back-end.\n  - The backend is checked against the openeo API specification defined in the [openeo-api](https://github.com/Open-EO/openeo-api/) project.\n  - There are some tests which might run for a long time (as they execute actual process graphs on the backends)\n    these can be skipped by adding `-m \"not longrunning\"` to the pytest command.\n  - Usage example of just running these tests against a desired openEO backend URL:\n    ```bash\n    pytest src/openeo_test_suite/tests/general \\\n      -U https://openeo.example \\\n      --html=reports/general.html\n    ```\n- **WP5 Individual process testing** (lead implementation partner: M. Mohr)\n  - Main location: [`src/openeo_test_suite/tests/processes/processing`](./src/openeo_test_suite/tests/processes/processing)\n  - Provides tests to validate individual openEO processes,\n    based on the expected input-output examples\n    defined in the [openeo-processes](https://github.com/Open-EO/openeo-processes) project\n    under [Open-EO/openeo-processes#468](https://github.com/Open-EO/openeo-processes/pull/468)\n  - Very basic usage example of just running these tests:\n    ```bash\n    pytest src/openeo_test_suite/tests/processes/processing \\\n        --html=reports/individual-processes.html\n    ```\n    Note that this invocation will not actually execute anything,\n    see [WP5 Specifics](#wp5-specifics) for more information and functional examples.\n- **WP6 Full process graph execution and validation** (lead implementation partner: EURAC)\n  - Main location: [`src/openeo_test_suite/tests/workflows`](./src/openeo_test_suite/tests/workflows)\n  - Provides tests to run full processes graphs and evaluate the results.\n  - Usage example of just running these tests against a desired openEO backend URL:\n    ```bash\n    pytest src/openeo_test_suite/tests/workflows \\\n        -U https://openeo.example \\\n        --s2-collection SENTINEL2_L2A \\\n        --html=reports/workflows.html\n    ```\n   - See [WP6 Specifics](#wp6-specifics) for some more details and examples.\n\n\n## Installation and setup\n\nClone this repository and some git submodules with additional assets/data, e.g.:\n\n```bash\ngit clone --recurse-submodules https://github.com/Open-EO/openeo-test-suite.git\n```\n\nAs always in Python usage and development,\nit is recommended to work in some sort of virtual environment (`venv`, `virtualenv`, `conda`, `docker`, ...)\nto run and develop this project.\nDepending on your use case and workflow, you can choose to reuse an existing environment or create a new one.\n\n### Example virtual environment setup\n\nPython's standard library includes a [`venv` module](https://docs.python.org/3/library/venv.html)\nto create virtual environments.\nA common practice is to create a virtual environment in a subdirectory `venv` of your project,\nwhich can be achieved by running this from the project root:\n\n```bash\npython -m venv --prompt . venv\n```\n\nNote: the `--prompt .` option is a trick to automatically\nuse the project root directory name in your shell prompt\nwhen the virtual environment is activated.\nIt's optional, but it generally makes your life easier\nwhen you have multiple virtual environments on your system.\n\n### Virtual environment activation\n\nThe following instructions will assume that the virtual environment of your choosing is activated.\nFor example, when using the `venv` approach from above,\nactivate the virtual environment in your shell with:\n\n```bash\nsource venv/bin/activate\n```\n\n### Install `openeo-test-suite` package (with dependencies)\n\nInstall the project and its dependencies in your virtual environment with:\n\n```bash\npip install -e .\n```\n\nNote: the `-e` option installs the project in \"editable\" mode,\nwhich makes sure that code changes will be reflected immediately in your virtual environment\nwithout the need of (re)installing the project.\n\n\n### Additional optional dependencies related to runners for individual process testing \u003ca name=\"runner-dependencies\"\u003e\u003c/a\u003e\n\nThe individual process testing module of the test suite allows to pick\na specific process \"runner\" (see [WP5 specifics](#wp5-specifics) for more documentation).\nSome of these runners require additional optional dependencies to be installed in your virtual environment,\nwhich can be done by providing an appropriate \"extra\" identifier in the `pip install` command:\n\n- For the \"dask\" runner:\n    ```bash\n    pip install -e .[dask]\n    ```\n- For the \"vito\" runner:\n    ```bash\n    pip install -e .[vito]\n    ```\n\nNote that it might be not possible to install both \"dask\" and \"vito\" extras\nseamlessly in the same environment because of conflicting dependency constraints.\n\n\n## Test suite run options \u003ca name=\"run-options\"\u003e\u003c/a\u003e\n\nMost modules of the test suite at least require an openEO backend URL to run against.\nIt can be specified through a `pytest` command line option:\n\n    pytest --openeo-backend-url=openeo.example\n\n    # Or using the short form `-U` (at the cost of being less descriptive):\n    pytest -U openeo.example\n\nIf no explicit protocol is specified, `https://` will be assumed automatically.\n\n### Process selection\n\nVarious tests depend on the availability of certain openEO processes.\nIt is possible to only run tests for a subset of processes with these\nprocess selection options:\n\n- `--processes` to define a comma-separated list of processes to test against.\n  - Example: `--processes=min,load_stac,apply,reduce_dimension`.\n  - Note that processes specified with this option are always selected,\n    irrespective of their \"experimental\" status\n    and the presence of the `--experimental` option discussed below.\n- `--process-levels` to select whole groups of processes based on\n  predefined [process profiles](https://openeo.org/documentation/1.0/developers/profiles/processes.html),\n  specified as a comma-separated list of levels.\n  - Example: `--process-levels=L1,L2`.`\n  - A level does not imply other levels, so each desired level must be specified explicitly.\n    For example, L2 does **not** include L1 automatically.\n\nIf neither `--processes` nor `--process-levels` are specified, all processes are considered.\nIf both are specified, the union of both will be considered.\n\n- `--experimental`: By default, experimental processes (or experimental process tests) are ignored.\n  Enabling this option will consider experimental processes and tests.\n  Note that experimental processes explicitly selected with `--processes` will be\n  kept irrespective of this option.\n\n\n### Recommended `pytest` options\n\npytest provides a [lot of command-line options](https://docs.pytest.org/en/8.0.x/reference/reference.html#command-line-flags)\nto fine-tune how the test suite is executed (test selection, reporting, ...).\nSome recommended options to use in practice:\n\n- `-vv`: increase verbosity while running the test,\n  e.g. to have a better idea of the progress of slow tests.\n- `--tb=short`/`--tb=line`/`--tb=no`: avoid output of full stack traces,\n  which give little to no added value for some test modules.\n\n\n### Authentication of the basic `connection` fixture\n\nThe test suite provides a basic `connection` fixture\n(an `openeo.Connection` object as defined in the `openeo` Python library package)\nto interact with the backend.\n\nThere are several ways to set up authentication for this connection fixture,\nbuilding on the [\"dynamic authentication method selection\" feature of the `openeo` Python library package](https://open-eo.github.io/openeo-python-client/auth.html#oidc-authentication-dynamic-method-selection),\nwhich is driven by the `OPENEO_AUTH_METHOD` environment variable:\n\n- `OPENEO_AUTH_METHOD=none`: no authentication will be done\n- `OPENEO_AUTH_METHOD=basic`: basic authentication will be triggered.\n  Username and password can be specified through additional environment variables\n  `OPENEO_AUTH_BASIC_USERNAME`, and `OPENEO_AUTH_BASIC_PASSWORD`.\n  Alternatively, it is also possible to handle basic auth credentials through\n  the [auth configuration system and `openeo-auth` tool](https://open-eo.github.io/openeo-python-client/auth.html#auth-config-files-and-openeo-auth-helper-tool)\n  from the `openeo` Python library package.\n- `OPENEO_AUTH_METHOD=client_credentials`: OIDC with \"client credentials\" grant,\n  which [assumes some additional environment variables to set the client credentials](https://open-eo.github.io/openeo-python-client/auth.html#oidc-client-credentials-using-environment-variables).\n- If nothing is specified (the default), the default behavior of `connection.authenticate_oidc()` is followed:\n  - Valid OIDC refresh tokens will be used if available\n  - Otherwise, the OIDC device code flow is initiated.\n    Make sure to check the logging/output of the test suite run\n    for instructions on how to complete the authentication flow.\n\n\n## Test module specifics\n\nSome test modules have specific considerations and options to be aware of.\n\n\n### WP5. Individual process testing: process runners \u003ca name=\"wp5-specifics\"\u003e\u003c/a\u003e\n\nThe goal of the **individual process testing** module of the test suite\nis testing each openEO process individually with one or more pairs of input and expected output.\nBecause there are a lot of these tests (order of thousands),\nit is very time-consuming to run these through the standard, HTTP based openEO REST API.\nAs a countermeasure, the test suite ships with several experimental **runners**\nthat aim to execute the tests directly against a (locally running) back-end implementation\nto eliminate HTTP-related and other overhead.\nNote that this typically requires additional dependencies to be installed in your test environment.\n\nThe desired runner can be specified through the `--runner` option,\nwith currently one of the following options:\n\n- `skip` (**default**): Skip all individual process tests.\n  - This is the default to avoid accidentally running a very heavy/costly test suite.\n- `http`: Run the individual processing tests through a standard openEO REST API.\n  - Requires `--openeo-backend-url` to be set as described above.\n  - As noted above, this will very likely result a very heavy test suite run by default.\n    It is therefore recommended to limit the test suite scope\n    in some way: e.g. limited process selection through `--processes`\n    or running against a dedicated/localhost deployment.\n  - Another limitation of this runner is that not all process tests\n    can be executed as some input-output pairs are not JSON encodable.\n    These tests will be marked as skipped.\n- `dask`: Executes the tests directly via the [openEO Dask implementation](https://github.com/Open-EO/openeo-processes-dask) (as used by EODC, EURAC, and others)\n  - Requires [openeo_processes_dask](https://github.com/Open-EO/openeo-processes-dask) package being installed in test environment.\n    See [installation instructions](#runner-dependencies) above for more practical info.\n  - Covers all implemented processes.\n- `vito`: Executes the tests directly via the\n  [openEO Python Driver implementation](https://github.com/Open-EO/openeo-python-driver) (as used by CDSE, VITO/Terrascope, and others).\n  - Requires [openeo_driver](https://github.com/Open-EO/openeo-python-driver) package being installed in test environment.\n    See [installation instructions](#runner-dependencies) above for more practical info.\n  - Only covers a subset of processes due to the underlying architecture of the back-end implementation.\n    In particular, it only covers the pure Python code paths, but not the PySpark related aspects.\n\nSee [openeo_test_suite/lib/process_runner](./src/openeo_test_suite/lib/process_runner)\nfor more details about these runners and inspiration to implement your own runner.\n\n\n\n#### Usage examples of individual process testing with runner option\n\nThe individual process tests can be run by specifying the `src/openeo_test_suite/tests/processes/processing` as test path.\nSome use examples with different options discussed above:\n\n```bash\n# Basic default behavior: run all individual process tests,\n# but with the default runner (skip), so no tests will actually be executed.\npytest src/openeo_test_suite/tests/processes\n\n# Run tests for a subset of processes with the HTTP runner\n# against the openEO Platform backend at openeo.cloud\npytest --runner=http --openeo-backend-url=openeo.cloud \\\n    --processes=min,max \\\n    src/openeo_test_suite/tests/processes/processing\n\n# Run tests for a subset of processes with the VITO runner\npytest --runner=vito --process-levels=L1,L2,L2A \\\n    src/openeo_test_suite/tests/processes/processing\n\n# Run all individual process tests with the Dask runner\npytest --runner=dask src/openeo_test_suite/tests/processes\n```\n\n\n\n### WP6. Full process graph execution and validation \u003ca name=\"wp6-specifics\"\u003e\u003c/a\u003e\n\nThese tests are designed to run using synchronous openEO process calls\nand a Sentinel-2(-like) collection.\n\nThe S2 collection to use must be specified through the `--s2-collection` option,\nwhich supports two forms:\n\n- a normal openEO collection name (e.g. `--s2-collection=SENTINEL2_L2A`)\n  which will be loaded through the standard openEO `load_collection` process.\n- a STAC URL (typically starting with `https://`),\n  which will be loaded with `load_stac`.\n\n  The following two example STAC collections are provided in the context of this test suite project.\n  Each of these exists to cater to subtle differences between some back-end implementations\n  regarding temporal and band dimension naming and how that is handled in `load_stac`.\n   - https://stac.eurac.edu/collections/SENTINEL2_L2A_SAMPLE:\n     defines `\"t\"` as temporal dimension name, and `\"bands\"` as bands dimension name.\n     Recommended to be used with VITO/CDSE openEO backends.\n   - https://stac.eurac.edu/collections/SENTINEL2_L2A_SAMPLE_2:\n     uses `\"time\"` as temporal dimension name, and `\"band\"` as bands dimension name.\n     Recommended to be used with EURAC and EODC openEO backends.\n\nIf the back-end does not support `load_stac`, a collection name must be used.\n\n#### Usage examples:\n\n```bash\n# Compact\npytest src/openeo_test_suite/tests/workflows \\\n    -U openeo.dataspace.copernicus.eu \\\n    --s2-collection=SENTINEL2_L2A\n\n# With full back-end URL,\n# a STAC collection to use instead of a predefined openEO collection\n# and a list of process levels to test against\npytest src/openeo_test_suite/tests/workflows \\\n    --openeo-backend-url=https://openeo.dataspace.copernicus.eu/openeo/1.2 \\\n    --s2-collection=https://stac.eurac.edu/collections/SENTINEL2_L2A_SAMPLE \\\n    --process-levels=L1,L2\n```\n\n\n\n\n## Reporting\n\nBeing a `pytest` based test suite, various plugins can be used to generate reports in different formats as desired.\n\n### HTML report\n\nProducing reports in HTML format is enabled by default (using the `pytest-html` plugin).\nBasic usage example\n(note that this usually has to be combined with other `pytests` options commented elsewhere in this document):\n\n    pytest --html=reports/report.html\n\n### JUnitXML report\n\nJUnitXML reports are useful for integration with CI systems like Jenkins.\nSupport for it is built-in in `pytest` through the `--junit-xml` option:\n\n    pytest --junit-xml=reports/report.xml\n\n\n## Extending the test suite and adding new tests\n\nHow to extend the test suite depends largely on the module or aspect you want to extend.\nSome general guidelines:\n\n- General test suite framework: add new functionality or extend existing utilities\n  `src/openeo_test_suite/lib`. Preferably also add \"internal\" tests,\n  which should be able to run without the presence of a real openEO backend.\n- Validation of collection metadata:\n  extend existing tests or add new tests at `src/openeo_test_suite/tests/collections`.\n- Validation of process metadata:\n  add new tests to `src/openeo_test_suite/tests/processes/metadata`.\n- General openEO API compliance validation:\n  add new tests to `src/openeo_test_suite/tests/general`.\n- Individual process testing:\n  - new input-output pairs for existing or new processes:\n    add them in the [openeo-processes](https://github.com/Open-EO/openeo-processes) project\n    or under [Open-EO/openeo-processes#468](https://github.com/Open-EO/openeo-processes/pull/468)\n  - extend functionality (e.g. process runners, data conversion aspects, ...):\n    see [openeo_test_suite/lib/process_runner](./src/openeo_test_suite/lib/process_runner)\n- Full process graph execution and validation:\n  add new tests to `src/openeo_test_suite/tests/workflows`,\n  use existing tests there as inspiration on how to write new tests.\n\n\n## Development and contributions\n\nThe test suite builds on [`pytest`](https://pytest.org/),\na featureful testing library for Python that makes it easy to write succinct, readable tests\nand can scale to support complex functional testing.\nIt has some particular features (fixtures, leveraging the `assert` statement, test parameterization, ...)\nthat are worth getting familiar with.\n\n### Pre-commit hooks\n\nThis project provides (git) pre-commit hooks to tidy up simple code quality problems and catch syntax issues before committing.\nWhile not enforced, it is encouraged and appreciated to enable this in your working copy before contributing code.\n\n#### Pre-commit Setup\n\n- Install the general [pre-commit](https://pre-commit.com/) tool.\n  The simplest option is to install it directly in the virtual environment you are using for this project (e.g. `pip install pre-commit`)\n  You can also install it globally\n  (e.g. using a package manager like [pipx](https://pypa.github.io/pipx/), conda, homebrew, ...)\n  so you can use this tool across different projects.\n- Install the project specific git hook scripts and utilities\n  (defined in the `.pre-commit-config.yaml` configuration file)\n  by running this in the root of your working copy:\n\n      pre-commit install\n\n#### Pre-commit Usage\n\nWhen you commit new changes, the pre-commit hook will automatically run each of the configured linters/formatters/...\nSome of these just flag issues (e.g. invalid JSON files) while others even automatically fix problems (e.g. clean up excessive whitespace).\nIf there is some kind of violation, the commit will be blocked. Address these problems and try to commit again.\n\nSome pre-commit tools directly edit your files (e.g. fix code style) instead of just flagging issues.\nThis might feel intrusive at first, but once you get the hang of it, it should allow to streamline your development workflow.\nIn particular, it is recommended to use git's staging feature to prepare your commit.\nPre-commit’s proposed changes are not staged automatically, so you can more easily keep them separate for review.\n\n\u003e **Note**\n\u003e You can temporarily disable pre-commit for these rare cases where you intentionally want to commit violating code style,\n\u003e e.g. through git commit command line option `-n/--no-verify`.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fopen-eo%2Fopeneo-test-suite","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fopen-eo%2Fopeneo-test-suite","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fopen-eo%2Fopeneo-test-suite/lists"}