{"id":21683757,"url":"https://github.com/open-eo/openeo-result-validation-engine","last_synced_at":"2025-04-12T07:51:46.040Z","repository":{"id":37596585,"uuid":"176923172","full_name":"Open-EO/openeo-result-validation-engine","owner":"Open-EO","description":"Image-based validation of Earth Observation cloud processing service results","archived":false,"fork":false,"pushed_at":"2022-12-08T04:53:36.000Z","size":88012,"stargazers_count":3,"open_issues_count":9,"forks_count":2,"subscribers_count":0,"default_branch":"master","last_synced_at":"2025-03-26T03:04:56.563Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/Open-EO.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2019-03-21T10:33:46.000Z","updated_at":"2023-06-22T20:36:13.000Z","dependencies_parsed_at":"2023-01-24T13:00:16.457Z","dependency_job_id":null,"html_url":"https://github.com/Open-EO/openeo-result-validation-engine","commit_stats":null,"previous_names":[],"tags_count":5,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Open-EO%2Fopeneo-result-validation-engine","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Open-EO%2Fopeneo-result-validation-engine/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Open-EO%2Fopeneo-result-validation-engine/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Open-EO%2Fopeneo-result-validation-engine/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/Open-EO","download_url":"https://codeload.github.com/Open-EO/openeo-result-validation-engine/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248537009,"owners_count":21120690,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-25T16:13:25.168Z","updated_at":"2025-04-12T07:51:46.017Z","avatar_url":"https://github.com/Open-EO.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# openeo-result-validation-engine\n\n[![Codacy Badge](https://api.codacy.com/project/badge/Grade/6fe179a5dc8e4ac29bdf94f1e9d40fa4)](https://app.codacy.com/app/m-mohr/openeo-result-validation-engine?utm_source=github.com\u0026utm_medium=referral\u0026utm_content=Open-EO/openeo-result-validation-engine\u0026utm_campaign=Badge_Grade_Dashboard)\n[![Build Status](https://travis-ci.org/Open-EO/openeo-result-validation-engine.svg?branch=master)](https://travis-ci.org/Open-EO/openeo-result-validation-engine)\n\nThe openeo-validation-engine uses the [OpenEO-sentinel-reference-jobs](https://github.com/Sijoma/openeo-sentinel-reference-jobs) as a submodule and fetches the results of these jobs from every specified back-end provider. The results are then validated and a report is generated for each job. These reports are then pushed to the gh-pages branch of this repository.\n\nDue to filesize limitations, comparison images are currently not stored in this process.\n\n\n## Installation\nWe use Python3 in a venv environment. Using a venv is not necessary but recommended. The following commands have to be run sequentially to install the application and its dependencies.\n\n```bash\ngit clone --recursive https://github.com/Open-EO/openeo-result-validation-engine\ncd openeo-result-validation-engine\npython3 -m venv ./venv\nsource venv/bin/activate\npip install -r requirements.txt\ncd openeo-python-client\npip install -r requirements.txt\npip install -e .\n```\n\n## Configuration\n\nOpenEO back-end providers can be added in the backendProvider.json. This JSON contains an array of JSON objects that\ncontain the information for a given provider.\n\n* `name`: The name of the backend-provider.\n* `baseURL`: The address of the openEO API endpoint from the backend-provider.\n* `credentials`: The account information required to authenticate openEO requests.\n  * `user`: The username, this is optional and can be left empty\n  * `password`: The password to authenticate, this is optional and can be left empty\n\n```json\n      {\n      \"name\": \"WWU_GEE\",\n      \"baseURL\": \"https://earthengine.openeo.org/v0.3\",\n      \"credentials\": {\n        \"user\": \"Example\",\n        \"password\": \"Example123\"\n        }\n      }\n```\n\n### Reference jobs\n\nReference jobs (jobs that are validated) can be added in the folder `openeo-sentinel-reference-jobs`. They are currently\nseparated by arbitrary regions to make it a bit easier to distinguish them. Each Job has a folder that should be called\nJob-N (N is the counter). This folder then has to contain a folder that matches the `name` specified in the backendProvider.json,\ne.g. `openEO-GEE` in our current example. This folder then contains a single process graph.\n\nValidation rules can be configured on a per job basis, in a file called `validation-rules.json`. There are three possible options:\n\n* Specifying no rules; `default` validation rules get selected (stored in the folder `rule-configurations`)\n\n* Specifying a `name` of a ruleset; the validation engine then tries to find a json file with the name `name`-validation-rules.json\n\n```json\n{\n  \"ruleset-name\": \"quick\"\n}\n```\n\n* Specifying validation rules directly inside the json.\n\n```json\n{\n  \"validation\": {\n      \"rules\": {\n          \"input-data-checks\": {\n          },\n          \"output-data-checks\": {\n              \"matching-file-extensions\": \"png\",\n              \"file-size-check\": 0.2\n          },\n          \"color-checks\": {\n            \"threshold-correlation\": 0.5,\n            \"threshold-intersection\": 345,\n            \"threshold-chi-squared\": 1500,\n            \"threshold-hellinger\": 0.5,\n            \"threshold-chi-square-alt\": 5000,\n            \"threshold-kullback-leibler\": 5000\n          },\n          \"pixel-checks\": {\n            \"resolution-allow-divergence\": 0.5,\n            \"image-similarity-measures\": 0.95,\n            \"resize-factor\": 0.5\n          },\n          \"classification-checks\": {\n            \"matching-boundaries\": 0.20,\n            \"resize-factor\": 0.5\n          },\n          \"nan-value-check\": {\n            \"allow-nan\": false\n          }\n      }\n  }\n}\n```\n\n## Usage\n\nActivate the Python venv and start the ValidationEngine.\n\n```bash\nsource venv/bin/activate\npython ValidationEngine.py\n```\n\n### Additional CLI arguments\n\nRun `python ValidationEngine.py --help` to get information on all available CLI commands.\n\nIf you want to prevent the validation engine from downloading the same OpenEO jobs again you can pass the argument `--offline True`.\nExample:\n`python ValidationEngine.py --offline True`\n\nTo speed up the validation, a resize factor can be used. This factor reduces the size of the image and thus all computations are quicker. A value of 0.10 results in an image that is 10% of the original size. This of course reduces the quality of the validation.\n\nExample:\n`python ValidationEngine.py --resize 0.10`\n\nMocked examples (jobs stored in the folder mock-examples) can be run with:\n\n```bash\npython ValidationEngine.py --mock True \n```\n\nTo run only a specific reference job, the job-identifier can be passed as an argument.\nThe structure of the id is as follows: \u003cRegion\u003e-Job-\u003cNumber\u003e\nExamples: Island-Job-2, Netherlands-Job-2\n\n```bash\npython ValidationEngine.py --refJob Switzerland-Job-1 \n```\n\n\n## Validation against local files\n\nAdd a similar json-object as the one below to the `backendProvider.json`.\n\n```json\n  {\n    \"name\": \"Local-Python\",\n    \"local\": true\n  }\n```\n\nIn the folder `openeo-sentinel-reference-jobs`, add a folder with the `name` of the backend provider, e.g. `Local-Python`.\nThis folder then should contain a json that stores the path to the locally stored file instead of the process-graph.\n\n```json\n{\n  \"file\": \"/path/to/file/file.tiff\"\n}\n```\n\nThis allows to validate local results against openEO cloud-processed results and it also enables validation of just local results.\n\n## Validation \"Studies\"\n\nThe ValidationEngine runs every 24 hours to compute the validation results for two \"studies\". \n\nValidation reports for the comparison between back-end providers EURAC and WWU/GEE can be found in the branch [gh-pages](https://github.com/Open-EO/openeo-result-validation-engine/tree/gh-pages) of this repository. It only contains the JSON reports as the raw data from EURAC is to large.\n\nValidation reports for the comparison between the COPERNICUS/S2 and COPERNICUS/S2_SR data set of the WWU/GEE back end can be found in the branch [reports-GEE_S2-vs-GEE_SR](https://github.com/Open-EO/openeo-result-validation-engine/tree/reports-GEE_S2-vs-GEE_SR) of this repository. These branch also contains imagery, as we are not able to retrieve native resolution from the WWU/GEE back end and thus are able to store the images on GitHub.\n\n## Encrypting Git Token\n\nTo publish the validation reports to Github this repository needs a valid GitHub Token.\n\n1. install the Travis CI Command Line Client by running \n`gem install travis` and login with `travis login --org`\n2. Generate SSH-KEY\n`ssh-keygen -t rsa -b 4096 -C \"your_github_email@example.com\"`\n3. Add deploy_key.pub to the repository https://github.com/Open-EO/openeo-result-validation-engine/settings/keys\n4. Encrypt the deploy_key.pub file with the Travis CLI `travis encrypt-file deploy_key --add (This will generate an encryption label)`\n5. Adjust encryption label in .travis.yml line 21\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fopen-eo%2Fopeneo-result-validation-engine","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fopen-eo%2Fopeneo-result-validation-engine","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fopen-eo%2Fopeneo-result-validation-engine/lists"}