{"id":13737678,"url":"https://github.com/sentinel-hub/eo-grow","last_synced_at":"2025-04-07T11:04:07.501Z","repository":{"id":36962171,"uuid":"452658982","full_name":"sentinel-hub/eo-grow","owner":"sentinel-hub","description":"Earth observation framework for scaled-up processing in Python","archived":false,"fork":false,"pushed_at":"2025-03-18T10:21:49.000Z","size":2784,"stargazers_count":42,"open_issues_count":0,"forks_count":2,"subscribers_count":7,"default_branch":"main","last_synced_at":"2025-03-31T10:01:35.940Z","etag":null,"topics":["eo-data","eo-research","machine-learning","python"],"latest_commit_sha":null,"homepage":"https://eo-grow.readthedocs.io/en/latest/","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/sentinel-hub.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2022-01-27T11:39:50.000Z","updated_at":"2025-03-11T15:35:03.000Z","dependencies_parsed_at":"2023-10-11T08:14:40.892Z","dependency_job_id":"0d133266-4aaf-4eff-a23f-b5730b545543","html_url":"https://github.com/sentinel-hub/eo-grow","commit_stats":{"total_commits":367,"total_committers":7,"mean_commits":52.42857142857143,"dds":0.3215258855585831,"last_synced_commit":"ed5978c9bac86cfa74351cde095a62bd60fc0582"},"previous_names":[],"tags_count":32,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sentinel-hub%2Feo-grow","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sentinel-hub%2Feo-grow/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sentinel-hub%2Feo-grow/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sentinel-hub%2Feo-grow/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/sentinel-hub","download_url":"https://codeload.github.com/sentinel-hub/eo-grow/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":247640460,"owners_count":20971557,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["eo-data","eo-research","machine-learning","python"],"created_at":"2024-08-03T03:01:57.022Z","updated_at":"2025-04-07T11:04:07.477Z","avatar_url":"https://github.com/sentinel-hub.png","language":"Python","funding_links":[],"categories":["Python","Geospatial and Earth Observation"],"sub_categories":[],"readme":"[![Package version](https://badge.fury.io/py/eo-grow.svg)](https://pypi.org/project/eo-grow)\n[![Supported Python versions](https://img.shields.io/pypi/pyversions/eo-grow.svg?style=flat-square)](https://pypi.org/project/eo-grow)\n[![Build Status](https://github.com/sentinel-hub/eo-grow/actions/workflows/ci_action.yml/badge.svg?branch=main)](https://github.com/sentinel-hub/eo-grow/actions)\n[![Docs status](https://readthedocs.org/projects/eo-grow/badge/?version=latest)](https://eo-grow.readthedocs.io/en/latest/)\n[![License](https://img.shields.io/pypi/l/eo-grow.svg)](https://github.com/sentinel-hub/eo-grow/blob/master/LICENSE)\n[![Overall downloads](http://pepy.tech/badge/eo-grow)](https://pepy.tech/project/eo-grow)\n[![Last month downloads](https://pepy.tech/badge/eo-grow/month)](https://pepy.tech/project/eo-grow)\n[![Code coverage](https://codecov.io/gh/sentinel-hub/eo-grow/branch/main/graph/badge.svg)](https://codecov.io/gh/sentinel-hub/eo-grow)\n\n# eo-grow\n**Earth observation framework for scaled-up processing in Python.**\n\nAnalyzing Earth Observation (EO) data is complex and solutions often require custom tailored algorithms. In the EO domain most problems come with an additional challenge: *How do we apply the solution on a larger scale?*\n\nWorking with EO data is made easy by the [`eo-learn`](https://github.com/sentinel-hub/eo-learn) package, while the **`eo-grow`** package takes care of running the solutions at a large scale. In **`eo-grow`** an `EOWorkflow` based solution is wrapped in a **pipeline** object, which takes care of parametrization, logging, storage, multi-processing, EOPatch management and more. However, **pipelines** are not necessarily bound to `EOWorkflow` execution and can be used for other tasks such as training ML models.\n\nFeatures of **`eo-grow`** include:\n- Direct use of `EOWorkflow` procedures.\n- Parametrizing workflows by using validated configuration files, making executions easy to reproduce and adjust.\n- Easy use of both local and AWS S3 storage with no required code adaptation.\n- Splitting large areas of interest into grids and defining collections of EOPatches.\n- Workflows can be run either single-process, multi-process, or even on multiple machines (by using `ray` clusters).\n- Execution reports and customizable logging.\n- Options for skipping already processed data when re-running a pipeline.\n- Offers a CLI interface for running pipelines, validating configuration files, and generating templates.\n- A collection of basic pipelines, with methods that can be overridden to tailor to a large amount of use-cases.\n\n\n## General Structure Overview\n\nThe core object of **`eo-grow`** is the `Pipeline`. Each pipeline has a `run_procedure` method, which is executed after the pipeline is set up. By default, the `run_procedure` executes an `EOWorkflow` which is built by the (user-defined) `build_workflow` method.\n\nEach pipeline is linked to so called *managers*:\n- `StorageManager` handles loading and saving of files,\n- `AreaManager` defines the area of interest and how it should be split into EOPatches,\n- `LoggingManager` provides control over logging.\n\n![eo-grow-structure](docs/source/figures/eo-grow-structure.png)\n\nManagers and pipelines usually require a large amount of parameters (setting storage paths, configuring log parameters, etc.), which are provided in `.json` configuration files. Each **`eo-grow`** object contains a special `Schema` class, which is a `pydantic` model describing the parameters of the object. Config files are then validated before execution to catch issues early. Templates for config files can be generated with the `eogrow-template` CLI command.\n\nTo make config files easier to write **`eo-grow`** uses a simple [config language](https://github.com/sentinel-hub/eo-grow/tree/main/docs/source/config-language.md) that supports importing other configs, variables, and more.\n\n\n## Install\n\n### PyPi distribution\n\nThe **`eo-grow`** package requires Python version `\u003e= 3.8` and can be installed with\n\n```\npip install eo-grow\n```\n\n\n## Command Line Interface\n\nRunning pipelines is easiest by using the CLI provided by **`eo-grow`**. For all options use the `--help` flag with each command.\n\n- `eogrow \u003cconfig\u003e` executes the pipeline defined in the `\u003cconfig\u003e` file.\n- `eogrow-validate \u003cconfig\u003e` only performs validation of the `\u003cconfig\u003e` file.\n- `eogrow-test \u003cconfig\u003e` initializes the pipeline/object but does not run it. Useful for testing if managers are set correctly or for generating area-split grids.\n- `eogrow-ray \u003ccluster\u003e \u003cconfig\u003e` executes the pipeline defined in `\u003cconfig\u003e` on the active Ray cluster defined by the `\u003ccluster\u003e` file.\n- `eogrow-template \u003cimport path\u003e \u003ctemplate\u003e` generates a template config for the object specified by the `\u003cimport path\u003e` and saves it to the `\u003ctemplate\u003e` file (or outputs it directly if `\u003ctemplate\u003e` is not provided).\n\n## Documentation\n\nFor more information on the package visit [readthedocs](https://eo-grow.readthedocs.io/en/latest/).\n\nExplanatory examples can be found [here](https://github.com/sentinel-hub/eo-grow/tree/main/examples).\n\nMore details on the config language used by **`eo-grow`** can be found [here](https://github.com/sentinel-hub/eo-grow/tree/main/docs/source/config-language.md).\n\n\n## Questions and Issues\n\nFeel free to ask questions about the package and its use cases at [Sentinel Hub forum](https://forum.sentinel-hub.com/) or raise an issue on [GitHub](https://github.com/sentinel-hub/eo-grow/issues).\n\n\n## License\n\nSee [LICENSE](https://github.com/sentinel-hub/eo-grow/blob/main/LICENSE).\n\n## Acknowledgements\n\nThis project has received funding from the European Union’s Horizon 2020 research and innovation programme under Grant Agreement No. 101004112. The project is co-funded also by the European Union’s Horizon 2020 research and innovation programme under Grant Agreement No. 101086461.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsentinel-hub%2Feo-grow","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fsentinel-hub%2Feo-grow","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsentinel-hub%2Feo-grow/lists"}