{"id":20814183,"url":"https://github.com/ahuarte47/geodataflow","last_synced_at":"2025-07-19T03:04:54.669Z","repository":{"id":62224679,"uuid":"557589412","full_name":"ahuarte47/geodataflow","owner":"ahuarte47","description":"Geoprocessing framework for geographical \u0026 Earth Observation (EO) data","archived":false,"fork":false,"pushed_at":"2024-12-27T19:27:58.000Z","size":3442,"stargazers_count":64,"open_issues_count":1,"forks_count":5,"subscribers_count":3,"default_branch":"main","last_synced_at":"2025-06-26T04:07:01.795Z","etag":null,"topics":["api-rest","earth-observation","geospatial","gis","python","remote-sensing"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/ahuarte47.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGES.rst","contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":"CODE_OF_CONDUCT.md","threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":"AUTHORS.rst","dei":null,"publiccode":null,"codemeta":null}},"created_at":"2022-10-26T00:24:15.000Z","updated_at":"2025-04-23T06:29:51.000Z","dependencies_parsed_at":"2025-02-23T07:11:27.012Z","dependency_job_id":null,"html_url":"https://github.com/ahuarte47/geodataflow","commit_stats":{"total_commits":63,"total_committers":1,"mean_commits":63.0,"dds":0.0,"last_synced_commit":"f6628c21c2d5bcb827eaac887c4fd517483a88ed"},"previous_names":[],"tags_count":7,"template":false,"template_full_name":null,"purl":"pkg:github/ahuarte47/geodataflow","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ahuarte47%2Fgeodataflow","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ahuarte47%2Fgeodataflow/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ahuarte47%2Fgeodataflow/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ahuarte47%2Fgeodataflow/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/ahuarte47","download_url":"https://codeload.github.com/ahuarte47/geodataflow/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ahuarte47%2Fgeodataflow/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":265879196,"owners_count":23843082,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["api-rest","earth-observation","geospatial","gis","python","remote-sensing"],"created_at":"2024-11-17T21:10:24.918Z","updated_at":"2025-07-19T03:04:54.644Z","avatar_url":"https://github.com/ahuarte47.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# GeodataFlow\n\nGeospatial processing framework for geographical \u0026 Earth Observation (EO) data in Python.\n\n**GeodataFlow** is a Geoprocessing framework for fetching, translating and manipulating Geospatial data (*Raster*, *Vector*, *EO/STAC collections*) by using a *Pipeline* or sequence of operations on input data. It is very much like the [_GDAL_](https://gdal.org/) library which handles raster and vector data.\n\nThe project is split up into several namespace packages or components:\n\n* [geodataflow.core](geodataflow/core/)\n\n  The main subpackage of **GeodataFlow** which implements basic building blocks (Pipeline engine \u0026 Modules) and commonly used functionalities.\n\n* [geodataflow.api](geodataflow/api/)\n\n  WebAPI component using [_FastAPI_](https://fastapi.tiangolo.com/) which provides access to GeodataFlow backend via API REST calls.\n\n  ![api](geodataflow/api/docs/api.png)\n\n* [workbench/ui](geodataflow/ui/)\n\n  GeodataFlow **Workbench** is a static _javascript_ application for users easily draw and run their own Workflows in the Web Browser.\n\n  ![workbench](geodataflow/ui/docs/workbench.png)\n\n  NOTE: There is no any installer for GeodataFlow **Workbench** yet, but you can test it loading the [docker-compose.yml](docker-compose.yml). Please, read related section below.\n\nBackends:\n\n* [spatial](geodataflow/spatial/)\n\n  Installs the `geodataflow.spatial` backend implementation for GeodataFlow using _GDAL/OGR_.\n\n* [dataframes](geodataflow/dataframes/)\n\n  Installs the `geodataflow.dataframes` backend implementation for GeodataFlow using _Geopandas_.\n\n* _pySpark_, _Geospatial SQL_, _... ?_\n\nVideos demostrating **GeodataFlow**:\n* [What is GeodataFlow](https://www.youtube.com/watch?v=jgWlyff2K00\u0026list=PLEmI_ksy4QHaAg64XbwnHFpwaLc8Hbr4-\u0026index=1)\n* [Tranforming Features](https://www.youtube.com/watch?v=IJljavT-BZg\u0026list=PLEmI_ksy4QHaAg64XbwnHFpwaLc8Hbr4-\u0026index=2)\n* [Reading items from a STAC Catalog](https://www.youtube.com/watch?v=XUJVZaMEDtc\u0026list=PLEmI_ksy4QHaAg64XbwnHFpwaLc8Hbr4-\u0026index=3)\n* [Downloading a NDVI raster from a STAC Catalog](https://www.youtube.com/watch?v=NodnUUYZJkU\u0026list=PLEmI_ksy4QHaAg64XbwnHFpwaLc8Hbr4-\u0026index=4)\n* [Filtering Features applying Spatial Relations](https://www.youtube.com/watch?v=5OxrXffwUeM\u0026list=PLEmI_ksy4QHaAg64XbwnHFpwaLc8Hbr4-\u0026index=5)\n* [Plotting a NDVI Time Series Graph of a polygon](https://www.youtube.com/watch?v=qKuUW-PXizg\u0026list=PLEmI_ksy4QHaAg64XbwnHFpwaLc8Hbr4-\u0026index=6)\n* [Loading Datasets from Google Earth Engine (GEE)](https://www.youtube.com/watch?v=qKuUW-PXizg\u0026list=PLEmI_ksy4QHaAg64XbwnHFpwaLc8Hbr4-\u0026index=7)\n* [Preview the stages of a Workflow in a Table or Map](https://www.youtube.com/watch?v=qKuUW-PXizg\u0026list=PLEmI_ksy4QHaAg64XbwnHFpwaLc8Hbr4-\u0026index=8)\n\n### Workflow examples\n\nAssuming you are using `geodataflow.spatial` (GDAL/OGR) as active backend implementation, **GeodataFlow** can run workflows as the following:\n\n+ Converting a Shapefile to GeoPackage:\n  ```bash\n  # ==============================================================\n  # Pipeline sample to convert a Shapefile to GeoPackage.\n  # ==============================================================\n  {\n    \"pipeline\": [\n      {\n        \"type\": \"FeatureReader\",\n        \"connectionString\": \"input.shp\"\n      },\n      # Extract the Centroid of input geometries.\n      {\n        \"type\": \"GeometryCentroid\"\n      },\n      # Transform CRS of geometries.\n      {\n        \"type\": \"GeometryTransform\",\n        \"sourceCrs\": 4326,\n        \"targetCrs\": 32630\n      },\n      # Save features to Geopackage.\n      {\n        \"type\": \"FeatureWriter\",\n        \"connectionString\": \"output.gpkg\"\n      }\n    ]\n  }\n  ```\n\n+ Fetching metadata of a S2L2A Product (STAC):\n  ```bash\n  # ==============================================================\n  # Pipeline sample to fetch metadata of a S2L2A Product (STAC).\n  # ==============================================================\n  {\n    \"pipeline\": [\n      {\n        \"type\": \"FeatureReader\",\n\n        # Define the input AOI in an embedded GeoJson.\n        \"connectionString\": {\n          \"type\": \"FeatureCollection\",\n          \"crs\": {\n            \"type\": \"name\",\n            \"properties\": { \"name\": \"EPSG:4326\" }\n          },\n          \"features\": [\n            {\n              \"type\": \"Feature\",\n              \"properties\": { \"id\": 0, \"name\": \"My AOI for testing\" },\n              \"geometry\": {\n                \"type\": \"Polygon\",\n                \"coordinates\": [[\n                    [-1.746826,42.773227],\n                    [-1.746826,42.860866],\n                    [-1.558685,42.860866],\n                    [-1.558685,42.773227],\n                    [-1.746826,42.773227]\n                ]]\n              }\n            }\n          ]\n        }\n      },\n      # Transform CRS of geometries.\n      {\n        \"type\": \"GeometryTransform\",\n        \"sourceCrs\": 4326,\n        \"targetCrs\": 32630\n      },\n      # Fetch metadata of EO Products that match one SpatioTemporial criteria.\n      {\n        \"type\": \"EOProductCatalog\",\n\n        \"driver\": \"STAC\",\n        \"provider\": \"https://earth-search.aws.element84.com/v0/search\",\n        \"product\": \"sentinel-s2-l2a-cogs\",\n\n        \"startDate\": \"2021-09-25\",\n        \"endDate\": \"2021-10-05\",\n        \"closestToDate\": \"2021-09-30\",\n        \"filter\": \"\",\n\n        \"preserveInputCrs\": true\n      },\n      # Save features to Geopackage.\n      {\n        \"type\": \"FeatureWriter\",\n        \"connectionString\": \"output.gpkg\"\n      }\n    ]\n  }\n  ```\n\n## Installation\n\nBecause **GeodataFlow** is composed by several namespace packages, some of them are optional (e.g. Backend implementations). You will need to install the ones you want by adding them as an extra to the command-line that runs the installer.\n\nIn order to read and write Cloud Optimized Geotiffs (COG), GDAL version 3.1 or greater is required. If your system GDAL is older than version 3.1, consider using Docker or Conda to get a modern GDAL.\n\n### Using pypi\n\nTo install the latest stable version from [_pypi_](https://pypi.org/), write this in the command-line:\n\n```bash\n\u003e pip install geodataflow[api,dataframes,eodag,gee]\n```\n\nThe `geodataflow` package installs `geodataflow.core` and `geodataflow.spatial` ones by default. You can use namespace package installers as well  (e.g. [api](/geodataflow/api/README.md#installation)), they have the same effect than the generic one.\n\nOptional extras for Backends:\n\n* eodag\n\n  EODAG - [Earth Observation Data Access Gateway](https://eodag.readthedocs.io/en/stable/) is a Python package for searching and downloading remotely sensed images while offering an unified API for data access regardless of the data provider.\n\n* gee\n\n  GEE - [Google Earth Engine API](https://developers.google.com/earth-engine) is a geospatial processing service. With _Earth Engine_, you can perform geospatial processing at scale, powered by Google Cloud Platform. _GEE_ requires authentication, please, read available documentation [here](https://developers.google.com/earth-engine/guides/python_install#authentication).\n\n\nTo view all available CLI tool commands and options:\n```bash\n\u003e geodataflow --help\n```\n\nListing all available modules:\n```bash\n\u003e geodataflow --modules\n```\n\nRun a workflow in the command-line interface:\n```bash\n\u003e geodataflow --pipeline_file \"/geodataflow/spatial/tests/data/test_eo_stac_catalog.json\"\n```\n\n### Using docker-compose\n\n[docker-compose.yml](docker-compose.yml) builds images and starts  GeodataFlow API and Workbench components to easily run Workflows with **GeodataFlow**.\n\n`PACKAGE_WITH_GEODATAFLOW_PIPELINE_CONTEXT` in the yml file indicates the backend implementation to load. The default value is `geodataflow.spatial`. If you prefer to use another backend, please, change it before starting.\n\nWrite in the command-line from the root folder of the project:\n```bash\n\u003e docker-compose up\n```\n\nThen, type in your favorite Web Browser:\n* http://localhost:9630/docs to check the interface of the REST WebAPI service.\n* http://localhost:9640/workbench.html to access to the Workbench UI designer and where you can design and run Workflows!\n\nTo remove all resources:\n```bash\n\u003e docker-compose down --rmi all -v --remove-orphans\n```\n\n## Testing\n\nEach package provides a collection of tests, run tests on `tests` folders to validate them.\n\n## Contribute\n\nHave you spotted a typo in our documentation? Have you observed a bug while running **GeodataFlow**? Do you have a suggestion for a new feature?\n\nDon't hesitate and open an issue or submit a pull request, contributions are most welcome!\n\n## License\n\n**GeodataFlow** is licensed under Apache License v2.0.\nSee [LICENSE](LICENSE) file for details.\n\n## Credits\n\n**GeodataFlow** is built on top of amazingly useful open source projects. See [NOTICE](../../NOTICE) file for details about those projects\nand their licenses.\n\nThank you to all the authors of these projects!\n\n## Authors\n\n**GeodataFlow** has been created by `Alvaro Huarte` \u003chttps://www.linkedin.com/in/alvarohuarte\u003e.\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fahuarte47%2Fgeodataflow","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fahuarte47%2Fgeodataflow","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fahuarte47%2Fgeodataflow/lists"}