{"id":18819078,"url":"https://github.com/geoscienceaustralia/dea-conflux","last_synced_at":"2025-09-07T18:48:35.508Z","repository":{"id":37693447,"uuid":"394521997","full_name":"GeoscienceAustralia/dea-conflux","owner":"GeoscienceAustralia","description":"Bulk polygon drills on the Open Data Cube.","archived":false,"fork":false,"pushed_at":"2024-10-03T10:45:37.000Z","size":451,"stargazers_count":6,"open_issues_count":6,"forks_count":5,"subscribers_count":8,"default_branch":"main","last_synced_at":"2025-03-27T13:46:11.961Z","etag":null,"topics":["gis","python"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/GeoscienceAustralia.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2021-08-10T04:17:02.000Z","updated_at":"2025-03-06T05:53:21.000Z","dependencies_parsed_at":"2023-10-23T06:25:59.170Z","dependency_job_id":"f66469ae-b1ad-4a2e-906b-fa3c927917c2","html_url":"https://github.com/GeoscienceAustralia/dea-conflux","commit_stats":null,"previous_names":[],"tags_count":18,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/GeoscienceAustralia%2Fdea-conflux","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/GeoscienceAustralia%2Fdea-conflux/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/GeoscienceAustralia%2Fdea-conflux/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/GeoscienceAustralia%2Fdea-conflux/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/GeoscienceAustralia","download_url":"https://codeload.github.com/GeoscienceAustralia/dea-conflux/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248797039,"owners_count":21163069,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["gis","python"],"created_at":"2024-11-08T00:20:08.499Z","updated_at":"2025-04-13T23:32:57.237Z","avatar_url":"https://github.com/GeoscienceAustralia.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# DEA Conflux\n\n![GitHub](https://img.shields.io/github/license/GeoscienceAustralia/dea-conflux)\n[![Test](https://github.com/GeoscienceAustralia/dea-conflux/actions/workflows/test.yml/badge.svg)](https://github.com/GeoscienceAustralia/dea-conflux/actions/workflows/test.yml) [![Lint](https://github.com/GeoscienceAustralia/dea-conflux/actions/workflows/lint.yml/badge.svg)](https://github.com/GeoscienceAustralia/dea-conflux/actions/workflows/lint.yml) [![Version](https://img.shields.io/docker/v/geoscienceaustralia/dea-conflux?label=version)](https://hub.docker.com/r/geoscienceaustralia/dea-conflux)\n[![codecov](https://codecov.io/gh/GeoscienceAustralia/dea-conflux/branch/main/graph/badge.svg)](https://app.codecov.io/gh/GeoscienceAustralia/dea-conflux)\n\nThis is a prototype tool for processing bulk polygon drills.\n\n- License: Apache 2.0\n- Contact:sai.ma@ga.gov.au , bex.dunn@ga.gov.au\n\n## Installation\n\nInstall with `pip`:\n\n```bash\npip install git+https://github.com/GeoscienceAustralia/dea-conflux.git\n```\n\nOr clone the repository and install from the local version.\n\n```bash\ngit clone https://github.com/GeoscienceAustralia/dea-conflux.git\ncd dea-conflux\npip install -e .\n```\n\n## Usage\n\nConflux provides a command-line tool `dea-conflux` for running each step of the polygon drill. Descriptions of the commands are available with `dea-conflux --help`. Conflux requires a Datacube configuration to work.\n\nTo run Conflux on a single scene, give it a scene UUID present in the Datacube, a plugin describing the polygon drill, a place to put output files, and a shapefile defining the polygons:\n\n```bash\ndea-conflux run-one --uuid SCENE_ID --plugin PLUGIN_PATH -o OUTPUT_PATH -s SHAPEFILE_PATH\n```\n\nConflux can also read from an AWS SQS queue. Messages must be the UUID of a scene.\n\n```bash\ndea-conflux run-from-queue --queue QUEUE_NAME --plugin PLUGIN_PATH -o OUTPUT_PATH -s SHAPEFILE_PATH\n```\n\nNote:\n\nThe dea-conflux only allows to run on `partial` mode right now. The polygons which partial overlap with the scene will also be processed.\n\n## Plugins\n\nA plugin defines the inputs and outputs of a polygon drill. It is a Python file with the extensions `.conflux.py`. Examples are provided in the `examples/` directory.\n\nPlugins must provide:\n\n- a product name for the drill,\n- a version string for the drill,\n- the resampling method to use on input rasters,\n- the CRS to project the input rasters into,\n- the resolution to resample the input rasters into,\n- a dictionary of input products and bands,\n- a transform function, and\n- a summarise function.\n\n#### Transform function\nThe transform is run to produce rasters to summarise and should contain operations like masking and band index calculation.\nTransform will be applied to the whole scene, not to a polygon if you are using a polygon.\n\n#### Summarise function\nThe summarise function aggregates a dataset into a number of measurements that summarise a polygon, i.e. the outputs of the drill.\nSummarise is applied to just the pixels in the polygon.\n\n## Pre-commit setup\n\n\t❯ pip install pre-commit\n\t❯ pre-commit install\n\nYour code will now be formatted and validated before each commit by running `pre-commit run -a`\n\n## The run-from-queue processing pipeline\n\nThe dea-conflux has two main processing functions. The `run_one` designs to do the local test and the `run_from_queue` design to run big scale processing. The `run_from_queue` needs a queue system as backend. We are using the [AWS SQS queue](https://aws.amazon.com/sqs/) as example.\n\n\u003cimg src=\"./doc/dea-conflux-control-flow.svg\"\u003e\n\n## The run_one processing pipeline\n\nThe `run_one` feature designs to do the single scene processing. The user have to provide the expected scene UUID in opendatacube.\n\n\u003cimg src=\"./doc/dea-conflux-control-flow-local.svg\"\u003e\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fgeoscienceaustralia%2Fdea-conflux","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fgeoscienceaustralia%2Fdea-conflux","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fgeoscienceaustralia%2Fdea-conflux/lists"}