{"id":21318925,"url":"https://github.com/sydney-informatics-hub/dataharvester","last_synced_at":"2025-07-12T03:31:13.943Z","repository":{"id":59648881,"uuid":"537453055","full_name":"Sydney-Informatics-Hub/dataharvester","owner":"Sydney-Informatics-Hub","description":"Preprocess, aggregate, visualise and download geospatial data from a range of Australian (and international) data sources","archived":false,"fork":false,"pushed_at":"2023-04-12T03:43:12.000Z","size":4455,"stargazers_count":9,"open_issues_count":4,"forks_count":3,"subscribers_count":1,"default_branch":"main","last_synced_at":"2024-04-16T03:53:29.524Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"https://sydney-informatics-hub.github.io/dataharvester/","language":"R","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":"januarharianto/dataharvester","license":"gpl-3.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/Sydney-Informatics-Hub.png","metadata":{"files":{"readme":"README.Rmd","changelog":null,"contributing":null,"funding":null,"license":"LICENSE.md","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2022-09-16T12:42:01.000Z","updated_at":"2024-04-09T01:57:04.000Z","dependencies_parsed_at":"2023-02-19T06:15:26.308Z","dependency_job_id":null,"html_url":"https://github.com/Sydney-Informatics-Hub/dataharvester","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Sydney-Informatics-Hub%2Fdataharvester","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Sydney-Informatics-Hub%2Fdataharvester/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Sydney-Informatics-Hub%2Fdataharvester/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Sydney-Informatics-Hub%2Fdataharvester/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/Sydney-Informatics-Hub","download_url":"https://codeload.github.com/Sydney-Informatics-Hub/dataharvester/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":225790750,"owners_count":17524654,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-21T19:28:34.779Z","updated_at":"2024-11-21T19:28:35.635Z","avatar_url":"https://github.com/Sydney-Informatics-Hub.png","language":"R","funding_links":[],"categories":[],"sub_categories":[],"readme":"---\noutput: github_document\n---\n\n\u003c!-- README.md is generated from README.Rmd. Please edit that file --\u003e\n\n```{r, include = FALSE}\nknitr::opts_chunk$set(\n  collapse = TRUE,\n  comment = \"#\u003e\",\n  fig.path = \"man/figures/README-\",\n  out.width = \"100%\"\n)\n```\n\n```{r echo=FALSE, fig.align=\"center\", out.width=300}\nknitr::include_graphics(\"man/figures/logo_r.png\")\n```\n\n---\n\n\u003c!-- badges: start --\u003e\n[![License](https://img.shields.io/badge/License-GPLV3-blue)](#license)\n[![pkgdown](https://github.com/Sydney-Informatics-Hub/dataharvester/actions/workflows/pkgdown.yaml/badge.svg)](https://github.com/Sydney-Informatics-Hub/dataharvester/actions/workflows/pkgdown.yaml)\n[![issues - dataharvester](https://img.shields.io/github/issues/Sydney-Informatics-Hub/dataharvester)](https://github.com/Sydney-Informatics-Hub/dataharvester/issues)\n[![R-CMD-check](https://github.com/Sydney-Informatics-Hub/dataharvester/actions/workflows/R-CMD-check.yaml/badge.svg)](https://github.com/Sydney-Informatics-Hub/dataharvester/actions/workflows/R-CMD-check.yaml)\n[![Codecov test coverage](https://codecov.io/gh/Sydney-Informatics-Hub/dataharvester/branch/main/graph/badge.svg)](https://app.codecov.io/gh/Sydney-Informatics-Hub/dataharvester?branch=main)\n\u003c!-- badges: end --\u003e\n\n\n---\n\n**Important: This is a pre-release version of `dataharvester` which is undergoing active development. Please report any bugs in the [issues](https://github.com/Sydney-Informatics-Hub/dataharvester/issues) tab.**\n\n`dataharvester` is an R interface to the [AgReFed](https://www.agrefed.org.au/) Geodata-Harvester.\nUse `dataharvester` to preprocess, aggregate, visualise and download geospatial data from a range of **Australian** (and international) data sources, including:\n\n- [Soil and Landscape Grid of Australia (SLGA)](https://is.gd/i8nF0Z)\n- [SILO Climate Database - Queensland Government, Australia](https://is.gd/ifJ8tB) \n- [Digital Elevation Model (DEM) of Australia](https://is.gd/ZLFwGs)\n- [Digital Earth Australia (DEA) Geoscience Earth Observations](https://is.gd/gRSlVG)\n- [GSKY Data Server for DEA Geoscience Earth Observations](https://is.gd/zFHxfD)\n- [Google Earth Engine](https://is.gd/VdO3Jx)\n\nWith connectivity to the Earth Engine API, perform petabyte-scale operations which include temporal cloud/shadow masking and automatic calculation of spectral indices (e.g. NDVI) for the following collections:\n\n- Landsat [5](https://developers.google.com/earth-engine/datasets/catalog/landsat-5) (TM), [7](https://developers.google.com/earth-engine/datasets/catalog/landsat-7) (ETM+), [8](https://developers.google.com/earth-engine/datasets/catalog/landsat-8) (OLI/TRS) and [9](https://developers.google.com/earth-engine/datasets/catalog/landsat-9) (OLI-2/TRS-2)\n- [Sentinel-2](https://developers.google.com/earth-engine/datasets/catalog/sentinel-2) (Surface Reflectance) and [Sentinel-3](https://developers.google.com/earth-engine/datasets/catalog/COPERNICUS_S3_OLCI) (Ocean and Land Colour Instrument)\n- [MODIS](https://developers.google.com/earth-engine/datasets/catalog/modis) (or Moderate Resolution Imaging Spectroradiometer) products\n\nFor [all other Earth Engine collections](https://developers.google.com/earth-engine/datasets/), while `dataharvester` does not provide officual support, users can still perform basic operations to filter, reduce, visualise and download data.\n\n## Demo\n\nBelow is an output when the following 3 lines of code is run on a configuration file, which downloads 16 images from 7 API sources:\n\n```r\nlibrary(dataharvester)\ninitialise_harvester(\"r-reticulate\")\nharvest(\"~/Documents/democonfig.yaml\", plot = TRUE, contour = FALSE)\n\n```\n\n\n\n![](man/figures/harvestdemo.gif)\n\n## Installation\n\nInstall the development version of this package from GitHub using `install_github()` from `remotes` or `devtools`:\n\n```{r install, eval=FALSE}\n# install.packages(\"remotes\") # uncomment and run this line if necessary\nremotes::install_github(\"sydney-informatics-hub/dataharvester\")\n\nlibrary(dataharvester) # load package\n```\n\n## Examples\n\n### \"Headless\" run\n\nRun `initialise_harvester()` after loading the package. The function helps you initialise the package, verifies package dependencies and connect to the Earth Engine API. Note that the default environment is `r-reticulate`, but you can provide a different name if this environment is being used in other projects.\n\nThe first initialisation may take a few minutes if dependencies need to be installed.\n\n```{r example, eval=FALSE}\nlibrary(dataharvester)\ninitialise_harvester(\"r-reticulate\", earthengine = TRUE)\n```\n\n**Note**: connecting to the Earth Engine API requires an existing Google account. Depending on your OS, a web browser popup may appear where you will be guided to connect to the API. Otherwise, a link will be provided, and you will need to follow the instructions to finalise the authentication process. More information can be found in the documentation.\n\nThen, run `harvest()`, which parses a YAML config file:\n\n```{r eg-harvest, eval=FALSE}\nharvest(path_to_config = \"path/to/config.yaml\")\n```\n\n### Manual downloads\n\nThe `harvest()` function is essentially a wrapper for all the API calls available in the Data-Harvester. Individual functions are available for each of the data sources. For example, the code below downloads Bulk Density data from SLGA for a specified region and saves it to the `downloads` folder in your working directory:\n\n```{r eg-manual, eval=FALSE}\nslga \u003c- download_slga(\n  layer = \"Bulk_Density\",\n  bounding_box = c(149.769345, -30.335861, 149.949173, -30.206271),\n  out_path = \"downloads/\",\n  resolution = 6,\n  depth_min = 0,\n  depth_max = 5\n)\n```\n\nAccess to the Google Earth Engine API is aimed at simplifying the most common tasks for beginners. For example, below is example code that performs all of the following (in 9 lines of code):\n\n- Access to the Google Earth Engine Data Catalog\n- Filter by date and coordinates\n- Scaling, offsetting, cloud and shadow masking\n- Automatically calculate Awesome Spectral Indices (e.g. NDVI)\n- Pixel stretching for image preview\n- Mapping of results\n- Downloading of image(s) at specified resolution and location\n- Splitting of large images automatically to allow file sizes that exceed the limit(s) set by the API (~32 MB)\n\n```r\n# Make sure that GEE API is initialised\ninitialise_harvester(\"r-reticulate\", earthengine = TRUE)\n# 1. Define dataset\nimg \u003c- collect_ee(\n  collection = \"LANDSAT/LC09/C02/T1_L2\",\n  coords = c(149.769345, -30.335861, 149.949173, -30.206271),\n  date = \"2021-06-01\",\n  end_date = \"2022-06-01\"\n)\n# 2. Preprocess (cloud masking, scale and offsetting enabled by default)\nimg \u003c- preprocess_ee(img, spectral = \"NDVI\")\n# 3. Visualise (optional)\nimg \u003c- map_ee(img, bands = \"NDVI\")\n# 4. Download at 100m/px resolution\nimg \u003c- download_ee(img, bands = \"NDVI\", scale = 100, out_path = \"downloads/\")\n\n```\nNote that the above functions can be chained/simplified using native (`|\u003e`) or `dplyr` (`%\u003e%`) pipes.\n\nThese functions are not meant to provide full functionality from the GEE API. At any time, a user may take an Earth Engine object and process it directly using the API - in this case, with [`rgee`](https://github.com/r-spatial/rgee) in R.\n\n```r\n# Extract ee.Image or ee.ImageCollection object\ngee_obj \u003c- img$ee_image\n\n# Now use `gee_obj` anywhere that can process Earth Engine Object data...\n```\n\n## Spatial/temporal buffering\n\n**NEW** Once data is downloaded, `dataharvester` provides functions to assist in data aggregation over space and time. *This will be documented soon.*\n\n\n```r\n\n#TODO\n\n```\n\n\n## Acknowledgments\n\n\nAcknowledgements are an important way for us to demonstrate the value we bring to your research. Your research outcomes are vital for ongoing funding of the Sydney Informatics Hub.\nIf you make use of this software for your research project, please include the following acknowledgement:\n\n\u003e “This research was supported by the Sydney Informatics Hub, a Core Research Facility of the University of Sydney, and the Agricultural Research Federation (AgReFed).\"\n\n## Attribution\n\nThis software was developed by the Sydney Informatics Hub, a core research facility of the University of Sydney, as part of the Data Harvesting project for the Agricultural Research Federation (AgReFed).\nAgReFed is supported by the Australian Research Data Commons (ARDC) and the Australian Government through the National Collaborative Research Infrastructure Strategy (NCRIS).\n\nWe would also like to acknowledge the use of the following Python packages in `dataharvester`, as we have used their code in the Data-Harvester: [geemap](https://github.com/giswqs/geemap), [wxee](https://github.com/aazuspan/wxee) and [eemont](https://github.com/davemlz/eemont).\n\n## License\n\n\u0026copy; 2022 The University of Sydney\n\nThis is free software: you can redistribute it and/or modify it under the terms of the GNU Lesser General Public License (LGPL version 3) as published by the Free Software Foundation.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsydney-informatics-hub%2Fdataharvester","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fsydney-informatics-hub%2Fdataharvester","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsydney-informatics-hub%2Fdataharvester/lists"}