{"id":16276179,"url":"https://github.com/cboettig/neonstore","last_synced_at":"2025-09-10T00:46:43.873Z","repository":{"id":43458272,"uuid":"245499373","full_name":"cboettig/neonstore","owner":"cboettig","description":":package: A local content-based storage system for NEON data","archived":false,"fork":false,"pushed_at":"2024-04-16T12:28:59.000Z","size":12968,"stargazers_count":9,"open_issues_count":5,"forks_count":4,"subscribers_count":3,"default_branch":"master","last_synced_at":"2025-04-03T08:22:16.587Z","etag":null,"topics":["database","ecology","neon-data","provenance","r"],"latest_commit_sha":null,"homepage":"https://cboettig.github.io/neonstore","language":"R","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"other","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/cboettig.png","metadata":{"files":{"readme":"README.Rmd","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null}},"created_at":"2020-03-06T19:20:49.000Z","updated_at":"2024-11-08T16:29:51.000Z","dependencies_parsed_at":"2023-09-24T14:52:46.320Z","dependency_job_id":"7479cba5-d8c0-4e02-aea3-d46509846142","html_url":"https://github.com/cboettig/neonstore","commit_stats":{"total_commits":402,"total_committers":3,"mean_commits":134.0,"dds":0.09203980099502485,"last_synced_commit":"0c5a5ab66f428cb9cab4b4fb7211fa95b4c440c1"},"previous_names":[],"tags_count":7,"template":false,"template_full_name":null,"purl":"pkg:github/cboettig/neonstore","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/cboettig%2Fneonstore","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/cboettig%2Fneonstore/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/cboettig%2Fneonstore/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/cboettig%2Fneonstore/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/cboettig","download_url":"https://codeload.github.com/cboettig/neonstore/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/cboettig%2Fneonstore/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":274365927,"owners_count":25272293,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-09-09T02:00:10.223Z","response_time":80,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["database","ecology","neon-data","provenance","r"],"created_at":"2024-10-10T18:47:17.404Z","updated_at":"2025-09-10T00:46:43.830Z","avatar_url":"https://github.com/cboettig.png","language":"R","funding_links":[],"categories":[],"sub_categories":[],"readme":"---\noutput: \n  github_document:\n    df_print: tibble\n---\n\n\u003c!-- README.md is generated from README.Rmd. Please edit that file --\u003e\n\n```{r setup, include = FALSE}\nknitr::opts_chunk$set(\n  collapse = TRUE,\n  comment = \"#\u003e\",\n  fig.path = \"man/figures/README-\",\n  out.width = \"100%\",\n  cache = FALSE\n)\nlibrary(neonstore)\nlibrary(dplyr)\nlibrary(stringr)\nSys.setenv(\"NEONSTORE_HOME\" = tempfile())\nSys.setenv(\"NEONSTORE_DB\" = tempfile())\n\n```\n\n# neonstore\n\n\u003c!-- badges: start --\u003e\n[![R build status](https://github.com/cboettig/neonstore/workflows/R-CMD-check/badge.svg)](https://github.com/cboettig/neonstore/actions)\n[![Codecov test coverage](https://codecov.io/gh/cboettig/neonstore/branch/master/graph/badge.svg)](https://app.codecov.io/gh/cboettig/neonstore?branch=master)\n[![CRAN status](https://www.r-pkg.org/badges/version/neonstore)](https://CRAN.R-project.org/package=neonstore)\n[![R-CMD-check](https://github.com/cboettig/neonstore/actions/workflows/R-CMD-check.yaml/badge.svg)](https://github.com/cboettig/neonstore/actions/workflows/R-CMD-check.yaml)\n\u003c!-- badges: end --\u003e\n\n`neonstore` provides quick access and persistent storage of NEON data tables. \n`neonstore` emphasizes simplicity and a clean data provenance trail, see \nProvenance section below.\n\n## Installation\n\nInstall the development version from [GitHub](https://github.com/) with:\n\n``` r\n# install.packages(\"devtools\")\ndevtools::install_github(\"cboettig/neonstore\")\n```\n## Quickstart\n\n```{r, message=FALSE}\nlibrary(neonstore)\nlibrary(tidyverse)\n```\n\nDiscover data products of interest:\n\n```{r}\nproducts \u003c- neon_products()\nproducts |\u003e\n  filter(str_detect(keywords, \"bird\")) |\u003e \n  select(productName, productCode)\n```\n\nYou may also prefer to explore the [NEON Data Portal](https://data.neonscience.org/) website interactively.\n\n## Download-based workflow\n\nOnce we have identified a data product code, we can download all associated data files, e.g. in the bird survey data.  Optionally, we can restrict this download to a set of sites or date ranges of interest, (see function documentation for details).  \n\n```{r}\nneon_download(\"DP1.10003.001\")\n```\n\nView your store of NEON products:\n\n```{r }\nneon_index()\n```\n\nThese files will persist between sessions, so you only need to download once\nor to retrieve updates. `neon_index()` can take arguments to filter by product\nor pattern (regular expression) in table name, e.g. `neon_index(table = \"brd\")`.\n\n## Database backend\n\n`neonstore` now supports a backend relation database as well. Import data from\nthe raw downloaded files using `neon_store()`:\n\n```{r}\nneon_store(product = \"DP1.10003.001\")\n```\n\nAccess an imported table using `neon_table()` instead of `neon_read()`:\n\n```{r}\nneon_table(\"brd_countdata\")\n```\n\n\nNote that we need to include the product name in the table name when accessing the database, as table names alone may not be unique. RStudio users can also list and explore all tables interactively in the Connections pane in RStudio using the function `neon_pane()`.  \n\n## Larger-than-RAM data\n\nWhen working across data from many sites or years simultaneously, it is easy for data to be too big for R to fit into working memory.  This is especially true when working with sensor data.  `neonstore` makes it easy to work with such data using dplyr-operations though.  Just include the option `lazy = TRUE`, and [most dplyr](https://dbplyr.tidyverse.org/reference/index.html) operations will execute quickly on disk instead (by leveraging the `dbplyr` backend and the power of the `duckdb` database). \n\n```{r}\nbrd \u003c- neon_table(\"brd_countdata\", lazy=TRUE)\n# unique species per site?\nbrd |\u003e \n  distinct(siteID, scientificName) |\u003e \n  count(siteID, sort=TRUE) |\u003e \n  collect()\n```\n\nUse the function `collect()` at the end of a chain of dplyr functions to bring the resulting data into R.\n\n## NEW: Cloud-based workflow\n\nIt is now possible to access data directly from NEON's cloud storage system without downloading.  (Note: this still must ping the NEON API to obtain the most recent list of files, and this list is subject to rate limits).  Like the local database approach, this strategy works for larger-than-RAM data, and can be substantially faster than downloading.  However, if you work frequently with the same data products and have ample disk space available, you will find the one-time wait for downloading to be faster.\n\n```{r}\nbrd \u003c- neon_cloud(\"brd_countdata\", product=\"DP1.10003.001\")\n\nbrd |\u003e \n  distinct(siteID, scientificName) |\u003e \n  count(siteID, sort=TRUE) |\u003e \n  collect()\n```\n\n\n\n## Note on API limits\n\nIf `neon_download()` exceeds the API request limit (with or without the token),\n`neonstore` will simply pause for the required amount of time to avoid \nrate-limit-based errors.  \n\n[The NEON API now rate-limits requests.](https://data.neonscience.org/data-api/rate-limiting/#api-tokens).\nUsing a personal token will increase the number of requests you can make before\nencountering this delay.  See link for directions on registering for a token.\nThen pass this token in `.token` argument of `neon_download()`,\nor for frequent use, add this token as  an environmental variable, `NEON_DATA`\nto your local `.Renviron` file in your user's home directory. \n`neon_download()` must first query each the API of each NEON site which collects\nthat product, for each month the product is collected.\n\n\n(It would be much more efficient on the NEON server if the API could take\nqueries of the from `/data/\u003cproduct\u003e/\u003csite\u003e`, and pool the results, rather than\nrequire each month of sampling separately!)\n\n```{r include=FALSE}\nunlink(\"my_neondata.zip\")\nunlink(\"index.csv\")\nSys.unsetenv(\"NEONSTORE_HOME\")\nSys.unsetenv(\"NEONSTORE_DB\")\n```\n\n## Non-stacking files and low-level interface\n\nAt it's core, `neonstore` is simply a mechanism to download files from the NEON API.\nWhile the `.csv` files from the Observation Systems (OS, e.g. bird count surveys),\nand Instrument Systems (e.g. aquatic sensors) are typically stacked into large\ntables, other products, such as the `.laz` and `.tif` images produced by the\nairborne observation platform (AOP) sensors such as LIDAR and cameras still\nrequire the user to work directly with the downloaded files returned\nby `neon_index()`.  Note that the local database can process Eddy Covariance \ndata (h5 files), but at present this does not work with `neon_cloud()`.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcboettig%2Fneonstore","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fcboettig%2Fneonstore","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcboettig%2Fneonstore/lists"}