https://github.com/kmarkert/bq-usgs-nwis-functions
Repo to set up to remotely call USGS NWIS API for gauge data into BigQuery tables
https://github.com/kmarkert/bq-usgs-nwis-functions
Last synced: 3 months ago
JSON representation
Repo to set up to remotely call USGS NWIS API for gauge data into BigQuery tables
- Host: GitHub
- URL: https://github.com/kmarkert/bq-usgs-nwis-functions
- Owner: KMarkert
- License: apache-2.0
- Created: 2023-05-20T16:29:55.000Z (about 2 years ago)
- Default Branch: main
- Last Pushed: 2023-05-31T17:49:00.000Z (almost 2 years ago)
- Last Synced: 2025-01-09T20:09:24.436Z (5 months ago)
- Language: Python
- Size: 349 KB
- Stars: 0
- Watchers: 1
- Forks: 1
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
- License: LICENSE
Awesome Lists containing this project
README
# bq-usgs-nwis-functions
This repository is for deploying cloud functions for to call [USGS National Water
Information System (NWIS)](https://waterdata.usgs.gov/nwis) service and load the
data into BigQuery tables for further analysis.
## Setup
If you do not already have a cloud project to deploy the functions to, you will
need to create one. You can either do this through the
[Cloud Console](https://console.cloud.google.com) or create on using
[`gcloud`](https://cloud.google.com/sdk/gcloud/reference/projects/create). Once
a project is created, the billing account will need to be linked to the project.
After the cloud project is ready, update the parameters in `config.sh` to the
information for your setup.
## Deploy
With the setup complete, to deploy the functions and setup BigQuery run the
following command:
```
bash deploy.sh
```
After executing the deploy command and everything completed successfully, you
should see an active cloud funtion and the external connection plus table with
routine in BigQuery.
## Run a Big Query job to get streamflow data
After everything is running, you can run a query to request streamflow timeseries
from a USGS site using the following example query:
```
SELECT
time,
'11425500' as site,
usgs_nwis.get_streamflow('11425500', time) as streamflow
FROM
UNNEST(
GENERATE_DATE_ARRAY(DATE('1980-01-01'), DATE('2020-12-31'), INTERVAL 1 DAY)
) AS time
```
If you plot the results, you should be able to see the following results:
To run a query with multiple sites:
```
SELECT
time,
site,
NULLIF(usgs_nwis.get_streamflow(site, time), -999999.0) AS streamflow
FROM
UNNEST ( ['11425500', '11446500']) AS site,
UNNEST( GENERATE_DATE_ARRAY(DATE('1980-01-01'), DATE('2022-12-31'), INTERVAL 1 DAY) ) AS time
```
## Acknowledgements
This repo was inspired by the great demo found here:
https://github.com/dojowahi/earth-engine-on-bigquery. A fair amount of code was
reused from the repo.