Ecosyste.ms: Awesome
An open API service indexing awesome lists of open source software.
https://github.com/datastreamapp/schema
DataStream Schema
https://github.com/datastreamapp/schema
json-schema wqx
Last synced: 6 days ago
JSON representation
DataStream Schema
- Host: GitHub
- URL: https://github.com/datastreamapp/schema
- Owner: datastreamapp
- License: mit
- Created: 2018-02-06T19:37:25.000Z (almost 7 years ago)
- Default Branch: main
- Last Pushed: 2024-11-04T17:31:32.000Z (11 days ago)
- Last Synced: 2024-11-04T18:24:00.989Z (11 days ago)
- Topics: json-schema, wqx
- Language: JavaScript
- Homepage: https://datastream.org
- Size: 5.25 MB
- Stars: 12
- Watchers: 9
- Forks: 5
- Open Issues: 1
-
Metadata Files:
- Readme: docs/README.md
- License: LICENSE
Awesome Lists containing this project
README
DataStream Open Data Schema for Water Quality Data
Download the latest version in JSON Schema, Google Sheets, Excel template formats.
## DataStream
DataStream ([DataStream.org](https://datastream.org)) is an online open-access platform for sharing water quality and sediment quality data. Data is uploaded, stored and shared in DataStream’s Open Data Schema -- a model based on the WQX standard for the Exchange of Water Quality Data. DataStream is free to use and allows users to query, visualize, and download data in this standardized format. Water monitoring groups across Canada — including communities, researchers, Indigenous nations and governments at all levels — are using DataStream to publish their data.
DataStream was developed by [The Gordon Foundation](https://gordonfoundation.ca) and is carried out in collaboration with regional partners and monitoring networks. Data contributors maintain ownership of their data which are published under open data licenses.
## DataStream Open Data Schema (DS-WQX)
To ensure consistent formatting of water data and to avoid ambiguous or missing information, we developed an observation-level data schema based on the WQX standard for the Exchange of Water Quality Data.
The WQX schema was developed by the US Environmental Protection Agency (EPA) and the US Geological Society (USGS) and is an implementation of the ESAR (Environmental Sampling, Analysis and Results) data standard. It was designed to enable multiple monitoring entities to share results in a common format. In the US, the WQX schema is used on the US EPA’s Water Quality Portal to share over 340 million water quality data records data from 400 federal, state, tribal and other partners.
In 2018, The Gordon Foundation led a comprehensive review process to determine how this model could be adapted to best meet the needs of diverse water monitoring initiatives in Canada. DataStream’s science and technical advisory team, regional partners and collaborators, data contributors, government representatives as well as other members of the water community were engaged in this process.
DataStream’s open data schema (DS-WQX v1.0) was fully implemented across all regional DataStream platforms in 2019.
## Changelog
The DataStream open data schema will continue to evolve to meet user needs and therefore is subject to various updates over time (e.g. addition of new allowed values). To view the most recent version number and record of changes please see the [DataStream Upload Template](https://datastreamorg.sharepoint.com/:x:/s/Datastream/EaqcNGHom7BFlRi6bRY4VDoBy6ECq6v3bbUyeb0B3S3HGg?e=75aBTl).
## Dataset Metadata
In addition to the observation-level information in DS-WQX, DataStream uses dataset level metadata. [DataStream Metadata](https://github.com/datastreamapp/schema/tree/main/schemas/meta)
## Special Case Tests
In addition to our schema enforcing allowed values the column conditional logic; we have included additional check for common errors to the frontend flavour of our schema.
- `Dissolved oxygen (DO)` should not be in `%`
- `Dissolved oxygen saturation` should not be less than `0%`
- `Hardness` should not be less than or equal to `0`
- `pH` should be within `0` and `14`
- `Temperature` should be within `-100 degC` and `100 degC`
## Testing
We aim to have our test be as robust as possible. This is accomplished by having acceptance and rejection tests.
## Install
You can download the compiled DS-WQX schema from above.
Alternately, you can build it from the source to include in your project.
```bash
# Public
$ npm i
$ npm run build
$ cat primary.json
# Private
$ npm i @datastreamapp/schema
```
## Use
### CSV Template
The `csv` template follows `R` import/export best practices.
### JavaScript
```javascript
import validate from '@datastreamapp/schema'
const data = {}; // Single row of data
const valid = validate(data);
if (!valid) console.error(validate.errors)
```
### Schema Flavours
Supports JSON Schema Draft 2019-09 Specification in non-strict mode. Strict mode removes `if`, `then`, `additionalProperties` from the schema.
- `primary`: This includes only JSON schema specification supported parameters
- `frontend`: Includes value coercion and supplementary conditional checks
- `backend`: Includes all possible allowed values, value coercion and no conditional logic.
## Publish
```bash
# change version in package.json
npm test
cd dist
npm publish
```
## Contributing
### Commenting on the Schema
If you wish to comment on the schema please [open an issue](https://github.com/datastreamapp/schema/issues).
* For information on opening an issue review github's [creating an issue](https://help.github.com/en/github/managing-your-work-on-github/creating-an-issue) document
You can also email us at or visit us at:
* [DataStream](https://datastream.org)
* [Mackenzie DataStream](https://mackenziedatastream.ca)
* [Atlantic DataStream](https://atlanticdatastream.ca)
* [Lake Winnipeg DataStream](https://lakewinnipegdatastream.ca)
* [Great Lakes DataStream](https://greatlakesdatastream.ca)
* [Pacific DataStream](https://pacificdatastream.ca)
### Development
```bash
brew install nvm
nvm get 12
npm i
```
### Publishing
```bash
# update version in `package.json`
npm run test
cd dist
npm publish
```
## References
- [WQX](https://github.com/datastreamapp/wqx)
- [R Import/Export](https://cran.r-project.org/doc/manuals/r-release/R-data.html)
## Citation
If you use this schema, please cite it as below.
DataStream Initiative. ([Year]). DataStream WQX Schema ([current version]). GitHub. https://github.com/datastreamapp/schema