https://github.com/openwashdata/washopenresearch
dataset about open research data availability in Water, Sanitation and Hygiene (WASH)
https://github.com/openwashdata/washopenresearch
open-data open-research-data open-science openwashdata sanitation wash
Last synced: 4 months ago
JSON representation
dataset about open research data availability in Water, Sanitation and Hygiene (WASH)
- Host: GitHub
- URL: https://github.com/openwashdata/washopenresearch
- Owner: openwashdata
- License: cc-by-4.0
- Created: 2023-12-21T06:27:43.000Z (over 2 years ago)
- Default Branch: main
- Last Pushed: 2024-12-17T11:29:28.000Z (over 1 year ago)
- Last Synced: 2025-09-04T21:46:09.040Z (9 months ago)
- Topics: open-data, open-research-data, open-science, openwashdata, sanitation, wash
- Language: Jupyter Notebook
- Homepage: https://openwashdata.github.io/washopenresearch/
- Size: 18.5 MB
- Stars: 3
- Watchers: 1
- Forks: 0
- Open Issues: 1
-
Metadata Files:
- Readme: README.Rmd
- License: LICENSE.md
- Citation: CITATION.cff
Awesome Lists containing this project
README
---
output: github_document
always_allow_html: true
editor_options:
markdown:
wrap: 72
chunk_output_type: console
---
```{r, include = FALSE}
knitr::opts_chunk$set(
collapse = TRUE,
comment = "#>",
fig.path = "man/figures/README-",
out.width = "100%",
message = FALSE,
warning = FALSE,
fig.retina = 2,
fig.align = 'center'
)
library(tidyverse)
library(wordcloud2)
```
# washopenresearch
[](https://creativecommons.org/licenses/by/4.0/)
[](https://github.com/openwashdata/washopenresearch/actions/workflows/R-CMD-check.yaml)
[](https://zenodo.org/doi/10.5281/zenodo.11185699)
The goal of washopenresearch is to provide an overview of open research
data related to Water Sanitation and Hygiene (WASH). The current version
contains two datasets from the following sources:
- `washdev`: Open access journal [*Journal of Water, Sanitation and
Hygiene for Development*](https://iwaponline.com/washdev)
- `uncnewsletter`: Research section of the newsletter [North Carolina
Water
News](https://waterinstitute.unc.edu/our-work/nc-water-news-newsletter)
{width="515"}
## Installation
You can install the development version of washopenresearch from
[GitHub](https://github.com/) with:
``` r
# install.packages("devtools")
devtools::install_github("openwashdata/washopenresearch")
```
Alternatively, you can download the individual datasets as a CSV or XLSX
file from the table below.
```{r, echo=FALSE, message=FALSE, warning=FALSE}
extdata_path <- "https://github.com/openwashdata/washopenresearch/raw/main/inst/extdata/"
read_csv("data-raw/dictionary.csv") |>
distinct(file_name) |>
dplyr::mutate(file_name = str_remove(file_name, ".rda")) |>
dplyr::rename(dataset = file_name) |>
mutate(
CSV = paste0("[Download CSV](", extdata_path, dataset, ".csv)"),
XLSX = paste0("[Download XLSX](", extdata_path, dataset, ".xlsx)")
) |>
knitr::kable()
```
## Data
The package provides access to two datasets `washdev` and
`uncnewsletter`. Each dataset collects information on scientific
articles about (1) article metadata (e.g. title, first author,
correspondence author), (2) supplementary material information, (3) data
availability statement, and (4) semantic information (e.g. keywords).
```{r}
library(washopenresearch)
```
### washdev
The dataset `washdev` contains data on open access articles of the
*Journal of Water, Sanitation & Hygiene for Development* (Vol.1 Issue
1 - Vol.13 Issue 11). It has `r nrow(washdev)` observations from March
2011 to November 2023.
```{r}
washdev |>
head(3) |>
gt::gt() |>
gt::as_raw_html()
```
For an overview of the variable names, see the following table.
```{r echo=FALSE, message=FALSE, warning=FALSE}
readr::read_csv("data-raw/dictionary.csv") |>
dplyr::filter(file_name == "washdev.rda") |>
dplyr::select(variable_name:description) |>
knitr::kable() |>
kableExtra::kable_styling("striped") |>
kableExtra::scroll_box(height = "200px")
```
### uncnewsletter
The dataset `uncnewsletter` contains data on a curated list of articles
published at the Research section of the newsletter North Carolina Water
News. It has `r nrow(uncnewsletter)` observations from 2020 to 2023.
```{r}
uncnewsletter |>
head(3) |>
gt::gt() |>
gt::as_raw_html()
```
For an overview of the variable descriptions, see the following table.
```{r echo=FALSE, message=FALSE, warning=FALSE}
readr::read_csv("data-raw/dictionary.csv") |>
dplyr::filter(file_name == "uncnewsletter.rda") |>
dplyr::select(variable_name:description) |>
knitr::kable() |>
kableExtra::kable_styling("striped") |>
kableExtra::scroll_box(height = "200px")
```
## Example
### washdev
1. What are the top 10 countries(or regions) the first authors from in
the *Journal of Water, Sanitation and Hygiene for Development*?
```{r}
library(washopenresearch)
washdev |>
filter(!is.na(first_author_affiliation_country)) |>
group_by(first_author_affiliation_country) |>
summarise(count=n()) |>
arrange(desc(count)) |>
head(10) |>
ggplot() +
geom_col(aes(x = reorder(first_author_affiliation_country, count),
y = count)) +
labs(title = "Top 10 countries of first author",
subtitle = "in the Journal of Water, Sanitation and Hygiene for Development",
x = "First Author Country", y = "Count") +
scale_x_discrete(labels = scales::label_wrap(15))+
coord_flip() +
theme_classic()
```
2. What are the top choices of keywords in WASH Dev?
Each publication may provide a list of keywords, typically 5-7, to
summarize the topics of the article. Here we compile all keywords and
calculate their frequency to be used.
```{r washdev_keyword_frequency, echo=TRUE}
keywords_freq <- washdev$keywords |>
unlist() |>
str_to_lower() |>
table() |>
as.data.frame() |>
as_tibble() |>
arrange(desc(Freq))
# Top 20 keywords
ggplot(data = head(keywords_freq, 20)) +
geom_bar(aes(x = reorder(Var1, Freq), y=Freq), stat = "identity") +
coord_flip() +
labs(title = "Top 20 Keywords in WASH Dev Journal", x = "Keywords", y = "Count") +
theme_bw()
```
```{r washdev_wordcloud, echo=FALSE, eval=FALSE}
keywords_freq <- keywords_freq |>
rename(word=Var1, freq=Freq) |>
as.data.frame()
wc <- wordcloud2(keywords_freq)
library("webshot")
library("htmlwidgets")
# webshot::install_phantomjs()
saveWidget(wc, "man/figures/wc.html", selfcontained = F)
webshot("man/figures/wc.html", "man/figures/washdev_wordcloud.png", delay = 3)
```
### uncnewsletter
1. What are the top 10 source websites of the publications selected by
the newsletter?
```{r}
uncnewsletter |>
group_by(url_source) |>
summarise(count=n()) |>
arrange(desc(count)) |>
head(10) |>
ggplot() +
geom_col(aes(x = reorder(url_source, count),
y = count)) +
labs(title = "Top 10 publication websites",
subtitle = "in the selection of North Carolina Water News",
x = "Website URL", y = "Count") +
scale_x_discrete(labels = scales::label_wrap(15))+
coord_flip() +
theme_classic()
```
## Method
We describe the raw data collection procedure of each dataset in this
section. To reproduce the collection, you need to have python3 installed
and install python libraries
```
pip install requirements.txt
```
### washdev
The collection of `washdev` is via web scraping using Python. The script
can be found in `inst/python/washdev_scraping.py`. First, each
publication link is scraped from iterating the table of contents of all
volumes. This step delivers a table containing the variables paper ID,
volume number, issue number, publication url, journal title, publication
title, and published year. This table will be merged to get the final
dataset.
Then, for each publication, we retrieve the needed variables from the
publication's html file using the publication url. The retrieval is
rule-based to find the relevant fields (e.g. supplementary materials)
and extract the value.
### uncnewsletter
The collection of `uncnewsletter` is a combination of web scraping and
manual annotation. We first use the newsletter archive to scrape all
publication website links. The code can be found at
`inst/python/uncnewsletter_scraping.py`. Two annotators worked on the
manual extraction of the needed variables on these publications. For
each publication, an annotator follows the guide to fill in the value on
an collaborative spreadsheet. The guide is converted into the data
dictionary for this dataset.
## License
Data are available as
[CC-BY](https://github.com/openwashdata/washopenresearch/blob/main/LICENSE.md).
## Citation
Please cite this package using:
```{r}
citation("washopenresearch")
```