{"id":38654555,"url":"https://github.com/openwashdata/washopenresearch","last_synced_at":"2026-01-17T09:25:39.112Z","repository":{"id":214441216,"uuid":"734215029","full_name":"openwashdata/washopenresearch","owner":"openwashdata","description":"dataset about open research data availability in Water, Sanitation and Hygiene (WASH)","archived":false,"fork":false,"pushed_at":"2024-12-17T11:29:28.000Z","size":19372,"stargazers_count":3,"open_issues_count":1,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-09-04T21:46:09.040Z","etag":null,"topics":["open-data","open-research-data","open-science","openwashdata","sanitation","wash"],"latest_commit_sha":null,"homepage":"https://openwashdata.github.io/washopenresearch/","language":"Jupyter Notebook","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"cc-by-4.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/openwashdata.png","metadata":{"files":{"readme":"README.Rmd","changelog":null,"contributing":null,"funding":null,"license":"LICENSE.md","code_of_conduct":null,"threat_model":null,"audit":null,"citation":"CITATION.cff","codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2023-12-21T06:27:43.000Z","updated_at":"2024-12-17T11:29:32.000Z","dependencies_parsed_at":"2023-12-28T08:28:14.043Z","dependency_job_id":"bc59a22a-a969-4077-90bd-a85585f9e369","html_url":"https://github.com/openwashdata/washopenresearch","commit_stats":null,"previous_names":["openwashdata/washopenresearch"],"tags_count":1,"template":false,"template_full_name":null,"purl":"pkg:github/openwashdata/washopenresearch","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/openwashdata%2Fwashopenresearch","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/openwashdata%2Fwashopenresearch/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/openwashdata%2Fwashopenresearch/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/openwashdata%2Fwashopenresearch/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/openwashdata","download_url":"https://codeload.github.com/openwashdata/washopenresearch/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/openwashdata%2Fwashopenresearch/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":28505550,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-01-17T06:57:29.758Z","status":"ssl_error","status_checked_at":"2026-01-17T06:56:03.931Z","response_time":85,"last_error":"SSL_read: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["open-data","open-research-data","open-science","openwashdata","sanitation","wash"],"created_at":"2026-01-17T09:25:38.958Z","updated_at":"2026-01-17T09:25:39.072Z","avatar_url":"https://github.com/openwashdata.png","language":"Jupyter Notebook","funding_links":[],"categories":[],"sub_categories":[],"readme":"---\noutput: github_document\nalways_allow_html: true\neditor_options: \n  markdown: \n    wrap: 72\n  chunk_output_type: console\n---\n\n\u003c!-- README.md is generated from README.Rmd. Please edit that file --\u003e\n\n```{r, include = FALSE}\nknitr::opts_chunk$set(\n  collapse = TRUE,\n  comment = \"#\u003e\",\n  fig.path = \"man/figures/README-\",\n  out.width = \"100%\",\n  message = FALSE,\n  warning = FALSE,\n  fig.retina = 2,\n  fig.align = 'center'\n)\n\nlibrary(tidyverse)\nlibrary(wordcloud2)\n```\n\n# washopenresearch\n\n\u003c!-- badges: start --\u003e\n\n[![License: CC BY\n4.0](https://img.shields.io/badge/License-CC_BY_4.0-lightgrey.svg)](https://creativecommons.org/licenses/by/4.0/)\n[![R-CMD-check](https://github.com/openwashdata/washopenresearch/actions/workflows/R-CMD-check.yaml/badge.svg)](https://github.com/openwashdata/washopenresearch/actions/workflows/R-CMD-check.yaml)\n[![DOI](https://zenodo.org/badge/734215029.svg)](https://zenodo.org/doi/10.5281/zenodo.11185699)\n\n\u003c!-- badges: end --\u003e\n\nThe goal of washopenresearch is to provide an overview of open research\ndata related to Water Sanitation and Hygiene (WASH). The current version\ncontains two datasets from the following sources:\n\n-   `washdev`: Open access journal [*Journal of Water, Sanitation and\n    Hygiene for Development*](https://iwaponline.com/washdev)\n-   `uncnewsletter`: Research section of the newsletter [North Carolina\n    Water\n    News](https://waterinstitute.unc.edu/our-work/nc-water-news-newsletter)\n\n![](man/figures/washdev_wordcloud.png){width=\"515\"}\n\n## Installation\n\nYou can install the development version of washopenresearch from\n[GitHub](https://github.com/) with:\n\n``` r\n# install.packages(\"devtools\")\ndevtools::install_github(\"openwashdata/washopenresearch\")\n```\n\nAlternatively, you can download the individual datasets as a CSV or XLSX\nfile from the table below.\n\n```{r, echo=FALSE, message=FALSE, warning=FALSE}\n\nextdata_path \u003c- \"https://github.com/openwashdata/washopenresearch/raw/main/inst/extdata/\"\n\nread_csv(\"data-raw/dictionary.csv\") |\u003e \n  distinct(file_name) |\u003e \n  dplyr::mutate(file_name = str_remove(file_name, \".rda\")) |\u003e \n  dplyr::rename(dataset = file_name) |\u003e \n  mutate(\n    CSV = paste0(\"[Download CSV](\", extdata_path, dataset, \".csv)\"),\n    XLSX = paste0(\"[Download XLSX](\", extdata_path, dataset, \".xlsx)\")\n  ) |\u003e \n  knitr::kable()\n\n```\n\n## Data\n\nThe package provides access to two datasets `washdev` and\n`uncnewsletter`. Each dataset collects information on scientific\narticles about (1) article metadata (e.g. title, first author,\ncorrespondence author), (2) supplementary material information, (3) data\navailability statement, and (4) semantic information (e.g. keywords).\n\n```{r}\nlibrary(washopenresearch)\n```\n\n### washdev\n\nThe dataset `washdev` contains data on open access articles of the\n*Journal of Water, Sanitation \u0026 Hygiene for Development* (Vol.1 Issue\n1 - Vol.13 Issue 11). It has `r nrow(washdev)` observations from March\n2011 to November 2023.\n\n```{r}\nwashdev |\u003e \n  head(3) |\u003e \n  gt::gt() |\u003e\n  gt::as_raw_html()\n```\n\nFor an overview of the variable names, see the following table.\n\n```{r echo=FALSE, message=FALSE, warning=FALSE}\nreadr::read_csv(\"data-raw/dictionary.csv\") |\u003e\n  dplyr::filter(file_name == \"washdev.rda\") |\u003e\n  dplyr::select(variable_name:description) |\u003e \n  knitr::kable() |\u003e \n  kableExtra::kable_styling(\"striped\") |\u003e \n  kableExtra::scroll_box(height = \"200px\")\n```\n\n### uncnewsletter\n\nThe dataset `uncnewsletter` contains data on a curated list of articles\npublished at the Research section of the newsletter North Carolina Water\nNews. It has `r nrow(uncnewsletter)` observations from 2020 to 2023.\n\n```{r}\nuncnewsletter |\u003e \n  head(3) |\u003e \n  gt::gt() |\u003e\n  gt::as_raw_html()\n```\n\nFor an overview of the variable descriptions, see the following table.\n\n```{r echo=FALSE, message=FALSE, warning=FALSE}\nreadr::read_csv(\"data-raw/dictionary.csv\") |\u003e\n  dplyr::filter(file_name == \"uncnewsletter.rda\") |\u003e\n  dplyr::select(variable_name:description) |\u003e \n  knitr::kable() |\u003e \n  kableExtra::kable_styling(\"striped\") |\u003e \n  kableExtra::scroll_box(height = \"200px\")\n```\n\n\n## Example\n\n### washdev\n\n1.  What are the top 10 countries(or regions) the first authors from in\n    the *Journal of Water, Sanitation and Hygiene for Development*?\n\n```{r}\nlibrary(washopenresearch)\n\nwashdev |\u003e \n  filter(!is.na(first_author_affiliation_country)) |\u003e\n  group_by(first_author_affiliation_country) |\u003e\n  summarise(count=n()) |\u003e\n  arrange(desc(count)) |\u003e\n  head(10) |\u003e\n  ggplot() +\n    geom_col(aes(x = reorder(first_author_affiliation_country, count), \n                 y = count)) +\n    labs(title = \"Top 10 countries of first author\",\n        subtitle = \"in the Journal of Water, Sanitation and Hygiene for Development\",\n        x = \"First Author Country\", y = \"Count\") +\n    scale_x_discrete(labels = scales::label_wrap(15))+\n    coord_flip() +\n    theme_classic()\n```\n\n2.  What are the top choices of keywords in WASH Dev?\n\nEach publication may provide a list of keywords, typically 5-7, to\nsummarize the topics of the article. Here we compile all keywords and\ncalculate their frequency to be used.\n\n```{r washdev_keyword_frequency, echo=TRUE}\nkeywords_freq \u003c- washdev$keywords |\u003e\n    unlist() |\u003e\n    str_to_lower() |\u003e\n  table() |\u003e\n  as.data.frame() |\u003e\n  as_tibble() |\u003e\n  arrange(desc(Freq))\n\n# Top 20 keywords\nggplot(data = head(keywords_freq, 20)) +\n  geom_bar(aes(x = reorder(Var1, Freq), y=Freq), stat = \"identity\") +\n  coord_flip() +\n  labs(title = \"Top 20 Keywords in WASH Dev Journal\", x = \"Keywords\", y = \"Count\") +\n  theme_bw()\n```\n\n```{r washdev_wordcloud, echo=FALSE, eval=FALSE}\n\nkeywords_freq \u003c- keywords_freq |\u003e\n  rename(word=Var1, freq=Freq) |\u003e\n  as.data.frame()\n\nwc \u003c- wordcloud2(keywords_freq)\nlibrary(\"webshot\")\nlibrary(\"htmlwidgets\")\n# webshot::install_phantomjs()\nsaveWidget(wc, \"man/figures/wc.html\", selfcontained = F)\nwebshot(\"man/figures/wc.html\", \"man/figures/washdev_wordcloud.png\", delay = 3)\n```\n\n### uncnewsletter\n\n1.  What are the top 10 source websites of the publications selected by\n    the newsletter?\n\n```{r}\nuncnewsletter |\u003e \n  group_by(url_source) |\u003e\n  summarise(count=n()) |\u003e\n  arrange(desc(count)) |\u003e\n  head(10) |\u003e\n  ggplot() +\n    geom_col(aes(x = reorder(url_source, count), \n                 y = count)) +\n   labs(title = \"Top 10 publication websites\",\n        subtitle = \"in the selection of North Carolina Water News\",\n        x = \"Website URL\", y = \"Count\") +\n   scale_x_discrete(labels = scales::label_wrap(15))+\n   coord_flip() +\n   theme_classic()\n```\n\n## Method\n\nWe describe the raw data collection procedure of each dataset in this\nsection. To reproduce the collection, you need to have python3 installed\nand install python libraries\n\n```         \npip install requirements.txt\n```\n\n### washdev\n\nThe collection of `washdev` is via web scraping using Python. The script\ncan be found in `inst/python/washdev_scraping.py`. First, each\npublication link is scraped from iterating the table of contents of all\nvolumes. This step delivers a table containing the variables paper ID,\nvolume number, issue number, publication url, journal title, publication\ntitle, and published year. This table will be merged to get the final\ndataset.\n\nThen, for each publication, we retrieve the needed variables from the\npublication's html file using the publication url. The retrieval is\nrule-based to find the relevant fields (e.g. supplementary materials)\nand extract the value.\n\n### uncnewsletter\n\nThe collection of `uncnewsletter` is a combination of web scraping and\nmanual annotation. We first use the newsletter archive to scrape all\npublication website links. The code can be found at\n`inst/python/uncnewsletter_scraping.py`. Two annotators worked on the\nmanual extraction of the needed variables on these publications. For\neach publication, an annotator follows the guide to fill in the value on\nan collaborative spreadsheet. The guide is converted into the data\ndictionary for this dataset.\n\n## License\n\nData are available as\n[CC-BY](https://github.com/openwashdata/washopenresearch/blob/main/LICENSE.md).\n\n## Citation\n\nPlease cite this package using:\n\n```{r}\ncitation(\"washopenresearch\")\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fopenwashdata%2Fwashopenresearch","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fopenwashdata%2Fwashopenresearch","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fopenwashdata%2Fwashopenresearch/lists"}