{"id":13400981,"url":"https://github.com/hrbrmstr/reapr","last_synced_at":"2025-10-29T01:30:45.810Z","repository":{"id":141238723,"uuid":"165945643","full_name":"hrbrmstr/reapr","owner":"hrbrmstr","description":"🕸→ℹ️ Reap Information from Websites","archived":false,"fork":false,"pushed_at":"2019-02-01T12:57:50.000Z","size":93,"stargazers_count":13,"open_issues_count":1,"forks_count":1,"subscribers_count":5,"default_branch":"master","last_synced_at":"2024-07-31T19:28:03.208Z","etag":null,"topics":["html","r","r-cyber","rstats","rvest","web-scraping","xpath"],"latest_commit_sha":null,"homepage":"","language":"R","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"other","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/hrbrmstr.png","metadata":{"files":{"readme":"README.Rmd","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null}},"created_at":"2019-01-16T00:34:05.000Z","updated_at":"2024-02-06T13:53:42.000Z","dependencies_parsed_at":null,"dependency_job_id":"4cab35cc-d678-4e82-a1b6-c6175f2de355","html_url":"https://github.com/hrbrmstr/reapr","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/hrbrmstr%2Freapr","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/hrbrmstr%2Freapr/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/hrbrmstr%2Freapr/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/hrbrmstr%2Freapr/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/hrbrmstr","download_url":"https://codeload.github.com/hrbrmstr/reapr/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":219858513,"owners_count":16556043,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["html","r","r-cyber","rstats","rvest","web-scraping","xpath"],"created_at":"2024-07-30T19:00:57.594Z","updated_at":"2025-10-29T01:30:45.451Z","avatar_url":"https://github.com/hrbrmstr.png","language":"R","funding_links":[],"categories":["R"],"sub_categories":[],"readme":"---\noutput: rmarkdown::github_document\neditor_options: \n  chunk_output_type: console\n---\n```{r pkg-knitr-opts, include=FALSE}\nknitr::opts_chunk$set(collapse=TRUE, fig.retina=2, message=FALSE, warning=FALSE)\noptions(width=120)\n```\n\n[![Travis-CI Build Status](https://travis-ci.org/hrbrmstr/reapr.svg?branch=master)](https://travis-ci.org/hrbrmstr/reapr) \n[![Coverage Status](https://codecov.io/gh/hrbrmstr/reapr/branch/master/graph/badge.svg)](https://codecov.io/gh/hrbrmstr/reapr)\n[![CRAN_Status_Badge](http://www.r-pkg.org/badges/version/reapr)](https://cran.r-project.org/package=reapr)\n\n# reapr\n\nReap Information from Websites\n\n## Description\n\nThere's no longer need to fear getting at the gnarly bits of web pages.\nFor the vast majority of web scraping tasks, the 'rvest' package does a\nphenomenal job providing just enough of what you need to get by. But, if you\nwant more of the details of the site you're scraping, some handy shortcuts to\npage elements in use and the ability to not have to think too hard about\nserialization during scraping tasks, then you may be interested in reaping\nmore than harvesting. Tools are provided to interact with web sites content\nand metadata more granular level than 'rvest' but at a higher level than\n'httr'/'curl'.\n\n## NOTE\n\nThis is very much a WIP but there are enough basic features to let others kick the tyres\nand see what's woefully busted or in need of attention.\n\n## What's Inside The Tin\n\nThe following functions are implemented:\n\n- `reap_url`:\tRead HTML content from a URL\n- `mill`:\tTurn a 'reapr_doc' into plain text without cruft\n- `reapr`:\tReap Information from Websites\n- `reap_attr`:\tReap text, names and attributes from HTML\n- `reap_attrs`:\tReap text, names and attributes from HTML\n- `reap_children`:\tReap text, names and attributes from HTML\n- `reap_name`:\tReap text, names and attributes from HTML\n- `reap_node`:\tReap nodes from an reaped HTML document\n- `reap_nodes`:\tReap nodes from an reaped HTML document\n- `reap_table`:\tExtract data from HTML tables\n- `reap_text`:\tReap text, names and attributes from HTML\n- `add_response_url_from`:\tAdd a 'reapr_doc' response prefix URL to a data frame\n\n## Installation\n\n```{r install-ex, eval=FALSE}\ndevtools::install_git(\"https://git.sr.ht/~hrbrmstr/reapr\")\n# or \ndevtools::install_git(\"https://gitlab.com/hrbrmstr/reapr.git\")\n# or\ndevtools::install_github(\"hrbrmstr/reapr\")\n```\n\n## Usage\n\n```{r lib-ex}\nlibrary(reapr)\nlibrary(hrbrthemes) # sr.hr/~hrbrmstr/hrbrthemes | git[la|hu]b.com/hrbrmstr/hrbrthemes\nlibrary(tidyverse) # for some examples only\n\n# current version\npackageVersion(\"reapr\")\n\n```\n\n## Basic Reaping\n\n```{r basic-reap}\nx \u003c- reap_url(\"http://rud.is/b\")\n\nx\n```\n\nThe formatted object print-output shows much of what you get with a reaped URL.\n\n`reapr::real_url()`:\n\n- Uses `httr::GET()` to make web connections and retrieve content. This enables\n  it to behave more like an actual (non-javascript-enabled) browser. You can\n  pass anything `httr::GET()` can handle to `...` (e.g. `httr::user_agent()`)\n  to have as much granular control over the interaction as possible.\n- Returns a richer set of data. After the `httr::response` object is obtained\n  many tasks are performed including:\n    - timestamping the URL crawl\n    - extraction of the asked-for URL and the final URL (in the case of redirects)\n    - extraction of the IP address of the target server\n    - extraction of both plaintext and parsed (`xml_document`) HTML\n    - extraction of the plaintext webpage `\u003ctitle\u003e` (if any)\n    - generation of a dynamic list tags in the document which can be\n      fed directly to HTML/XML search/retrieval function (which may speed\n      up node discovery)\n    - extraction of the text of all comments in the HTML document\n    - inclusion of the full `httr::response` object with the returned object\n    - extraction of the time it took to make the complete request\n\nFinally, it works with other package member functions to check the validity\nof the parsed `xml_document` and auto-regen the parse (since it has the full\ncontent available to it) prior to any other operations. This also makes `reapr_doc`\nobject _serializable_ without having to spend your own cycles on that.\n\nIf you need more or need the above in different ways please file issues.\n\n## Pre-computed Tags\n\nOn document retrieval, `reapr` automagically builds convenient R-accessible lists of\nall the tags in the retrieved document. They aren't recursive, but they are a convenient\n\"bags\" of tags to use when you don't feel like crafting that perfect XPath.\n\nLet's see what tags RStudio favors most on their Shiny home page:\n\n```{r}\nx \u003c- reap_url(\"https://shiny.rstudio.com/articles/\")\n\nx\n\nenframe(sort(lengths(x$tag))) %\u003e%\n  mutate(name = factor(name, levels = name)) %\u003e%\n  ggplot(aes(value, name)) +\n  geom_segment(aes(xend = 0, yend = name), , size = 3, color = \"goldenrod\") +\n  labs(\n    x = \"Tag frequency\", y = NULL,\n    title = \"HTML Tag Distribution on RStudio's Shiny Homepage\"\n  ) +\n  scale_x_comma(position = \"top\") +\n  theme_ft_rc(grid = \"X\") +\n  theme(axis.text.y = element_text(family = \"mono\"))\n```\n\nLots and lots of `\u003cdiv\u003e`s!\n\n```{r}\nx$tag$div\n```\n\nLet's take a look at the article titles:\n\n```{r results = 'asis'}\nas.data.frame(x$tag$div) %\u003e% \n  filter(class == \"article-title\") %\u003e% \n  select(`Shiny Articles`=elem_content) %\u003e% \n  knitr::kable()\n```\n\nNo XPath or CSS selectors!\n\nLet's abandon the `tidyverse` for base R piping for a minute and do something similar to extract and convert the index of [CRAN Task Views](https://cloud.r-project.org/web/views/) to a markdown list (which will conveniently render here). Again, no XPath or CSS selectors required once we read in the URL:\n\n```{r results='asis'}\nx \u003c- reap_url(\"https://cloud.r-project.org/web/views/\")\n\nas.data.frame(x$tag$a) %\u003e% \n  add_response_url_from(x) %\u003e% \n  subset(!grepl(\"^http[s]://\", href)) %\u003e% \n  transform(href = sprintf(\"- [%s](%s%s)\", elem_content, prefix_url, href)) %\u003e% \n  .[, \"href\", drop=TRUE] %\u003e% \n  paste0(collapse = \"\\n\") %\u003e% \n  cat()\n```\n\nThis functionality is not a panacea since they are just bags of tags, but it may save you some time and frustration.\n\n## Tables\n\nUnlike `rvest` with it's magical and wonderful `html_table()` `reapr` provides more raw control\nover the content of `\u003ctable\u003e` elements. Let's look at the \"population change over time\" table from the Wikipedia page on the demography of the UK (\u003chttps://en.wikipedia.org/wiki/Demography_of_the_United_Kingdom\u003e):\n\n```{r}\nx \u003c- reap_url(\"https://en.wikipedia.org/wiki/Demography_of_the_United_Kingdom\")\n\nreap_node(x, \".//table[contains(., 'Intercensal')]\") %\u003e% \n  reap_table()\n```\n\nAs you can see, it doesn't do the cleanup work for you and has no way to even say there's a header. That's because you can do that with `rvest::html_table()`. The equivalent `reapr` function gives you the raw table and handles `colspan` and `rowspan` insanity by adding the missing cells and filling in the gaps. You can use `docxtractr::assign_colnames()` to make a given row the column titles and `docxtractr::mcga()` or `janitor::clean_names()` to name them proper R names then `readr::type_convert()` to finish the task.\n\nWhile that may seem overkill for this example (it is), it wouldn't be if the table were more gnarly (I'm working on an example for that which will replace this one when it's done).\n\nFor truly gnarly tables you can get an overview of the structure (without the data frame conversion):\n\n```{r}\nreap_node(x, \".//table[contains(., 'Intercensal')]\") %\u003e% \n  reap_table(raw = TRUE) -\u003e raw_tbl\n\nraw_tbl\n```\n\nAnd work with the `list` it gives back (which contains all the HTML element attributes as R attributes so you can pull data stored in them if need be).\n\n## reapr Metrics\n\n```{r cloc, echo=FALSE}\ncloc::cloc_pkg_md()\n```\n\n## Code of Conduct\n\nPlease note that this project is released with a [Contributor Code of Conduct](CONDUCT.md). \nBy participating in this project you agree to abide by its terms.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fhrbrmstr%2Freapr","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fhrbrmstr%2Freapr","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fhrbrmstr%2Freapr/lists"}