https://github.com/tidyverse/rvest

Simple web scraping for R
https://github.com/tidyverse/rvest

html r web-scraping

Last synced: 3 days ago
JSON representation

Simple web scraping for R

• Host: GitHub
• URL: https://github.com/tidyverse/rvest
• Owner: tidyverse
• License: other
• Created: 2014-07-23T21:22:27.000Z (over 10 years ago)
• Default Branch: main
• Last Pushed: 2024-10-25T15:53:24.000Z (about 2 months ago)
• Last Synced: 2024-10-29T15:02:35.494Z (about 1 month ago)
• Topics: html, r, web-scraping
• Language: R
• Homepage: https://rvest.tidyverse.org
• Size: 12.3 MB
• Stars: 1,493
• Watchers: 88
• Forks: 342
• Open Issues: 30
• Metadata Files:
  • Readme: README.Rmd
  • Changelog: NEWS.md
  • Contributing: .github/CONTRIBUTING.md
  • License: LICENSE
  • Code of conduct: .github/CODE_OF_CONDUCT.md
  • Codeowners: .github/CODEOWNERS
  • Support: .github/SUPPORT.md

Awesome Lists containing this project

• jimsghstars - tidyverse/rvest - Simple web scraping for R (R)

README

        
---

output: github_document

---

```{r, echo = FALSE}

knitr::opts_chunk$set(

  collapse = TRUE, 

  comment = "#>",

  fig.path = "README-"  

)

```

# rvest 

[![CRAN status](https://www.r-pkg.org/badges/version/rvest)](https://cran.r-project.org/package=rvest)

[![R-CMD-check](https://github.com/tidyverse/rvest/actions/workflows/R-CMD-check.yaml/badge.svg)](https://github.com/tidyverse/rvest/actions/workflows/R-CMD-check.yaml)

[![Codecov test coverage](https://codecov.io/gh/tidyverse/rvest/branch/main/graph/badge.svg)](https://app.codecov.io/gh/tidyverse/rvest?branch=main)

## Overview

rvest helps you scrape (or harvest) data from web pages.

It is designed to work with [magrittr](https://github.com/tidyverse/magrittr) to make it easy to express common web scraping tasks, inspired by libraries like [beautiful soup](https://www.crummy.com/software/BeautifulSoup/) and [RoboBrowser](http://robobrowser.readthedocs.io/en/latest/readme.html).

If you're scraping multiple pages, I highly recommend using rvest in concert with [polite](https://dmi3kno.github.io/polite/).

The polite package ensures that you're respecting the [robots.txt](https://en.wikipedia.org/wiki/Robots_exclusion_standard) and not hammering the site with too many requests.

## Installation

```{r, eval = FALSE}

# The easiest way to get rvest is to install the whole tidyverse:

install.packages("tidyverse")

# Alternatively, install just rvest:

install.packages("rvest")

```

## Usage

```{r, message = FALSE}

library(rvest)

# Start by reading a HTML page with read_html():

starwars <- read_html("https://rvest.tidyverse.org/articles/starwars.html")

# Then find elements that match a css selector or XPath expression

# using html_elements(). In this example, each  corresponds

# to a different film

films <- starwars %>% html_elements("section")

films

# Then use html_element() to extract one element per film. Here

# we the title is given by the text inside 


title <- films %>% 

  html_element("h2") %>% 

  html_text2()

title


# Or use html_attr() to get data out of attributes. html_attr() always

# returns a string so we convert it to an integer using a readr function

episode <- films %>% 

  html_element("h2") %>% 

  html_attr("data-id") %>% 

  readr::parse_integer()

episode

```

If the page contains tabular data you can convert it directly to a data frame with `html_table()`:

```{r}

html <- read_html("https://en.wikipedia.org/w/index.php?title=The_Lego_Movie&oldid=998422565")

html %>% 

  html_element(".tracklist") %>% 

  html_table()

```