{"id":13642795,"url":"https://github.com/yusuzech/r-web-scraping-cheat-sheet","last_synced_at":"2026-04-29T00:05:34.594Z","repository":{"id":48134550,"uuid":"133077322","full_name":"yusuzech/r-web-scraping-cheat-sheet","owner":"yusuzech","description":"Guide, reference and cheatsheet on web scraping using rvest, httr and Rselenium.","archived":false,"fork":false,"pushed_at":"2022-12-20T03:16:29.000Z","size":3033,"stargazers_count":385,"open_issues_count":0,"forks_count":104,"subscribers_count":22,"default_branch":"master","last_synced_at":"2024-08-02T01:17:18.413Z","etag":null,"topics":["cheatsheet","httr","r","rselenium","rvest","scrape-websites","web-scraping","webscraping"],"latest_commit_sha":null,"homepage":"","language":"R","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/yusuzech.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2018-05-11T18:52:53.000Z","updated_at":"2024-07-23T14:43:51.000Z","dependencies_parsed_at":"2023-01-29T23:31:17.348Z","dependency_job_id":null,"html_url":"https://github.com/yusuzech/r-web-scraping-cheat-sheet","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/yusuzech%2Fr-web-scraping-cheat-sheet","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/yusuzech%2Fr-web-scraping-cheat-sheet/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/yusuzech%2Fr-web-scraping-cheat-sheet/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/yusuzech%2Fr-web-scraping-cheat-sheet/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/yusuzech","download_url":"https://codeload.github.com/yusuzech/r-web-scraping-cheat-sheet/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":223839022,"owners_count":17211859,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["cheatsheet","httr","r","rselenium","rvest","scrape-websites","web-scraping","webscraping"],"created_at":"2024-08-02T01:01:36.357Z","updated_at":"2026-04-29T00:05:33.905Z","avatar_url":"https://github.com/yusuzech.png","language":"R","funding_links":[],"categories":["R","Topic Areas"],"sub_categories":["Web Scraping"],"readme":"# Web Scraping Reference: Cheat Sheet for Web Scraping using R\n\nInspired by Hartley Brody, this cheat sheet is about web scraping using [rvest](https://github.com/hadley/rvest),[httr](https://github.com/r-lib/httr) and [Rselenium](https://github.com/ropensci/RSelenium). It covers many topics in this [blog](https://blog.hartleybrody.com/web-scraping-cheat-sheet/).\n\nWhile Hartley uses python's requests and beautifulsoup libraries, this cheat sheet covers the usage of httr and rvest. While rvest is good enough for many scraping tasks, httr is required for more advanced techniques. Usage of Rselenium(web driver) is also covered.\n\nI also recommend the book [The Ultimate Guide to Web Scraping](https://blog.hartleybrody.com/guide-to-web-scraping/) by Hartley Brody. Though it uses Python libraries, the underlying logic of web scraping is the same. The same strategies can be applied using any languages including R.\n\nPlease post issues [here](https://github.com/yusuzech/r-web-scraping-cheat-sheet/issues) if you find any errors or have any recommendations.\n\n# Table of Contents\n1. \u003ca href=\"#rvest\"\u003eWeb Scraping using rvest and httr\u003c/a\u003e\n    1. \u003ca href=\"#rvest1\"\u003eUseful Libraries and Resources\u003c/a\u003e\n    2. \u003ca href=\"#rvest2\"\u003eMaking Simple Requests\u003c/a\u003e\n    3. \u003ca href=\"#rvest3\"\u003eInspecting Response\u003c/a\u003e\n    4. \u003ca href=\"#rvest4\"\u003eExtracting Elements from HTML\u003c/a\u003e\n    5. \u003ca href=\"#rvest5\"\u003eStoring Data in R\u003c/a\u003e\n        1. \u003ca href=\"#rvest5.1\"\u003eStoring Data as list\u003c/a\u003e\n        2. \u003ca href=\"#rvest5.2\"\u003eStoring Data as data.frame\u003c/a\u003e\n    6. \u003ca href=\"#rvest6\"\u003eSaving Data to disk\u003c/a\u003e\n        1. \u003ca href=\"#rvest6.1\"\u003eSaving Data to csv\u003c/a\u003e\n        2. \u003ca href=\"#rvest6.2\"\u003eSaving Data to SQLite Database\u003c/a\u003e\n    7. \u003ca href=\"#rvest7\"\u003eMore Advanced Topics\u003c/a\u003e\n        1. \u003ca href=\"#rvest7.1\"\u003eJavascript Heavy Websites\u003c/a\u003e\n        2. \u003ca href=\"#rvest7.2\"\u003eContent Inside iFrames\u003c/a\u003e\n        3. \u003ca href=\"#rvest7.3\"\u003eSessions and Cookies\u003c/a\u003e\n        4. \u003ca href=\"#rvest7.4\"\u003eDelays and Backing Off\u003c/a\u003e\n        5. \u003ca href=\"#rvest7.5\"\u003eSpoofing the User Agent\u003c/a\u003e\n        6. \u003ca href=\"#rvest7.6\"\u003eUsing Proxy Servers\u003c/a\u003e\n        7. \u003ca href=\"#rvest7.7\"\u003eSetting Timeouts\u003c/a\u003e\n        8. \u003ca href=\"#rvest7.8\"\u003eHandling Network Errors\u003c/a\u003e\n        9. \u003ca href=\"#rvest7.9\"\u003eDownloading Files\u003c/a\u003e\n        10. \u003ca href=\"#rvest7.10\"\u003eLogins and Sessions\u003c/a\u003e\n        11. \u003ca href=\"#rvest7.11\"\u003eWeb Scraping in Parallel\u003c/a\u003e\n2. \u003ca href=\"#rselenium\"\u003eWeb Scraping using Rselenium\u003c/a\u003e\n    1. \u003ca href=\"#rselenium1\"\u003eWhy RSelenium\u003c/a\u003e\n       1. \u003ca href=\"#rselenium1.1\"\u003ePros and Cons from Using RSelenium\u003c/a\u003e\n       2. \u003ca href=\"#rselenium1.2\"\u003eUseful Resources\u003c/a\u003e\n    2. \u003ca href=\"#rselenium2\"\u003eInteracting with the Web Driver In Rselenium\u003c/a\u003e\n       1. \u003ca href=\"#rselenium2.0\"\u003eHow to Start\u003c/a\u003e\n       2. \u003ca href=\"#rselenium2.1\"\u003eNavigating to different URLs\u003c/a\u003e\n       3. \u003ca href=\"#rselenium2.2\"\u003eSimulating Scrolls, Clicks, Text Inputs, Logins, and Other Actions\u003c/a\u003e\n    3. \u003ca href=\"#rselenium3\"\u003eExtract Content from the Web Page\u003c/a\u003e\n       1. \u003ca href=\"#rselenium3.1\"\u003eExtracting Content using Rselenium\u003c/a\u003e\n       2. \u003ca href=\"#rselenium3.2\"\u003eExtracting Content using Parsed Page Source and `rvest`\u003c/a\u003e\n    4. \u003ca href = \"#rselneium4\"\u003emiscellanea\u003c/a\u003e\n       1. \u003ca href=\"#rselenium4.1\"\u003eJavascript\u003c/a\u003e\n       2. \u003ca href=\"#rselenium4.2\"\u003eIframe\u003c/a\u003e\n3. \u003ca href=\"#change\"\u003eChange Log\u003c/a\u003e\n\n# 1. \u003ca name=\"rvest\"\u003eWeb Scraping using rvest and httr\u003c/a\u003e\n## 1.1. \u003ca name=\"rvest1\"\u003eUseful Libraries and Resources\u003c/a\u003e\n\n[rvest](https://github.com/hadley/rvest) is built upon the xml2 package and also accept config from the `httr` package. For the most part, we only need `rvest`. However, we need `httr` if we want to add extra configurations.\n\nTo install those two packages:\n\n```r\ninstall.packages(\"rvest\")\ninstall.packages(\"httr\")\n```\n\nTo load them:\n\n```r\nrequire(rvest)\nrequire(httr)\n```\n\nThere are many resources available online; these are what I found to be the most useful:\n\n1. [w3schools CSS selectors reference](https://www.w3schools.com/CSSref/css_selectors.asp) : if you forget CSS syntax, just check it here\n1. [w3schools XPATH reference](\u003chttps://www.w3schools.com/xml/xpath_intro.asp\u003e): XPATH is an alternative in selecting elements on websites. It's harder to learn but it's more flexible and robust.\n1. [CSS Diner](https://flukeout.github.io/) : the easiest way to learn and understand CSS by playing games.\n1. [Chrome CSS selector plugin](https://selectorgadget.com/): a convenient tool to use for choosing CSS selector.\n1. [ChroPath](\u003chttps://chrome.google.com/webstore/detail/chropath/ljngjbnaijcbncmcnjfhigebomdlkcjo\u003e): a very convenient tool for choosing XPATH.\n1. [Stack Overflow](https://stackoverflow.com/) : You can find answers to most of your problems, no matter it's web scraping, rvest or CSS.\n1. [Web Scraping Sandbox](http://toscrape.com/): Great place to test your web scraping skills.\n\n**Functions and classes in rvest/httr:**\n\nSometimes you may get confused about all the functions and classes you have. You can review this image at the moment.\n![](resources/functions_and_classes.png)\n\n\\*\\* Please notice that: somtimes response could be JSON or other formats instead of HTML. In those cases you need other functions to parse the content(e.g. jsonlite::fromJSON() to parse a JSON string to a list).\n\n\n## 1.2. \u003ca name=\"rvest2\"\u003eMaking Simple Requests\u003c/a\u003e\n\nrvest provides two ways of making request: `read_html()` and `html_session()`  \n`read_html()` can parse a HTML file or an url into xml document. `html_session()` is built on `GET()` from httr package and can accept configurations defined by httr package.  \n\nReading a url:\n\n```R\n#making GET request andparse website into xml document\npagesource \u003c- read_html(\"http://example.com/page\")\n\n#using html_session which creates a session and accept httr methods\nmy_session \u003c- html_session(\"http://example.com/page\")\n#html_session is built upon httr, you can also get response with a session\nresponse \u003c- my_session$response\n```\n\nAlternatively, GET and POST method are available in the httr package.\n\n```R\nlibrary(httr)\nresponse \u003c- GET(\"http://example.com/page\")\n#or\nresponse \u003c- POST(\"http://example.com/page\",\n    body = list(a=1,b=2))\n```\n\n## 1.3. \u003ca name=\"rvest3\"\u003eInspecting Response\u003c/a\u003e\n\nCheck status code:\n\n```R\nstatus_code(my_session)\nstatus_code(response)\n```\n\nGet response and content:\n\n```R\n#response\nresponse \u003c- my_session$response\n#retrieve content as raw\ncontent_raw \u003c- content(my_session$response,as = \"raw\")\n#retrieve content as text\ncontent_text \u003c- content(my_session$response,as = \"text\")\n#retrieve content as parsed(parsed automatically)\ncontent_parsed \u003c- content(my_session$response,as = \"parsed\")\n```\n\n\\*\\*note:\n\nContent may be parsed incorrectly sometimes. For those situations, you can parse the content to text or raw and use other libraries or functions to parse it correctly.\n\nSearch for specific string:\n\n```R\nlibrary(stringr)\n#regular expression can also be used here\nif(str_detect(content_text,\"blocked\")){\n    print(\"blocked from website\")\n    }\n```\n\ncheck content type:\n\n```R\nresponse$headers$`content-type`\n```\n\ncheck html structure:\n\n```R\nmy_structure \u003c- html_structure(content_parsed)\n```\n\n## 1.4. \u003ca name=\"rvest4\"\u003eExtracting Elements from HTML\u003c/a\u003e\n\nUsing the regular expression to scrape HTML is not a very good idea, but it does have its usage like scraping all emails from websites, there is a detailed discussion about this topic on [stackoverflow](https://stackoverflow.com/questions/1732348/regex-match-open-tags-except-xhtml-self-contained-tags).  \n\n**Using rvest:**\n\nI will scrape https://scrapethissite.com/ for demonstration, since it has static HTML:  \n\nFor the purpose of extracting elements, using `read_html()` or `html_session()` are both fine. When using `read_html()`, it returns a xml_document. When using `html_session()`, it creates a session and the response is included.\n\n```R\nmy_session \u003c- html_session(\"https://scrapethissite.com/pages/simple/\")\n```\n\nLook for nodes:\n\n```R\nmy_nodes \u003c- my_session %\u003e% html_elements(\".country\")\n```\n\nLook for attributes:\n\n```R\nmy_attributes \u003c- my_session %\u003e% html_elements(\".country-capital\") %\u003e% html_attr(\"class\")\n```\n\nLook for texts:\n\n```R\nmy_texts \u003c- my_session %\u003e% html_elements(\".country-capital\") %\u003e% html_text()\n```\n\n## 1.5. \u003ca name=\"rvest5\"\u003eStoring Data in R\u003c/a\u003e\n\nrvest can return a vector of elements or even table of elements, so it's easy to store it in R.\n\n### 1.5.1. \u003ca name=\"rvest5.1\"\u003eStoring Data as list\u003c/a\u003e\n\nUsually, rvest can return a vector, so it's very easy to store it.\n\n```R\nmy_texts \u003c- my_session %\u003e% html_elements(\".country-capital\") %\u003e% html_text()\n```\n\n### 1.5.2. \u003ca name=\"rvest5.2\"\u003eStoring Data as data.frame\u003c/a\u003e\n\nWe can concatenate vectors in a table or using `html_table()` to extract a HTML table directly into a data.frame.\n\n```R\nmy_country \u003c- my_session %\u003e% html_elements(\".country-name\") %\u003e% html_text()\nmy_capitals \u003c- my_session %\u003e% html_elements(\".country-capital\") %\u003e% html_text()\nmy_table \u003c- data.frame(country = my_country, capital = my_capitals)\n```\n\n## 1.6. \u003ca name=\"rvest6\"\u003eSaving Data to disk\u003c/a\u003e\n### 1.6.1. \u003ca name=\"rvest6.1\"\u003eSaving Data to csv\u003c/a\u003e\n\nIf the data is already stored as a data.frame:\n\n```R\nwrite.csv(my_table,file=\"my_table.csv\")\n```\n\n### 1.6.2. \u003ca name=\"rvest6.2\"\u003eSaving Data to SQLite Database\u003c/a\u003e\n\nAfter creating the database \"webscrape.db\":\n\n```R\nlibrary(RSQLite)\nconnection \u003c- dbConnect(SQLite(),\"webscrape.db\")\ndbWriteTable(conn = connection,name = \"country_capital\",value = my_table)\ndbDisconnect(conn = connection)\n```\n\n## 1.7. \u003ca name=\"rvest7\"\u003eMore Advanced Topics\u003c/a\u003e\n\n\u003c!-- \nrvest and httr package provides some handy functions, but they lack the function of:\n\n+ prevent errors from breaking loops\n+ auto retry requests\n+ keep the record of failed requests\n\nSo I wrote a wrapper function on html_session which can deal with the issues above and make web scraping scripts much more robust. The function and usage can be found in this [repository](https://github.com/yusuzech/r-web-scraping-template). \n--\u003e\n\n### 1.7.1. \u003ca name=\"rvest7.1\"\u003eJavascript Heavy Websites\u003c/a\u003e\n\nFor javascript heavy websites, there are three possible solutions:\n\n1. Execute javascript in R\n2. Use Developer tools(e.g. [Network in Chrome](https://developers.google.com/web/tools/chrome-devtools/network-performance/))\n3. Using Rselenium or other web drivers\n\nThere are pros and cons of each method:  \n1. Executing Javascript in R is the most difficult one since it requires some knowledge of Javascript, but it makes web-scraping javascript heavy websites possible with rvest.  \n2. Using Developer tools is not difficult. The pro is that you only need to learn some examples and you can then work on it by yourself. The con is that if the website structure gets more completed, it requires more knowledge of HTTP.\n3. The Rselenium is absolutely the easiest solution. The pro is it's easy to learn and use. The con is that it can be unstable sometimes and related resources are very limited online. In many situations, you may need to refer to python codes with selenium package.\n\n#### 1.Execute javascript\n\nI learned how to use Javascript with this [post](https://datascienceplus.com/scraping-javascript-rendered-web-content-using-r/). Since I'm not an expert in Javascript, I recommend you to search for other related resources online.\n\nIf you don't know Javascipt, then this method is unlikely to be suitable for you. Since Javascript isn't an easy subject, I don't recommend you to learn it if your only purpose is to use it to do web-scraping in R. The following two methods are much easier.\n\n#### 2.Use Developer tools\n\nI learned this trick from Hartley's blog; the following section is quoted from his [post](https://blog.hartleybrody.com/web-scraping-cheat-sheet/):\n\n\u003eContrary to popular belief, you do not need any special tools to scrape websites that load their content via Javascript. For the information to get from their server and show up on a page in your browser, that information had to have been returned in an HTTP response somewhere.\n\u003e\n\u003eIt usually means that you won’t be making an HTTP request to the page’s URL that you see at the top of your browser window, but instead you’ll need to find the URL of the AJAX request that’s going on in the background to fetch the data from the server and load it into the page.\n\u003e\n\u003eThere’s not really an easy code snippet I can show here, but if you open the Chrome or Firefox Developer Tools, you can load the page, go to the “Network” tab and then look through the all of the requests that are being sent in the background to find the one that’s returning the data you’re looking for. Start by filtering the requests to only XHR or JS to make this easier.\n\u003e\n\u003eOnce you find the AJAX request that returns the data you’re hoping to scrape, then you can make your scraper send requests to this URL, instead of to the parent page’s URL. If you’re lucky, the response will be encoded with JSON which is even easier to parse than HTML.\n\nSo, as Hartley said, basically, everything displayed on your browser must be sent to you through JSON, HTML or other formats. What you need to do is to capture this file. \n\nI answered a couple of questions on Stack Overflow about scraping JavaScript rendered content. You may have the same problems as in the posts. Please check out the answers, and hope you get the ideas of how to use developer tools.\n\nhttps://stackoverflow.com/questions/50596714/how-to-scrap-a-jsp-page-in-r/50598032#50598032  \nhttps://stackoverflow.com/questions/50765111/hover-pop-up-text-cannot-be-selected-for-rvest-in-r/50769875#50769875  \nhttps://stackoverflow.com/questions/50900987/scraping-dl-dt-dd-html-data/50922733#50922733  \nhttps://stackoverflow.com/questions/50693362/unable-to-extract-thorough-data-using-rvest/50730204#50730204  \nhttps://stackoverflow.com/questions/50997094/trouble-reaching-a-css-node/50997121#50997121  \nhttps://stackoverflow.com/questions/50262108/scraping-javascript-rendered-content-using-r/50263349#50263349  \n\n#### 3.Using Rselenium or other web driver\n\nRselenium launches a Chrome/Firefox/IE browser where you can simulate human actions like clicking on links, scrolling up or down.  \n\nIt is a very convenient tool, and it renders JavaScript and Interactive content automatically, so you don't need to worry about the complex HTTP and AJAX stuff. However, there are also some limitations to it:  \n\n1. The first limitation is that:  it is very slow. Depending on the complexity of the websites, it could take seconds to render a single page while using httr/rvest takes less than one second. It is fine if you only want to scrape several hundred pages. However, if you want scrape thousands or ten thousands of pages, then the speed will become an issue.\n2. The second limitation is that: There are little online resources on Rselenium. In many situations, you can't find related posts on Stack Overflow that solve your problem. You may need to refer to Python/Java Selenium posts for answers, and sometimes answers can't be applied in R.\n\nMore detailed usage is explained in **Web Scraping using Rselenium**.\n\n### 1.7.2. \u003ca name=\"rvest7.2\"\u003eContent Inside iFrames\u003c/a\u003e\n\nIframes are other websites embedded in the websites you are viewing as explained on [Wikipedia](https://en.wikipedia.org/wiki/HTML_element#Frames):\n\n\u003e Frames allow a visual HTML Browser window to be split into segments, each of which can show a different document. This can lower bandwidth use, as repeating parts of a layout can be used in one frame, while variable content is displayed in another. This may come at a certain usability cost, especially in non-visual user agents,[[51\\]](https://en.wikipedia.org/wiki/HTML_element#cite_note-58) due to separate and independent documents (or websites) being displayed adjacent to each other and being allowed to interact with the same parent window. Because of this cost, frames (excluding the `\u003ciframe\u003e` element) are only allowed in HTML 4.01 Frame-set. Iframes can also hold documents on different servers. In this case, the interaction between windows is blocked by the browser. \n\n\n\nTherefore, to extract content in an iframe, you need to find the link to the iframe HTML.\n\n```R\n#example script\nlink_to_iframe \u003c- my_session(\"www.example.com\") %\u003e%\n    html_element(\"css to locate the iframe\") %\u003e%\n    html_attr(\"src\")\n#make another request to the iframe and use this session to extract information\niframe_session \u003c- html_session(link_to_iframe)\n```\n\n\n\nHere is a tutorial about iframes using [scrapethissite](https://blog.hartleybrody.com/web-scraping-cheat-sheet/#content-inside-iframes):\n\nIn this tutorial, we will get information embedded in an iframe: [**Tutorial Link**](example_script/iframe_tutorial.md)  \n\n\n\n### 1.7.3. \u003ca name=\"rvest7.3\"\u003eSessions and Cookies\u003c/a\u003e\n\n`rvest::html_session()` creates a session automatically, you can use `jump_to()` and `follow_link` to navigate to other web pages using the same session.\n\n\n\n```R\nlibrary(rvest)\nurl1 \u003c- \"https://scrapethissite.com/\"\nurl2 \u003c- \"https://scrapethissite.com/pages/simple/\"\nmy_session \u003c- html_session(url1)\nmy_session \u003c- my_session %\u003e% jump_to(url2) # navigate to another url\n```\n\n\n\nyou can check session history:\n\n\n\n```R\n\u003e session_history(my_session)\n  https://scrapethissite.com/\n- https://scrapethissite.com/pages/simple/\n```\n\n\n\n you can access the cookies:\n\n```R\nlibrary(httr)\ncookies(my_session) #cookies are saved in a table\n```\n\n\n\n### 1.7.4. \u003ca name=\"rvest7.4\"\u003eDelays and Backing Off\u003c/a\u003e\n\nYou can slow down your requests by pausing between requests:\n\n\n\n```R\nlibrary(httr)\nfor(my_url in my_urls){\n    response \u003c- httr::GET(my_url)\n    #do something\n    Sys.sleep(5) # sleep 5 seconds\n}\n```\n\n\n\nYou can also decide how you wait by measuring how long the site took to respond. So why should you do it, as Hartley says:\n\n\u003e Some also recommend adding a backoff that’s proportional to how long the site took to respond to your request. That way if the site gets overwhelmed and starts to slow down, your code will automatically back off. \n\n\n\n```R\nlibrary(httr)\nfor(my_url in my_urls){\n    t0 \u003c- Sys.time()\n    response \u003c- httr::GET(my_url)\n    t1 \u003c- Sys.time()\n    #do something\n    response_delay \u003c- as.numeric(t1-t0)\n    Sys.sleep(10*response_delay) # sleep 10 times longer than response_delay\n}\n\n```\n\n\n\n### 1.7.5. \u003ca name=\"rvest7.5\"\u003eSpoofing the User Agent\u003c/a\u003e\n\n**First of all, what is a user agent?**\n\n\u003e In computing, a user agent is software (a software agent) that is acting on behalf of a user. One common use of the term refers to a web browser telling a website information about the browser and operating system. This allows the website to customize content for the capabilities of a particular device, but also raises privacy issues.\n\nIn short, a user agent is a string that identifies you, and you can search \"most popular browser user agents\" on google to get a rough idea.\n\n**So why should I use spoof user agent?**  \n\n* You want to make your scraper look like a real user instead of a script. Some websites even don't allow an uncommon user agent to access.\n\n```R\nlibrary(rvest)\nmy_session \u003c- html_session(\"https://scrapethissite.com/\")\n# if you don't use custom user agent, your user agent will be something like:\n# RUN: my_session$response$request$options$useragent\n\"libcurl/7.59.0 r-curl/3.2 httr/1.3.1\"\n```\n\n**So how can I spoof it?**\n\n```R\nlibrary(rvest)\nlibrary(httr)\n#1.spoof it with common user agent\nua \u003c- user_agent(\"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/58.0.3029.110 Safari/537.36\")\nseesion_with_ua \u003c- html_session(\"https://scrapethissite.com/\",ua)\n#2.fill your contact information in user agent if you want the website owner to contact you\nua_string \u003c- \"Contact me at xyz123@gmail.com\"\nseesion_with_ua \u003c- html_session(\"https://scrapethissite.com/\",user_agent(ua_string))\n```\n\n### 1.7.6. \u003ca name=\"rvest7.6\"\u003eUsing Proxy Servers\u003c/a\u003e\n\nSome servers automatically ban an IP if it sees abnormal traffic from that IP. The way to avoid it is to use proxies, so you can spread your requests among different IPs and reduce the chance of the server banning you. \n\nThough there are many free proxies available, paid ones are usually more reliable. One provider I use is \"Proxy Bonanza.\"\n\nOnce you have free or paid proxies, you can use them when you make requests:\n\n```R\nlibrary(rvest)\nlibrary(httr)\nmy_proxy \u003c- use_proxy(url=\"http://example.com\",\n                     user_name = \"myusername\",\n                     password = \"mypassword\",\n                     auth = \"one of basic, digest, digest_ie, gssnegotiate, ntlm, any\")\n#use it in html_session(),POST() or GET()\nmy_session \u003c- html_session(\"https://scrapethissite.com/\",my_proxy)\nmy_response \u003c- GET(\"https://scrapethissite.com/\",my_proxy)\n```\n\n### 1.7.7. \u003ca name=\"rvest7.7\"\u003eSetting Timeouts\u003c/a\u003e\n\nSometimes you may encounter slow connections and want to move to other jobs instead of waiting. You can set timeout if you don't receive a response.    \n\n```R\nlibrary(rvest)\nlibrary(httr)\nmy_session \u003c- html_session(\"https://scrapethissite.com/\",timeout(5)) # if you don't receive reponse within 5 seconds, it will throw an error\n\n#you can use try() or tryCatch() to continue if the error occured\nfor(my_url in my_urls){\n    try(GET(my_url,timeout(5)))\n}\n```\n\nFor detailed usage of `try()` and `tryCatch()`, you can check the following posts:\n\nhttps://stackoverflow.com/questions/12193779/how-to-write-trycatch-in-r\n\n### 1.7.8. \u003ca name=\"rvest7.8\"\u003eHandling Network Errors\u003c/a\u003e\n\nYou can use `try()` or `tryCatch()` to handle unpredictable network issues.\n\nYou can retry if an error occurs. For more details, you can check the following posts:\n\nhttps://stackoverflow.com/questions/20770497/how-to-retry-a-statement-on-error\n\n### 1.7.9. \u003ca name=\"rvest7.9\"\u003eDownloading Files\u003c/a\u003e  \n\nMost of the times, the purpose of web scrapping is to extract texts from web pages. But there are also scenarios that you want to save images,web pages or other files.\n\n**Downloading files using `download.file()`**\n\n`download.file()` is a built-in function in R and it's very easy to use. But it doesn't support extra configurations such as proxies or user agents.\n\n```R\n#Download a web page\nmy_url \u003c- \"https://scrapethissite.com/\"\ndownload.file(url = my_url, # required: provide url \n              destfile = \"test.html\", # required: path and name for downloaded file\n              quiet = T, # optional: whether to suppress messages\n              mode = \"w\") #optional: default to \"w\", switch to other modes if file is garbled\n\n#Download a image\nimage_url \u003c- \"https://upload.wikimedia.org/wikipedia/commons/thumb/7/71/GreenSeaTurtle-2.jpg/200px-GreenSeaTurtle-2.jpg\"\ndownload.file(image_url,\"turtle1.jpg\",mode = \"wb\") #mode must be wb(binary) for images\n```\n\n\n\n**Downloading files using `writeBin()`**\n\nAnother option is to use `writeBin()` function to download files. This function can write the binary content from  the response to a file.\n\n```R\nlibrary(httr)\n#use writeBin() to write a html file\nmy_url \u003c- \"https://scrapethissite.com/\"\nresponse \u003c- GET(my_url)\nwriteBin(response$content,\"test1.html\")\n\n#downlaod images using `writeBin()`\nimage_url \u003c- \"https://upload.wikimedia.org/wikipedia/commons/thumb/7/71/GreenSeaTurtle-2.jpg/200px-GreenSeaTurtle-2.jpg\"\nresponse \u003c- GET(image_url)\nwriteBin(response$content,\"turtle2.jpg\")\n```\n\n### 1.7.10. \u003ca name=\"rvest7.10\"\u003eLogins and Sessions\u003c/a\u003e\n\nSometimes you need to log in to access certain content. You can do it using rvest and httr. You can also save your cookies so you don't need to log in every time.\n\n**Log in using rvest:**\n\n```R\nlibrary(rvest)\nurl \u003c- \"https://example.com/login\"\nmy_session \u003c- html_session(url) #Create a persistant session\nunfilled_forms \u003c- html_form(my_session) # find all forms in the web page\nlogin_form \u003c- unfilled_forms[[1]] # select the form you need to fill\nfilled_form \u003c- set_values(login_form,username=\"myusername\",password=\"mypassword\") # fill the form\nlogin_session \u003c- submit_form(my_session,filled_form) # Now you're logged in\n```\n\nThere is also a built-in [demo](https://github.com/hadley/rvest/blob/master/demo/united.R) for logins in rvest.\n\n**Using the session**\n\n`html_session()` simulates a session in an html browser, you can navigate to other pages using the same session with same cookies.\n\nTo stay in the same session, you can use `jump_to()` and `follow_link()`:\n\n```R\nlibrary(rvest)\nurl \u003c- \"www.example.com\"\nmy_session \u003c- html_session(url) # create a session\nmy_session1 \u003c- my_session %\u003e% jump_to(\"www.example.com/page1\") # go to another web page using the same session\nmy_session2 \u003c- my_session %\u003e% follow_link(css = \"p a\")\n```\n\n\n\nYou can also stay in the same session by using cookies: \n\n```R\nlibrary(rvest)\nlibrary(httr)\nurl \u003c- \"http://ww.example.com\"\nmy_session \u003c- html_session(url) # create a session\nmy_cookies_table \u003c- cookies(my_session) # get the cookies as a table\nmy_cookies \u003c- my_cookies_table$value %\u003e% setNames(my_cookies_table$name) # save the cookies as a named vector that you can use in your requests\nnew_session \u003c- html_session(url,set_cookies(my_cookies)) # making requests using the same cookies/session\n```\n\n### 1.7.11. \u003ca name=\"rvest7.11\"\u003eWeb Scraping in Parallel\u003c/a\u003e\n\nTo speed up web scraping process, you may want to do it in parallel. To achieve this goal, you can use [`doParallel`](https://cran.r-project.org/web/packages/doParallel/index.html) package or [`Rcrawler`](https://github.com/salimk/Rcrawler) package. The later package has parallel web scraping as a built in feature, it also has many other great features and the developers are adding even more.   For this part, we are focusing on `doParallel`.\n\n```R\nlibrary(rvest)\nlibrary(doParallel)\n#assume we have a list of urls\n#urls \u003c- c(\"url1\",\"url2,...\")\ncl \u003c- makeCluster(detectCores()-1) #using max cores - 1 for parallel processing\nregisterDoParallel(cl) #register cores\n\n# use foreach() %dopar%{} to make requests and parese texts in parallel\nresult \u003c- foreach(i = seq_along(urls),\n                  .packages = \"rvest\",\n                  .combine = \"c\",\n                  .errorhandling='pass') %dopar% {\n                      text \u003c- html_session(urls[i]) %\u003e% \n                          html_elements(\"css\") %\u003e%\n    \t\t\t\t\t  html_text()\n                      return(text)\n                  }\n```\n\nIn this [example](resources/rvest_parallel.md), it compares the speed between  `foreach` and `for` loop.\n\nAlso, you need to be aware of that: as you are making more concurrent requests, you are much more likely to get banned. So remember to use proxies or reduce the speed of requests. \n\n****\n\n# 2. \u003ca name=\"rselenium\"\u003eWeb Scraping using Rselenium\u003c/a\u003e\n\n## 2.1. \u003ca name=\"rselenium1\"\u003eWhy RSelenium\u003c/a\u003e\n\nAs mentioned in previous sections, in JavaScript heavy websites, the majority of content could be generated using JavaScript. When using a browser, it will make multiple requests for you while the original page is loading. However, if you use `httr` or `rvest`, only the original page will be loaded and JavaScript won't be executed. Hence some data is only available in browser.\n\n\n\n[RSelenium](https://github.com/ropensci/RSelenium) makes use of [Selenium Web Driver](https://www.seleniumhq.org/projects/webdriver/) which simulates a browser(usually Chrome and Firefox) and renders web pages automatically, it will execute all the JavaScript codes for you so you will get a parsed page source instead of unparsed(raw) one.\n\n### 2.1.1 \u003ca name=\"rselenium1.1\"\u003ePros and Cons from Using RSelenium\u003c/a\u003e\n\n**Pros:**\n\n1. Renders JavaScript generated content automatically:\n   * One of the greatest headache from web scraping is that what you see in browser and what you get in response are different. By using RSelenium, you can avoid this issue.\n\n**Cons:**\n\n1. It's very slow:\n   * Since the browser loads everything on the web page(including photos, ads or even videos), it will be really slow comparing to making requests using `httr` or `rvest`.\n2. Lacking support:\n   * Comparing to Selenium in Python, RSelenium doesn't have a large user base and hence lacks support. If you ever searched RSelenium related questions, you might already found out that: many solutions could be obsolete , you couldn't find related topics or the only solutions available were for Python.\n\n### 2.1.2 \u003ca name=\"rselenium1.2\"\u003eUseful Resources\u003c/a\u003e\n\n1. [Rselenium Github Page](https://github.com/ropensci/RSelenium): many useful resources are already listed here.\n2. [http://toscrape.com/](http://toscrape.com/): A sandbox for testing your web scraping script.\n\n## 2.2 \u003ca name=\"rselenium2\"\u003eInteracting with the Web Driver In Rselenium\u003c/a\u003e\n\n### 2.2.1. \u003ca name=\"rselenium2.0\"\u003eHow to Start\u003c/a\u003e\n\n```R\n# Load the Library\nlibrary(RSelenium)\n\n# start the server and browser(you can use other browsers here)\nrD \u003c- rsDriver(browser=c(\"firefox\"))\n\ndriver \u003c- rD[[\"client\"]]\n\n# navigate to an URL\ndriver$navigate(\"http://books.toscrape.com/\")\n\n#close the driver\ndriver$close()\n\n#close the server\nrD[[\"server\"]]$stop()\n```\n\n**Common Issues and Solutions when starting the server and browser** \n\n1. Port already in use.\n    - https://stackoverflow.com/questions/43991498/rselenium-server-signals-port-is-already-in-use.\n    - https://stackoverflow.com/questions/53464640/launch-rselenium-browser-regardless-of-port-open-closed\n    - Something the Java process behind RSelenium does not stop when you apply `rD[[\"server\"]]$stop()`, thereby to ensure shutting down a RSelenium instance you may write:\n    ```r\n    # Kill Java process behind RSelenium\n    system(paste0(\"Taskkill /F /T\" ,\" /PID \", driver$server$process$get_pid()))\n    ```\n    - Try restart Rstudio.\n2. session not created: This version of ChromeDriver only supports Chrome verssion XX.\n    - [Specify chrome version or install latest chrome driver](https://stackoverflow.com/questions/55201226/session-not-created-this-version-of-chromedriver-only-supports-chrome-version-7).\n3. Other issues:\n    - Try reinstall/update RSelenium or wdman package.\n\n\n### 2.2.2. \u003ca name = \"rselenium2.1\"\u003eNavigating to different URLs\u003c/a\u003e\n\n**Using `driver.navigate()`:**\n\n```\ndriver$navigate(\"http://books.toscrape.com/\")\n```\n\n**Simulating a click on the link:**\n\n```R\n# navigate to an URL\ndriver$navigate(\"http://toscrape.com/\")\n\n# find the element\nelements \u003c- driver$findElements(\"a\",using = \"css\")\n\n# click the first link \nelements[[1]]$clickElement()\n```\n\n**Go Back or Go Forward**\n\n```R\ndriver$goBack()\ndriver$goForward()\n```\n\n### 2.2.3. \u003ca name = \"rselenium2.2\"\u003eSimulating Scrolls, Clicks, Text Inputs, Logins, and Other Actions\u003c/a\u003e\n\n**1. Simulating Scrolls:**\n\nYou can scroll up or down by sending the `pageUp` or `pageDown` keypress to the browser. \n\n*Scroll Down Once:*\n\n```R\ndriver$navigate(\"http://quotes.toscrape.com/scroll\")\n# find the webpage body\nelement \u003c- driver$findElement(\"css\", \"body\")\n\n#scroll down once ----\nelement$sendKeysToElement(list(key = \"page_down\"))\n```\n\n*Scroll Down Multiple Times:*\n\n```R\nelement \u003c- driver$findElement(\"css\", \"body\")\n# Scroll down 10 times\nfor(i in 1:10){\n    element$sendKeysToElement(list(\"key\"=\"page_down\"))\n    # please make sure to sleep a couple of seconds to since it takes time to load contents\n    Sys.sleep(2) \n}\n```\n\n*Scroll Down Until the End(Not Recommended if There Are too Many Pages):*\n\n```r\nelement \u003c- driver$findElement(\"css\", \"body\")\nflag \u003c- TRUE\ncounter \u003c- 0\nn \u003c- 5\nwhile(flag){\n    counter \u003c- counter + 1\n    #compare the pagesource every n(n=5) time, since sometimes one scroll down doesn't render new content\n    for(i in 1:n){\n        element$sendKeysToElement(list(\"key\"=\"page_down\"))\n        Sys.sleep(2)\n    }\n    if(exists(\"pagesource\")){\n        if(pagesource == driver$getPageSource()[[1]]){\n            flag \u003c- FALSE\n            writeLines(paste0(\"Scrolled down \",n*counter,\" times.\\n\"))\n        } else {\n            pagesource \u003c- driver$getPageSource()[[1]]\n        }\n    } else {\n        pagesource \u003c- driver$getPageSource()[[1]]\n    }\n}\n```\n\n\n\n**2. Simulating Clicks:**\n\n*Click a Single Element by CSS:*\n\n```R\ndriver$navigate(\"http://quotes.toscrape.com/scroll\")\n\n#locate element using CSS(find the first match)\ndriver$navigate(\"https://scrapethissite.com/pages/ajax-javascript/#2011\")\nelement \u003c- driver$findElement(using = \"css\",\".year-link\")\nelement$clickElement()\n```\n\n*Click a Single Element by Moving Mouse to Element:*\n\n```R\ndriver$navigate(\"https://scrapethissite.com/pages/ajax-javascript/#2011\")\nelement \u003c- driver$findElement(using = \"css\",\".year-link\")\ndriver$mouseMoveToLocation(webElement = element)\ndriver$click()\n```\n\n*Click Multiple Items:*\n\n```R\ndriver$navigate(\"https://scrapethissite.com/pages/ajax-javascript/#2011\")\nelements \u003c- driver$findElements(using = \"css\",\".year-link\")\nfor(element in elements){\n    element$clickElement()\n    Sys.sleep(2)\n}\n```\n\n*Go to Different URLs by Clicking Links:*\n\n```R\ndriver$navigate(\"http://books.toscrape.com/\")\nelements \u003c- driver$findElements(using = \"css\",\".nav-list ul a\")\nfor(i in 1:length(elements)){\n    elements[[i]]$clickElement()\n    Sys.sleep(2)\n    #do something and go back to previous page\n    driver$goBack()\n    Sys.sleep(1)\n    # refresh the elements\n    elements \u003c- driver$findElements(using = \"css\",\".nav-list ul a\")\n}\n```\n\n**3. Simulating Text Input:**\n\n*Enter Text and Search*\n\n```R\ndriver$navigate(\"https://www.google.com/\")\n#selcet input box\nelement \u003c- driver$findElement(using = \"css\",'input[name=\"q\"]')\n#send text to input box. don't forget to use `list()` when sending text\nelement$sendKeysToElement(list(\"Web Scraping\"))\n#select search button\nelement \u003c- driver$findElement(using = \"css\",'input[name=\"btnK\"]')\nelement$clickElement()\n```\n\n\n\n*Clear Input Box*\n\n```R\ndriver$navigate(\"https://www.google.com/\")\n#selcet input box\nelement \u003c- driver$findElement(using = \"css\",'input[name=\"q\"]')\nelement$sendKeysToElement(list(\"Web Scraping\"))\n#clear input box\nelement$clearElement()\n```\n\n\n\n**4. Logins:**\n\nLogin is simple using RSelenium. Instead of doing post request, it's just a combination of sending texts to input boxes and click login button.\n\n\n\n```R\ndriver$navigate(\"http://quotes.toscrape.com/login\")\n#enter username\nelement \u003c- driver$findElement(using = \"css\",\"#username\")\nelement$sendKeysToElement(list(\"myusername\"))\n#enter password\nelement \u003c- driver$findElement(using = \"css\",\"#password\")\nelement$sendKeysToElement(list(\"mypassword\"))\n#click login button\nelement \u003c- driver$findElement(using = \"css\", 'input[type=\"submit\"]')\nelement$clickElement()\n```\n\n\n\n\n\n**5. Simulating Key and Button Presses:**\n\nYou can also send keys to the browser, you can check all available keys by running `RSelenium::selKeys`.\n\n*Available Keys:*\n\n```R\n [1] \"null\"         \"cancel\"       \"help\"         \"backspace\"    \"tab\"          \"clear\"        \"return\"       \"enter\"        \"shift\"        \"control\"     \n[11] \"alt\"          \"pause\"        \"escape\"       \"space\"        \"page_up\"      \"page_down\"    \"end\"          \"home\"         \"left_arrow\"   \"up_arrow\"    \n[21] \"right_arrow\"  \"down_arrow\"   \"insert\"       \"delete\"       \"semicolon\"    \"equals\"       \"numpad_0\"     \"numpad_1\"     \"numpad_2\"     \"numpad_3\"    \n[31] \"numpad_4\"     \"numpad_5\"     \"numpad_6\"     \"numpad_7\"     \"numpad_8\"     \"numpad_9\"     \"multiply\"     \"add\"          \"separator\"    \"subtract\"    \n[41] \"decimal\"      \"divide\"       \"f1\"           \"f2\"           \"f3\"           \"f4\"           \"f5\"           \"f6\"           \"f7\"           \"f8\"          \n[51] \"f9\"           \"f10\"          \"f11\"          \"f12\"          \"command_meta\"\n```\n\n\n\n*Send Key Combinations:*\n\n```R\ndriver$navigate(\"https://scrapethissite.com/pages/ajax-javascript/#2015\")\nSys.sleep(2)\n# Press Control+A to select the entire page\ndriver$findElement(using = \"css\",\"body\")$sendKeysToElement(list(key=\"control\",\"a\"))\n\n```\n\n\n\n\n## 2.3. \u003ca name=\"rselenium3\"\u003eExtracting Texts from Web Pages\u003c/a\u003e\n\n### 2.3.1. \u003ca name=\"rselenium3.1\"\u003eExtracting Content using Rselenium\u003c/a\u003e\n\nYou can use `findElement()` or `findElements()` to extract single or multiple elements from page source.\n\n**1.  Extract text from a single element:**\n\nuse `findElment()` method to select a single matching element, and use `getElementText()` method to extract text.\n\n```R\ndriver$navigate(\"https://scrapethissite.com/pages/simple/\")\nelement \u003c- driver$findElement(using = \"css\",\".country-capital\")\n# get element text\nelement$getElementText()[[1]]\n#get element attribute value(get the value for class)\nelement$getElementAttribute(\"class\")[[1]]\n```\n\n\n\n**2. Extract text from a list of elements:**\n\n`RSelenium` doesn't support vectorized calculation.So you need to use for loops, apply or map(in `purrr` package) as alternative to get lists of items.\n\nuse `findElements()` method to select all matching elements, and use `getElementText()` method to extract text.\n\n```R\ndriver$navigate(\"https://scrapethissite.com/pages/simple/\")\nSys.sleep(2)\nelements \u003c- driver$findElements(using = \"css\",\".country-capital\")\n```\n\n\n\n*Use for loop to extract elements:*\n\n```R\ntexts \u003c- c()\nfor(ele in elements){\n    texts \u003c- append(texts,ele$getElementText()[[1]])\n}\n```\n\n\n\n*Use apply functions to extract elements:*\n\n```R\n# use lapply\ntexts_lapply \u003c- lapply(elements,function(x) x$getElementText()[[1]])\n# use sapply\ntexts_sapply \u003c- sapply(elements,function(x) x$getElementText()[[1]])\n```\n\n\n\n*Use map functions(using `purrr`) to extract elements:*\n\n```R\nlibrary(purrr)\n# get result as a list\ntexts_purrr_map \u003c- map(elements,~ .x$getElementText()[[1]])\n# get result as a string vector\ntexts_purrr_map_chr \u003c- map_chr(elements,~ .x$getElementText()[[1]])\n```\n\n \n\n**3. Extract attribute(s) from a single element(s)**\n\nThe usage is the same as extracting text from element(s), the only difference it that you need to use `getElementAttribute()` method.\n\n```R\nbook_partial_links \u003c- lapply(elements,function(x) x$getElementAttribute(attrName=\"href\"))\n```\n\n\n\n\n\n### 2.3.2. \u003ca name=\"rselenium3.2\"\u003eExtracting Content using Parsed Page Source and `rvest`\u003c/a\u003e\n\nSince the browser will execute JavaScript codes by default, you can use `RSelenium` as a tool to get the parsed content from webpages. In this way, you can benefit from the vectorized calculation in `rvest`. \n\nAnother benefit of using `rvest` is that it runs much faster while extracting information from page source comparing to using `getElementText()` or `getElementAttribute()` method.\n\n```R\nlibrary(rvest)\ndriver$navigate(\"https://scrapethissite.com/pages/ajax-javascript/#2015\")\nSys.sleep(2)\n# get parsed page source\nparsed_pagesource \u003c- driver$getPageSource()[[1]]\n# using rvest to extract information\nresult \u003c- read_html(parsed_pagesource) %\u003e%\n    html_elements(\".film-title\") %\u003e%\n    html_text()\n```\n\n\n\n## 2.4. \u003ca name = \"rselneium4\"\u003emiscellanea\u003c/a\u003e\n\n### 2.4.1. \u003ca name=\"rselenium4.1\"\u003eJavascript\u003c/a\u003e\n\nCongratulations, no more concerns about AJAX or JavaScript as the browser will parse everything for you!\n\nBut, of course, if you still need to run some javascript, then you can use `your_driver$executeScript()`\n\n### 2.4.2. \u003ca name=\"rselenium4.2\"\u003eIframe\u003c/a\u003e\n\nLet's try to get all turtles family name in this [webpage](https://scrapethissite.com/pages/frames/), it doesn't work because the content is another webpage nested in the current one.\n\n```R\ndriver$navigate(\"https://scrapethissite.com/pages/frames/\")\n# select all turtles' name\nelements \u003c- driver$findElements(using = \"css\",\".family-name\")\nprint(elements)\n\u003e list()\n```\n\n\n\nTo read the content from inside iframe, we can use `switchToFrame` method.\n\n\u003e switchToFrame(Id)\n\u003e Change focus to another frame on the page. Id can be string|number|null|WebElement Object. If the Id is null, the server should switch to the page's default content.\n\n```R\ndriver$navigate(\"https://scrapethissite.com/pages/frames/\")\n# select iframe element\nelement \u003c- driver$findElement(using = \"css\",\"#iframe\")\n#switch to the iframe\ndriver$switchToFrame(element)\n\nelements \u003c- driver$findElements(using = \"css\",\".family-name\")\nfor (ele in elements){\n    print(ele$getElementText()[[1]])\n}\n#switch back to the default page\ndriver$switchToFrame()\n```\n\n# 3 \u003ca name = \"change\"\u003eChange Log\u003c/a\u003e\n\n- 2021-10-25: `html_node` changed to `html_element`, `html_nodes` changed to `html_elements`.\n    - Thanks [@victorcaquilpan](https://github.com/victorcaquilpan) for this suggestion.\n    - The reason for this chagne is mentioned in [rvest 1.0.0](https://www.tidyverse.org/blog/2021/03/rvest-1-0-0/)\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fyusuzech%2Fr-web-scraping-cheat-sheet","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fyusuzech%2Fr-web-scraping-cheat-sheet","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fyusuzech%2Fr-web-scraping-cheat-sheet/lists"}