{"id":19497096,"url":"https://github.com/rlesur/crrri","last_synced_at":"2025-04-25T22:31:44.978Z","repository":{"id":78261974,"uuid":"157903442","full_name":"RLesur/crrri","owner":"RLesur","description":"A Chrome Remote Interface written in R","archived":false,"fork":false,"pushed_at":"2022-08-22T15:48:30.000Z","size":1214,"stargazers_count":157,"open_issues_count":34,"forks_count":12,"subscribers_count":10,"default_branch":"master","last_synced_at":"2024-08-06T03:04:30.854Z","etag":null,"topics":["chrome-devtools","chrome-headless","r","r-package","rstats"],"latest_commit_sha":null,"homepage":"https://rlesur.github.io/crrri/","language":"R","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"other","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/RLesur.png","metadata":{"files":{"readme":"README.Rmd","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null}},"created_at":"2018-11-16T18:00:50.000Z","updated_at":"2024-07-02T08:05:07.000Z","dependencies_parsed_at":"2023-03-11T13:00:23.718Z","dependency_job_id":null,"html_url":"https://github.com/RLesur/crrri","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/RLesur%2Fcrrri","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/RLesur%2Fcrrri/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/RLesur%2Fcrrri/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/RLesur%2Fcrrri/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/RLesur","download_url":"https://codeload.github.com/RLesur/crrri/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":224019583,"owners_count":17242176,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["chrome-devtools","chrome-headless","r","r-package","rstats"],"created_at":"2024-11-10T21:43:08.159Z","updated_at":"2024-11-10T21:43:09.285Z","avatar_url":"https://github.com/RLesur.png","language":"R","funding_links":[],"categories":[],"sub_categories":[],"readme":"---\noutput: github_document\n---\n\n\u003c!-- README.md is generated from README.Rmd. Please edit that file --\u003e\n\n```{r setup, include = FALSE}\nknitr::opts_chunk$set(\n  collapse = TRUE,\n  eval = FALSE,\n  comment = \"#\u003e\",\n  fig.path = \"man/figures/README-\",\n  out.width = \"100%\"\n)\nSys.unsetenv(\"DEBUGME\")\n```\n# crrri\n\n\u003c!-- badges: start --\u003e\n[![Lifecycle: experimental](https://img.shields.io/badge/lifecycle-experimental-orange.svg)](https://www.tidyverse.org/lifecycle/#experimental)\n[![Codecov test coverage](https://codecov.io/gh/RLesur/crrri/branch/master/graph/badge.svg)](https://codecov.io/gh/RLesur/crrri?branch=master)\n[![CRAN status](https://www.r-pkg.org/badges/version/crrri)](https://cran.r-project.org/package=crrri)\n[![R build status](https://github.com/RLesur/crrri/workflows/R-CMD-check/badge.svg)](https://github.com/RLesur/crrri/actions)\n\u003c!-- badges: end --\u003e\n\n**Work in progress**\n\nThe goal of `crrri` is to provide a native Chrome Remote Interface in R using the [Chrome Debugging Protocol](https://chromedevtools.github.io/devtools-protocol/). This is a low-level implementation of the protocol heavily inspired by the [`chrome-remote-interface`](https://github.com/cyrus-and/chrome-remote-interface) JavaScript library written by [Andrea Cardaci](https://github.com/cyrus-and).\n\nThis package is intended to R packages developers who need to orchestrate Chromium/Chrome: **with `crrri`, you can easily interact with (headless) Chromium/Chrome using R**. We worked a lot to provide the most simple API. However, you will have the bulk of the work and learn how the Chrome DevTools Protocol works. Interacting with Chromium/Chrome using the DevTools Protocol is a highly technical task and prone to errors: you will be close to the metal and have full power (be cautious!).\n\nThis package is built on top of the [`websocket`](https://github.com/rstudio/websocket) and [`promises`](https://cran.r-project.org/package=promises) packages. The default design of the `crrri` functions is asynchronous: they return promises. You can also use `crrri` with callbacks if you prefer. \n\nWe are highly indebted to [Miles McBain](https://github.com/milesmcbain) for his seminal work on [`chradle`](https://github.com/milesmcbain/chradle) that inspired us. Many thanks!\n\n## System requirements\n\nFirst of all, you **do not need a `node.js` configuration** because **`crrri` is\nfully written in R**.\n\nYou only need a recent version of Chromium or Chrome. A standalone version works perfectly well on Windows. By default, `crrri` will try to find a chrome binary on your system to use, using the `find_chrome_binary()`. You can tell `crrri` to use a specific version by setting the value of the `HEADLESS_CHROME` environment variable to the path of Chromium or Chrome (this is the same environment variable that is used in [`decapitated`](https://github.com/hrbrmstr/decapitated)). You can check it is set correctly by executing `Sys.getenv(\"HEADLESS_CHROME\")` in your R console.\n\nOtherwise, you can also use the `bin` argument of the `Chrome` class `new()` method to provide the path directly.\n```r\nchrome \u003c- Chrome$new(bin = \"\u003cpath-to-chrome-binary-\u003e\")\n```\n\nNote that if ever you don't know where your binary is, you can use directly the `find_chrome_binary()` function, which will try to guess where your binary is (you might neeed to install the package). \n\nThis two calls are equivalent\n\n```r\nchrome \u003c- Chrome$new(bin = find_chrome_binary())\n# the default\nchrome \u003c- Chrome$new(bin = NULL)\n```\n\n## Installation\n\nYou can install the development version of `crrri` from GitHub with:\n\n```{r, eval=FALSE}\nremotes::install_github('rlesur/crrri')\n```\n## Using `crrri` interactively\n\nThe `crrri` package is a low-level interface and **is not intended to be used interactively**: the goal of `crrri` is to provide to R developers a set of classes and helper functions to build higher levels functions.\n\n**However, you can discover headless Chrome automation interactively in your R session using `crrri`**. This will help you to learn the [Chrome DevTools Protocol](https://chromedevtools.github.io/devtools-protocol), the `crrri` design and develop higher level functions.\n\n### A short-tour\n\nAssuming that you have configured the `HEADLESS_CHROME` environment variable (see [above](#system-requirements)), you can start headless Chrome:\n\n```{r}\nlibrary(crrri)\nchrome \u003c- Chrome$new()\n```\n\nThe `Chrome` class constructor is a **synchronous function**. That means the R session is on hold until the command terminates.\n\nThe `$connect()` method of the `Chrome` class will connect the R session to headless Chrome. As the connection process can take some time, the R session does not hold^[most of R users should think that this behavior is weird but it is extremely powerful!]: this is an **asynchronous function**. This function returns a promise which is fulfilled when R is connected to Chrome. \n\nHowever, you can pass a callback function to the `$connect()` method using its `callback` argument. In this case, the returned object will be a connection object:\n\n```{r}\nclient \u003c- chrome$connect(callback = function(client) {\n  client$inspect()\n})\n```\n\nThe `$inspect()` method of the connection object opens the Chrome DevTools Inspector in RStudio (\u003e= 1.2.1335) or in your default web browser (you can have some trouble if the inspector is not opened in Chromium/Chrome). It is convenient if you need to inspect the content of a web page because all that you need is in RStudio.\n\n![DevTools Inspector in RStudio viewer](https://user-images.githubusercontent.com/19177171/56867255-861c3900-69e3-11e9-88cd-2ef29075070f.png)\n\nIn order to discover the [Chrome DevTools Protocol](https://chromedevtools.github.io/devtools-protocol) commands and events listeners, it is recommended to extract one of the domains^[a domain is a set of commands, events listeners and types.] from the connection object:\n\n```{r}\nPage \u003c- client$Page\n```\n\nThe `Page` object represents the [`Page` domain](https://chromedevtools.github.io/devtools-protocol/tot/Page). It possesses methods to send commands or listen to specific events.\n\nFor instance, you can send to Chromium/Chrome the [`Page.navigate`](https://chromedevtools.github.io/devtools-protocol/tot/Page#method-navigate) command as follows:\n\n```{r}\nPage$navigate(url = \"http://r-project.org\")\n```\nOnce the page is loaded by headless Chrome, RStudio looks like this:\n\n![R Project website in headless Chrome](https://user-images.githubusercontent.com/19177171/56867262-8f0d0a80-69e3-11e9-828f-4dddb0bcd492.png)\n\nYou will see in the R console:\n\n```\n\u003cPromise [pending]\u003e\n```\n\nThis is a promise object that is fulfilled when Chromium/Chrome sends back to R a message telling that the command was well-received. This comes from the fact that the `Page$navigate()` function is also asynchronous. All the asynchronous methods possess a `callback` argument. When the R session receives the result of the command from Chrome, R executes this callback function passing the result object to this function. For instance, you can execute:\n\n```{r}\nPage$navigate(url = \"https://ropensci.org/\", callback = function(result) {\n  cat(\"The R session has received this result from Chrome!\\n\")\n  print(result)\n})\n```\n\nOnce the page is loaded, you will see both the web page and the result object object in RStudio:\n\n![rOpenSci website in headless Chrome](https://user-images.githubusercontent.com/19177171/56867269-9cc29000-69e3-11e9-8fa4-ca238d3b3566.png)\n\nTo inspect the result of a command you can pass the `print` function to the `callback` argument:\n\n```{r}\nPage$navigate(url = \"https://ropensci.org/\", callback = print)\n```\n\n```\n#\u003e $frameId\n#\u003e [1] \"3BB38B10082F28A946332100964486EC\"\n#\u003e \n#\u003e $loaderId\n#\u003e [1] \"9DCF07625678433563CB03FFF1E8A6AB\"\n```\nThe result object sent back from Chrome is also the value of the promises once fulfilled. Recall that if you do not use a callback function, you get a promise:\n\n```{r}\nasync_result \u003c- Page$navigate(url = \"http://r-project.org\")\n```\n\nYou can print the value of this promise once fulfilled with:\n\n```{r}\nasync_result %...\u003e% print()\n```\n\n```\n#\u003e $frameId\n#\u003e [1] \"3BB38B10082F28A946332100964486EC\"\n#\u003e \n#\u003e $loaderId\n#\u003e [1] \"7B2383E8F2F39273E18E4D918F1852A0\"\n```\n\nAs you can see, this leads to the same result as with a callback function.\n\nTo sum up, these two forms perform the same actions:\n\n```{r}\nPage$navigate(url = \"http://r-project.org\", callback = print)\nPage$navigate(url = \"http://r-project.org\") %...\u003e% print()\n```\n\nIf you interact with headless Chrome in the R console using `crrri`, these two forms are equivalent.  \n**However, if you want to use `crrri` to develop higher level functions, the most reliable way is to use promises.**\n\nDo not forget to close headless Chrome with:\n\n```{r}\nchrome$close()\n```\n\nSince the RStudio viewer has lost the connection, you will see this screen in RStudio:\n\n![headless Chrome closed](https://user-images.githubusercontent.com/19177171/56867276-a4823480-69e3-11e9-8530-831ac4dd144e.png)\n\nNow, you can take some time to discover all the commands and events of the [Chrome DevTools Protocol](https://chromedevtools.github.io/devtools-protocol/). The following examples will introduce some of them.\n\n### Domains, commands and events listeners\n\nWhile working interactively, you can obtain the list of available domains in your version of Chromium/Chrome.  \nFirst, launch Chromium/Chrome and connect the R session to headless Chromium/Chrome:\n\n```r\nchrome \u003c- Chrome$new()\nclient \u003c- chrome$connect(~ .x$inspect())\n```\n\nOnce connected, you just have to print the connection object to get informations about the connection and availables domains:\n\n```r\nclient\n```\n\n```\n#\u003e \u003cCDP CONNECTION\u003e\n#\u003e connected to: http://localhost:9222/\n#\u003e  target type: \"page\"\n#\u003e    target ID: \"9A576420CADEA9A514C5F027D30B410D\"\n#\u003e \u003cDOMAINS\u003e\n#\u003e \n#\u003e Accessibility (experimental)\n#\u003e \n#\u003e Animation (experimental)\n#\u003e \n#\u003e ApplicationCache (experimental)\n#\u003e \n#\u003e Audits (experimental): Audits domain allows investigation of page violations and possible improvements.\n#\u003e \n#\u003e Browser: The Browser domain defines methods and events for browser managing.\n#\u003e \n#\u003e CacheStorage (experimental)\n#\u003e \n#\u003e Cast (experimental): A domain for interacting with Cast, Presentation API, and Remote Playback API functionalities.\n...\n```\n\nThese informations are directly retrieved from Chromium/Chrome: you may obtain different informations depending on the Chromium/Chrome version.\n\nIn the most recent versions of the Chrome DevTools Protocol, more than 40 domains are available. A domain is a set of commands and events listeners.\n\nIn order to work with a domain, it is recommended to extract it from the connection object. For instance, if you want to access to the `Runtime` domain, execute:\n\n```r\nRuntime \u003c- client$Runtime\n```\n\nIf you print this object, this will open the online documentation about this domain in your browser:\n\n```r\nRuntime # opens the online documentation in a browser\n```\n\n## Presentations about crrri\n\n* **uros2019** - 20/05/2019 ([slides](https://speakerdeck.com/rlesur/headless-chrome-automation-with-r-the-crrri-package))\n* **useR!2019** - 12/07/2019 ([slides](https://cderv.gitlab.io/user2019-crrri/))\n\n## Examples\n\n### Generate a PDF\n\nHere is an example that produces a PDF of the [R Project website](https://www.r-project.org/): \n\n```{r, results='hide'}\nlibrary(promises)\nlibrary(crrri)\nlibrary(jsonlite)\n\nperform_with_chrome(function(client) {\n  Page \u003c- client$Page\n\n  Page$enable() %...\u003e% { # await enablement of the Page domain\n    Page$navigate(url = \"https://www.r-project.org/\") \n    Page$loadEventFired() # await the load event\n  } %...\u003e% {\n    Page$printToPDF() \n  } %...\u003e% { # await PDF reception\n    .$data %\u003e% base64_dec() %\u003e% writeBin(\"r_project.pdf\") \n  }\n})\n```\n\nAll the functions of the `crrri` package (commands and event listeners) return promises (as defined in the **promises** package) by default. When building higher level functions, do not forget that you have to deal with promises (those will prevent you to fall into the _Callback Hell_).\n\nFor instance, you can write a `save_as_pdf` function as follow:\n\n```{r}\nsave_url_as_pdf \u003c- function(url) {\n  function(client) {\n    Page \u003c- client$Page\n\n    Page$enable() %...\u003e% {\n      Page$navigate(url = url)\n      Page$loadEventFired()\n    } %...\u003e% {\n      Page$printToPDF()\n    } %...\u003e% {\n      .$data %\u003e%\n        jsonlite::base64_dec() %\u003e%\n        writeBin(paste0(httr::parse_url(url)$hostname, \".pdf\"))\n    }\n  }\n}\n```\n\nYou can pass several functions to `perform_with_chrome()`:\n\n```{r}\nsave_as_pdf \u003c- function(...) {\n  list(...) %\u003e%\n    purrr::map(save_url_as_pdf) %\u003e%\n    perform_with_chrome(.list = .)\n}\n```\n\nYou have created a `save_as_pdf()` function that can handle multiple URLs:\n\n```{r}\nsave_as_pdf(\"http://r-project.org\", \"https://ropensci.org/\", \"https://rstudio.com\")\n```\n\n### Transpose `chrome-remote-interface` JS scripts: dump the DOM\n\nWith `crrri`, you should be able to transpose with minimal efforts some JS scripts written with the [`chrome-remote-interface`](https://github.com/cyrus-and/chrome-remote-interface) node.js module.\n\nFor instance, take [this JS script](https://github.com/cyrus-and/chrome-remote-interface/wiki/Dump-HTML-after-page-load) that dumps the DOM:\n\n```js\nconst CDP = require('chrome-remote-interface');\n\nCDP(async(client) =\u003e {\n    const {Network, Page, Runtime} = client;\n    try {\n        await Network.enable();\n        await Page.enable();\n        await Network.setCacheDisabled({cacheDisabled: true});\n        await Page.navigate({url: 'https://github.com'});\n        await Page.loadEventFired();\n        const result = await Runtime.evaluate({\n            expression: 'document.documentElement.outerHTML'\n        });\n        const html = result.result.value;\n        console.log(html);\n    } catch (err) {\n        console.error(err);\n    } finally {\n        client.close();\n    }\n}).on('error', (err) =\u003e {\n    console.error(err);\n});\n```\n\nUsing `crrri`, you can write:\n\n```{r, results='hide'}\nlibrary(promises)\nlibrary(crrri)\n\nasync_dump_DOM \u003c- function(client) {\n  Network \u003c- client$Network\n  Page \u003c- client$Page\n  Runtime \u003c- client$Runtime\n  Network$enable() %...\u003e% { \n    Page$enable()\n  } %...\u003e% {\n    Network$setCacheDisabled(cacheDisabled = TRUE)\n  } %...\u003e% {\n    Page$navigate(url = 'https://github.com')\n  } %...\u003e% {\n    Page$loadEventFired()\n  } %...\u003e% {\n    Runtime$evaluate(\n      expression = 'document.documentElement.outerHTML'\n    )\n  } %...\u003e% (function(result) {\n    html \u003c- result$result$value\n    cat(html, \"\\n\")\n  }) \n}\n\nperform_with_chrome(async_dump_DOM)\n```\n\nIf you want to write a higher level function that dump the DOM, you can embed the main part of this script in a function:\n\n```{r}\ndump_DOM \u003c- function(url) {\n  perform_with_chrome(function(client) {\n    Network \u003c- client$Network\n    Page \u003c- client$Page\n    Runtime \u003c- client$Runtime\n    Network$enable() %...\u003e% { \n      Page$enable()\n    } %...\u003e% {\n      Network$setCacheDisabled(cacheDisabled = TRUE)\n    } %...\u003e% {\n      Page$navigate(url = url)\n    } %...\u003e% {\n      Page$loadEventFired()\n    } %...\u003e% {\n      Runtime$evaluate(\n        expression = 'document.documentElement.outerHTML'\n      )\n    } %...\u003e% (function(result) {\n      html \u003c- result$result$value\n      cat(html, \"\\n\")\n    }) \n  })\n}\n```\n\nNow, you can use it for dumping [David Gohel](https://github.com/davidgohel)'s [blog](http://www.ardata.fr/blog/):\n\n```{r, results='hide'}\ndumpDOM(url = \"http://www.ardata.fr/blog/\")\n```\n\nYou can find many other examples in the [wiki](https://github.com/cyrus-and/chrome-remote-interface/wiki) of the `chrome-remote-interface` module.\n\n\n## Development\n\n### Logging Messages\n\nIn `crrri`, there are two types of messages: \n\n- Those sent during connection/disconnection (mainly for crrri debugging) \n- Those tracking the exchanges between the R websocket client and the remote\nwebsocket server. These lasts are essential for R devs to develop higher levels\npackages, either during the development process and for debugging purposes.\n\n`crrri` uses [`debugme`](https://github.com/r-lib/debugme) for printing those\nmessages. It is disable by default and you won't see any messages - as a user we\nthink it is fine. However, if you are a developer, you would expect some\ninformation on what is going on.\n\nYou need to add `\"crrri\"` to the `DEBUGME` environment variable before loading\nthe package to activate the messaging feature. Currently in `crrri` there is\nonly one level of message.Also, `debugme` is a Suggested dependency and you may\nneed to install it manually if not already installed.\n\n## Credits\n\nAndrea Cardaci for `chrome-remote-interface`.\n\nMiles McBain for `chradle`.\n\nBob Rudis for `decapitated`.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Frlesur%2Fcrrri","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Frlesur%2Fcrrri","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Frlesur%2Fcrrri/lists"}