{"id":13858377,"url":"https://github.com/hrbrmstr/splashr","last_synced_at":"2025-10-13T05:16:20.964Z","repository":{"id":141238847,"uuid":"80864807","full_name":"hrbrmstr/splashr","owner":"hrbrmstr","description":":sweat_drops: Tools to Work with the 'Splash' JavaScript Rendering Service in R","archived":false,"fork":false,"pushed_at":"2020-02-19T17:59:44.000Z","size":12446,"stargazers_count":98,"open_issues_count":14,"forks_count":5,"subscribers_count":12,"default_branch":"master","last_synced_at":"2025-06-24T15:03:59.229Z","etag":null,"topics":["har","phantomjs","r","r-cyber","rstats","selenium","splash","web-scraping"],"latest_commit_sha":null,"homepage":null,"language":"R","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"other","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/hrbrmstr.png","metadata":{"files":{"readme":"README.Rmd","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null}},"created_at":"2017-02-03T20:04:11.000Z","updated_at":"2025-03-22T11:18:43.000Z","dependencies_parsed_at":null,"dependency_job_id":"fabbe0e6-c55d-446a-81d9-8262fd45bca5","html_url":"https://github.com/hrbrmstr/splashr","commit_stats":null,"previous_names":[],"tags_count":1,"template":false,"template_full_name":null,"purl":"pkg:github/hrbrmstr/splashr","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/hrbrmstr%2Fsplashr","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/hrbrmstr%2Fsplashr/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/hrbrmstr%2Fsplashr/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/hrbrmstr%2Fsplashr/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/hrbrmstr","download_url":"https://codeload.github.com/hrbrmstr/splashr/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/hrbrmstr%2Fsplashr/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":266127232,"owners_count":23880421,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["har","phantomjs","r","r-cyber","rstats","selenium","splash","web-scraping"],"created_at":"2024-08-05T03:02:06.562Z","updated_at":"2025-10-13T05:16:15.923Z","avatar_url":"https://github.com/hrbrmstr.png","language":"R","funding_links":[],"categories":["R"],"sub_categories":[],"readme":"---\noutput: rmarkdown::github_document\n---\n[![Travis-CI Build Status](https://travis-ci.org/hrbrmstr/splashr.svg?branch=master)](https://travis-ci.org/hrbrmstr/splashr)\n[![Coverage Status](https://img.shields.io/codecov/c/github/hrbrmstr/splashr/master.svg)](https://codecov.io/github/hrbrmstr/splashr?branch=master)\n[![CRAN_Status_Badge](http://www.r-pkg.org/badges/version/splashr)](https://cran.r-project.org/package=splashr)\n[![](http://cranlogs.r-pkg.org/badges/splashr)](http://cran.rstudio.com/web/packages/splashr/index.html)\n\n# `splashr` : Tools to Work with the 'Splash' JavaScript Rendering Service\n\nTL;DR: This package works with Splash rendering servers which are really just a REST API \u0026 `lua` scripting interface to a QT browser. It's an alternative to the Selenium ecosystem which was really engineered for application testing \u0026 validation. \n\nSometimes, all you need is a page scrape after javascript has been allowed to roam wild and free over meticulously crafted HTML tags. So, this package does not do _everything_ Selenium can in pure R (though, the Lua interface is equally as powerful and accessible via R), but if you're just trying to get a page back that needs javascript rendering, this is a nice, lightweight, consistent alternative.\n\nIt's also an alternative to the somewhat abandoned `phantomjs` (which you can use in R within or without a Selenium context as it's it's own webdriver) and it may be useful to compare renderings between this package \u0026 `phantomjs`.\n\nThe package uses the [`stevedore`](https://github.com/richfitz/stevedore) package to orchestrate Docker on your system (if you have Docker and more on how to use the `stevedore` integration below) but you can also do get it running in Docker on the command-line with two commands:\n\n    sudo docker pull scrapinghub/splash:latest --disable-browser-caches\n    sudo docker run -p 5023:5023 -p 8050:8050 -p 8051:8051 scrapinghub/splash:latest --disable-browser-caches\n    \nDo whatever you Windows ppl do with Docker on your systems to make ^^ work. \n\nFolks super-new to Docker on Unix-ish platforms should [make sure to do](https://github.com/hrbrmstr/splashr/issues/3#issuecomment-280686494):\n\n    sudo groupadd docker\n    sudo usermod -aG docker $USER\n\n(`$USER` is your username and shld be defined for you in the environment)\n\nIf using the [`stevedore`](https://github.com/richfitz/stevedore) package you can use the convience wrappers in this pacakge:\n\n    install_splash()\n    splash_container \u003c- start_splash()\n\nand then run:\n\n    stop_splash(splash_container)\n    \nwhen done. All of that happens on your `localhost` and you will not need to specify `splash_obj` to many of the `splashr` functions if you're running Splash in this default configuration as long as you use named parameters. You can also use the pre-defined `splash_local` object if you want to use positional parameters.\n\nNow, you can run Selenium in Docker, so this is not unique to Splash. But, a Docker context makes it so that you don't have to run or maintain icky Python stuff directly on your system. Leave it in the abandoned warehouse district where it belongs.\n\nAll you need for this package to work is a running Splash instance. You provide the host/port for it and it's scrape-tastic fun from there!\n\n### About Splash\n\n\u003e'Splash' \u003chttps://github.com/scrapinghub/splash\u003e is a javascript rendering service. It’s a lightweight web browser with an 'HTTP' API, implemented in Python using 'Twisted'and 'QT' [and provides some of the core functionality of the 'RSelenium' or 'seleniumPipes' R packages but with a Java-free footprint]. The (twisted) 'QT' reactor is used to make the sever fully asynchronous allowing to take advantage of 'webkit' concurrency via QT main loop. Some of Splash features include the ability to process multiple webpages in parallel; retrieving HTML results and/or take screenshots; disabling images or use Adblock Plus rules to make rendering faster; executing custom JavaScript in page context; getting detailed rendering info in HAR format.\n    \nThe following functions are implemented:\n\n- `render_html`:\tReturn the HTML of the javascript-rendered page.\n- `render_har`: Return information about Splash interaction with a website in [HAR](http://www.softwareishard.com/blog/har-12-spec/) format.\n- `render_jpeg`:\tReturn a image (in JPEG format) of the javascript-rendered page.\n- `render_png`:\tReturn a image (in PNG format) of the javascript-rendered page.\n- `execute_lua`: Execute a custom rendering script and return a result.\n- `splash`:\tConfigure parameters for connecting to a Splash server\n- `install_splash`:\tRetrieve the Docker image for Splash\n- `start_splash`:\tStart a Splash server Docker container\n- `stop_splash`:\tStop a running a Splash server Docker container\n\nMini-DSL (domain-specific language). These can be used to create a \"script\" without actually scripting in Lua. They are a less-powerful/configurable set of calls than what you can make with a full Lua function but the idea is to have it take care of very common but simple use-cases, like waiting a period of time before capturing a HAR/HTML/PNG image of a site:\n\n- `splash_plugins`:\tEnable or disable browser plugins (e.g. Flash).\n- `splash_click`:\tTrigger mouse click event in web page.\n- `splash_focus`:\tFocus on a document element provided by a CSS selector\n- `splash_images`:\tEnable/disable images\n- `splash_response_body`:\tEnable or disable response content tracking.\n- `splash_go`:\tGo to an URL.\n- `splash_wait`:\tWait for a period time\n- `splash_har`:\tReturn information about Splash interaction with a website in HAR format.\n- `splash_html`:\tReturn a HTML snapshot of a current page.\n- `splash_png`:\tReturn a screenshot of a current page in PNG format.\n- `splash_press`:\tTrigger mouse press event in web page.\n- `splash_release`:\tTrigger mouse release event in web page.\n- `splash_send_keys`:\tSend keyboard events to page context.\n- `splash_send_text`:\tSend text as input to page context, literally, character by character.\n- `splash_user_agent`:\tOverwrite the User-Agent header for all further requests. NOTE: There are many \"helper\" user agent strings to go with `splash_user_agent`. Look for objects in `splashr` starting with `ua_`.\n\n`httr` helpers. These help turn various bits of `splashr` objects into `httr`-ish things:\n\n- `as_req`:  Turn a HAR response entry into a working `httr` function you can use to make a request with\n- `as_request`:  Turn a HAR response entry into an `httr` `response`-like object (i.e. you can use `httr::content()` on it)\n\nHelpers:\n\n- `tidy_har`: Turn a gnHARly HAR object into a tidy data frame\n- `get_body_size`:\tRetrieve size of content | body | headers\n- `get_content_sie`:\tRetrieve size of content | body | headers\n- `get_content_type`\tRetrieve or test content type of a HAR request object\n- `get_headers_size`\tRetrieve size of content | body | headers\n- `is_binary`:\tRetrieve or test content type of a HAR request object\n- `is_content_type`:\tRetrieve or test content type of a HAR request object\n- `is_css`:\tRetrieve or test content type of a HAR request object\n- `is_gif`:\tRetrieve or test content type of a HAR request object\n- `is_html`:\tRetrieve or test content type of a HAR request object\n- `is_javascript`:\tRetrieve or test content type of a HAR request object\n- `is_jpeg`:\tRetrieve or test content type of a HAR request object\n- `is_json`:\tRetrieve or test content type of a HAR request object\n- `is_plain`:\tRetrieve or test content type of a HAR request object\n- `is_png`:\tRetrieve or test content type of a HAR request object\n- `is_svg`:\tRetrieve or test content type of a HAR request object\n- `is_xhr`: Retrieve or test content type of a HAR request object\n- `is_xml`:\tRetrieve or test content type of a HAR request object\n\nSome functions from `HARtools` are imported/exported and `%\u003e%` is imported/exported.\n\n### TODO\n\nSuggest more in a feature req!\n\n- \u003cstrike\u003eImplement `render.json`\u003c/strike\u003e\n- \u003cstrike\u003eImplement \"file rendering\"\u003c/strike\u003e\n- \u003cstrike\u003eImplement `execute` (you can script Splash!)\u003c/strike\u003e\n- \u003cstrike\u003eAdd integration with [`HARtools`](https://github.com/johndharrison/HARtools)\u003c/strike\u003e\n- \u003cstrike\u003e_Possibly_ writing R function wrappers to install/start/stop Splash\u003c/strike\u003e which would also support enabling javascript profiles, request filters and proxy profiles from with R directly, using [`harbor`](https://github.com/wch/harbor)\n- Re-implement `render_file()`\n- Testing results with all combinations of parameters\n\n### Installation\n\n```{r eval=FALSE}\n# CRAN\ninstall.packages(\"splashr\")\n\n# DEV\n# See DESCRIPTION for non-CINC-provided dependencies \ninstall.packages(\"splashr\", repos = c(\"https://cinc.rud.is/\"))\n```\n\n```{r message=FALSE, warning=FALSE, error=FALSE, include=FALSE}\noptions(width=120)\n```\n\n### Usage\n\nNOTE: ALL of these examples assume Splash is running in the default configuration on `localhost` (i.e. started with `start_splash()` or the docker example commands) unless otherwise noted.\n\n```{r message=FALSE, warning=FALSE, error=FALSE}\nlibrary(splashr)\nlibrary(magick)\nlibrary(rvest)\nlibrary(anytime)\nlibrary(tidyverse)\n\n# current version\npackageVersion(\"splashr\")\n\nsplash_active()\n\nsplash_debug()\n```\n\nNotice the difference between a rendered HTML scrape and a non-rendered one:\n\n```{r}\nrender_html(url = \"http://marvel.com/universe/Captain_America_(Steve_Rogers)\")\n\nxml2::read_html(\"http://marvel.com/universe/Captain_America_(Steve_Rogers)\")\n```\n\nYou can also profile pages:\n\n```{r}\nrender_har(url = \"http://www.poynter.org/\") -\u003e har\n\nprint(har)\n\ntidy_har(har)\n```\n\nYou can use [`HARtools::HARviewer`](https://github.com/johndharrison/HARtools/blob/master/R/HARviewer.R) — which this pkg import/exports — to get view the HAR in an interactive HTML widget.\n\nFull web page snapshots are easy-peasy too:\n\n```{r eval=FALSE}\nrender_png(url = \"http://www.marveldirectory.com/individuals/c/captainamerica.htm\")\n```\n\n```{r eval=TRUE, include=FALSE}\nrender_png(url = \"http://www.marveldirectory.com/individuals/c/captainamerica.htm\") %\u003e% \n  image_write(\"img/cap.png\")\n```\n\n![](img/cap.png)\n\n```{r eval=FALSE}\nrender_jpeg(url = \"http://static2.comicvine.com/uploads/scale_small/3/31666/5052983-capasr2016001-eptingvar-18bdb.jpg\") \n```\n\n```{r eval=TRUE, include=FALSE}\nrender_jpeg(url = \"http://static2.comicvine.com/uploads/scale_small/3/31666/5052983-capasr2016001-eptingvar-18bdb.jpg\") %\u003e% \n  image_write(\"img/cap.jpg\")\n```\n\n![](img/cap.jpg)\n\n### Executing custom Lua scripts\n\n```{r}\nlua_ex \u003c- '\nfunction main(splash)\n  splash:go(\"http://rud.is/b\")\n  splash:wait(0.5)\n  local title = splash:evaljs(\"document.title\")\n  return {title=title}\nend\n'\n\nsplash_local %\u003e% execute_lua(lua_ex) -\u003e res\n\nrawToChar(res) %\u003e% \n  jsonlite::fromJSON()\n```\n\n### Interacting With Flash sites\n\n```{r eval=FALSE}\nsplash_local %\u003e%\n  splash_plugins(TRUE) %\u003e%\n  splash_go(\"https://gis.cdc.gov/GRASP/Fluview/FluHospRates.html\") %\u003e%\n  splash_wait(4) %\u003e%\n  splash_click(460, 550) %\u003e%\n  splash_wait(2) %\u003e%\n  splash_click(230, 85) %\u003e%\n  splash_wait(2) %\u003e%\n  splash_png()\n```\n\n\u003cimg src=\"img/flash.png\" width=\"50%\"/\u003e\n\n```{r eval=FALSE}\nstop_splash(splash_vm)\n```\n\n### Code of Conduct\n\nPlease note that this project is released with a [Contributor Code of Conduct](CONDUCT.md). By participating in this project you agree to abide by its terms.","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fhrbrmstr%2Fsplashr","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fhrbrmstr%2Fsplashr","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fhrbrmstr%2Fsplashr/lists"}