{"id":13858223,"url":"https://github.com/hrbrmstr/jwatr","last_synced_at":"2025-10-29T01:30:56.501Z","repository":{"id":141238419,"uuid":"100738452","full_name":"hrbrmstr/jwatr","owner":"hrbrmstr","description":":card_index: Tools to Query and Create Web Archive Files Using the Java Web Archive Toolkit in R","archived":false,"fork":false,"pushed_at":"2017-09-04T03:44:06.000Z","size":40439,"stargazers_count":7,"open_issues_count":2,"forks_count":1,"subscribers_count":4,"default_branch":"master","last_synced_at":"2024-08-06T03:05:01.856Z","etag":null,"topics":["java","r","r-cyber","rstats","warc"],"latest_commit_sha":null,"homepage":"","language":"R","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/hrbrmstr.png","metadata":{"files":{"readme":"README.Rmd","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null}},"created_at":"2017-08-18T18:10:09.000Z","updated_at":"2024-05-18T09:51:58.000Z","dependencies_parsed_at":"2024-02-09T02:20:25.393Z","dependency_job_id":null,"html_url":"https://github.com/hrbrmstr/jwatr","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/hrbrmstr%2Fjwatr","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/hrbrmstr%2Fjwatr/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/hrbrmstr%2Fjwatr/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/hrbrmstr%2Fjwatr/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/hrbrmstr","download_url":"https://codeload.github.com/hrbrmstr/jwatr/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":219858513,"owners_count":16556043,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["java","r","r-cyber","rstats","warc"],"created_at":"2024-08-05T03:02:00.958Z","updated_at":"2025-10-29T01:30:49.125Z","avatar_url":"https://github.com/hrbrmstr.png","language":"R","funding_links":[],"categories":["R"],"sub_categories":[],"readme":"---\noutput: rmarkdown::github_document\n---\n\n`jwatr` : Tools to Query and Create Web Archive Files Using the Java Web Archive Toolkit\n\nThe Java Web Archive Toolkit ('JWAT') \u003chttps://sbforge.org/display/JWAT/Overview\u003e\nis a library of Java objects and methods which enables reading, writing and validating web archive files.\n\nWIP!!! Reading \u0026 writing need some optimization and edge case checking. There's also a chance I'll change the name to `warc` but some folks are using that package now and I dinna want to cause pain there yet.\n\nThe following functions are implemented:\n\n**Reading**\n\n- `read_warc`:\tRead a WARC file (compressed or uncompressed)\n- `warc_stream_in`:\tStream in records from a WARC file\n\n**Writing**\n\n- `warc_file`:\tCreate a new WARC file\n- `warc_write_warcinfo`:\tWrite a 'warcinfo' record to a WARC File\n- `warc_write_response`:\tWrite simple `httr::GET` requests or full `httr` `response` objects to a WARC file\n- `close_warc_file`:\tClose a WARC file\n\n**`httr` Wrappers**\n\n- `warc_GET`:\tWARC-ify an httr::GET request\n- `warc_POST`:\tWARC-ify an httr::GET request\n\n**Utility**\n\n- `response_list_to_warc_file`:\tTurns a list of 'httr' 'response' objects into a WARC file\n- `payload_content`:\tHelper function to convert WARC raw headers+payload into something useful\n- `is_compressed`:\tTest if a raw vector is gzip compressed\n\nNOTE: To read in typical (~800MB-1GB gzip'd WARC files) you should consider doing the following (in order) in your scripts:\n\n```{r eval=FALSE}\noptions(java.parameters = \"-Xmx2g\")\n\nlibrary(rJava)\nlibrary(jwatjars)\nlibrary(jwatr)\n```\n\nThat idiom generally provides enough heap space, but you may need to adjust the heap size if you've got larger payloads.\n\nAlternatively, you can set the same option in your R startup scripts, but that will likely come back to bite you when moving workloads around.\n\n### Installation\n\n```{r eval=FALSE}\ndevtools::install_github(\"hrbrmstr/jwatr\")\n```\n\n```{r message=FALSE, warning=FALSE, error=FALSE, include=FALSE}\noptions(width=120)\n```\n\n### Reading WARC Files\n\n```{r message=FALSE, warning=FALSE, error=FALSE}\nlibrary(rJava)\nlibrary(jwatr)\nlibrary(magick)\nlibrary(tidyverse)\n\n# current verison\npackageVersion(\"jwatr\")\n```\n\n```{r message=FALSE, warning=FALSE, error=FALSE}\n# small, uncompressed WARC file\nglimpse(read_warc(system.file(\"extdata/bbc.warc\", package=\"jwatr\")))\n\n# larger example\nxdf \u003c- read_warc(system.file(\"extdata/sample.warc.gz\", package=\"jwatr\"),\n                 warc_types = \"response\", include_payload = TRUE)\n\nglimpse(xdf)\n\n# get the payload content\npayload_content(url = xdf$target_uri[279], ctype = xdf$http_protocol_content_type[279], \n                xdf$http_raw_headers[[279]], xdf$payload[[279]])\n\n# or ingest the raw bits yourself\nimgs \u003c- filter(xdf, grepl(\"(png|gif|jpeg)$\", http_protocol_content_type))\n\nimgs\n\nimage_read(imgs$payload[[1]])\n```\n\n![](imgs/img1.jpeg)\n\n### Writing WARC Files\n\n```{r message=FALSE, warning=FALSE, error=FALSE}\nlibrary(jwatr)\nlibrary(httr)\nlibrary(magick)\nlibrary(tidyverse)\n\ntf \u003c- tempfile(\"test\")\nwf \u003c- warc_file(tf)\n\nwarc_write_response(wf, \"https://rud.is/b/\")\n\n# store a simple httr::GET request\nwarc_write_response(wf, GET(\"https://rud.is/b/\"))\n\nwarc_write_response(wf, \"https://www.rstudio.com/\")\nwarc_write_response(wf, \"https://www.r-project.org/\")\n\n# all valid content types work, like this PDF\nwarc_write_response(wf, \"http://che.org.il/wp-content/uploads/2016/12/pdf-sample.pdf\")\n\n# complex API calls can be made and the results stored in the WARC file as well\n# this API call returns a JSON object\nPOST(\n url = \"https://data.police.uk/api/crimes-street/all-crime\",\n query = list( lat = \"52.629729\", lng = \"-1.131592\", date = \"2017-01\")\n) -\u003e uk_res\n\nwarc_write_response(wf, uk_res)\nwarc_write_response(wf, \"https://journal.r-project.org/RLogo.png\")\n\nclose_warc_file(wf)\n\nxdf \u003c- read_warc(sprintf(\"%s.warc.gz\", tf), include_payload = TRUE)\n\nglimpse(xdf)\n\n# decode the WARC stored JSON response from the UK Crimes API\nglimpse(jsonlite::fromJSON(rawToChar(xdf[6,]$payload[[1]]), flatten=TRUE))\n\nselect(xdf, content_length, http_protocol_content_type)\n\nimage_read(xdf$payload[[5]])\n```\n\n![](imgs/img2.png)\n\n```{r echo=FALSE}\nunlink(tf)\n```\n\n### Streaming\n\nThe `warc_stream_in()` function provides a pure-R method for stream processing WARC\nfiles through the use of an R callback handler. One way of using this is to build a\ndata frame. The following example builds a data frame of WARC `response` records. Space\nis reserved for a 10,000-element list which will get truncated or expanded as necessary:\n\n```{r message=FALSE, warning=FALSE, error=FALSE}\nxdf \u003c- list(10000)\nxdf_i \u003c- 0\n\nmyfun \u003c- function(headers, payload, ...) {\n  headers \u003c- setNames(headers, gsub(\"-\", \"_\", names(headers)))\n  xdf_i \u003c\u003c- xdf_i + 1\n  headers$payload \u003c- list(payload)\n  xdf[xdf_i] \u003c\u003c- list(headers)\n}\n\n(n \u003c- warc_stream_in(\n  system.file(\"extdata/sample.warc.gz\", package=\"jwatr\"),\n  myfun,\n  warc_types = \"response\"\n))\n\nxdf \u003c- bind_rows(xdf)\n\nglimpse(xdf)\n\ncount(xdf, content_type)\n\ncat(rawToChar(xdf$payload[[1]]))\n```\n\n### Test Results\n\n```{r message=FALSE, warning=FALSE, error=FALSE}\nlibrary(jwatr)\nlibrary(testthat)\n\ndate()\n\ntest_dir(\"tests/\")\n```\n\n### Code of Conduct\n\nPlease note that this project is released with a [Contributor Code of Conduct](CONDUCT.md). By participating in this project you agree to abide by its terms.","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fhrbrmstr%2Fjwatr","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fhrbrmstr%2Fjwatr","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fhrbrmstr%2Fjwatr/lists"}