{"id":13857830,"url":"https://github.com/sckott/webmiddens","last_synced_at":"2025-08-28T23:39:00.252Z","repository":{"id":140975036,"uuid":"174437292","full_name":"sckott/webmiddens","owner":"sckott","description":"cache http requests","archived":false,"fork":false,"pushed_at":"2020-12-10T22:22:20.000Z","size":135,"stargazers_count":9,"open_issues_count":11,"forks_count":1,"subscribers_count":3,"default_branch":"master","last_synced_at":"2025-08-23T08:12:50.382Z","etag":null,"topics":["api","caching","fakeweb","http","http-cache","http-mocking","r","rstats","web"],"latest_commit_sha":null,"homepage":"https://sckott.github.io/webmiddens","language":"R","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"other","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/sckott.png","metadata":{"files":{"readme":"README.Rmd","changelog":"NEWS.md","contributing":".github/CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":"CODE_OF_CONDUCT.md","threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2019-03-07T23:41:54.000Z","updated_at":"2025-03-22T08:14:17.000Z","dependencies_parsed_at":"2023-06-29T06:01:07.007Z","dependency_job_id":null,"html_url":"https://github.com/sckott/webmiddens","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/sckott/webmiddens","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sckott%2Fwebmiddens","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sckott%2Fwebmiddens/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sckott%2Fwebmiddens/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sckott%2Fwebmiddens/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/sckott","download_url":"https://codeload.github.com/sckott/webmiddens/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sckott%2Fwebmiddens/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":272580866,"owners_count":24959402,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-08-28T02:00:10.768Z","response_time":74,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["api","caching","fakeweb","http","http-cache","http-mocking","r","rstats","web"],"created_at":"2024-08-05T03:01:48.242Z","updated_at":"2025-08-28T23:39:00.216Z","avatar_url":"https://github.com/sckott.png","language":"R","readme":"webmiddens\n==========\n\n```{r echo=FALSE}\nknitr::opts_chunk$set(\n  comment = \"#\u003e\",\n  collapse = TRUE,\n  warning = FALSE\n)\n```\n\n[![Project Status: Active – The project has reached a stable, usable state and is being actively developed.](https://www.repostatus.org/badges/latest/active.svg)](https://www.repostatus.org/#active)\n[![R-CMD-check](https://github.com/sckott/webmiddens/workflows/R-CMD-check/badge.svg)](https://github.com/sckott/webmiddens/actions?query=workflow%3AR-CMD-check)\n[![codecov](https://codecov.io/gh/sckott/webmiddens/branch/master/graph/badge.svg)](https://codecov.io/gh/sckott/webmiddens)\n\nsimple caching of HTTP requests/responses, hooking into webmockr (https://github.com/ropensci/webmockr)\nfor the HTTP request matching\n\nA midden is a debris pile constructed by a woodrat/pack rat (https://en.wikipedia.org/wiki/Pack_rat#Midden)\n\n### the need\n\n- `vcr` is meant really for testing, or script use. i don't think it fits\nwell into a use case where another pkg wants to cache responses\n- `memoise` seems close-ish but doesn't fit needs, e.g., no expiry, not specific\nto HTTP requests, etc.\n- we need something specific to HTTP requests, that allows expiration handling, a few different caching location options, works across HTTP clients, etc\n- caching just the http responses means the rest of the code in the function can change, and the response can still be cached\n    - the downside, vs. memoise, is that we're only caching the http response, so if there's still a lot of time spent processing the response, then the function will still be quite slow - BUT, if the HTTP response processing code is within a function, you could memoise that function\n- memoise is great, but since it caches the whole function call, you don't benefit from individually caching each http request, which we do here. if you cache each http request, then any time you do that same http request, it's response is already cached\n\n### brainstorming\n\n- use `webmockr` to match requests (works with `crul`; soon `httr`)\n- possibly match on, and expire based on headers: Cache-Control, Age, Last-Modified,\nETag, Expires (see Ruby's faraday-http-cache (https://github.com/plataformatec/faraday-http-cache#what-gets-cached))\n- caching backends: probably all binary to save disk space since most likely\nwe don't need users to be able to look at plain text of caches\n- expiration: set a time to expire. if set to `2019-03-08 00:00:00` and it's\n`2019-03-07 23:00:00`, then 1 hr from now the cache will expire, and a new real HTTP\nrequest will need to be made (i.e., the cache will be deleted whenever the next\nHTTP request is made)\n\n### http libraries\n\nright now we only support `crul`, but `httr` support should arrive soon\n\n### installation\n\n```{r eval=FALSE}\nremotes::install_github(\"sckott/webmiddens\")\n```\n\n### use_midden()\n\n```{r}\nlibrary(webmiddens)\nlibrary(crul)\n```\n\nLet's say you have some function `http_request()` that does an HTTP request that\nyou re-use in various parts of your project or package\n\n```{r}\nhttp_request \u003c- function(...) {\n  x \u003c- crul::HttpClient$new(\"https://httpbin.org\", opts = list(...))\n  x$get(\"get\")\n}\n```\n\nAnd you have a function `some_fxn()` that uses `http_request()` to do the HTTP \nrequest, then proces the results to a data.frame or list, etc. This is a super\ncommon pattern in a project or R package that deals with web resources.\n\n```{r}\nsome_fxn \u003c- function(...) {\n  res \u003c- http_request(...)\n  jsonlite::fromJSON(res$parse(\"UTF-8\"))\n}\n```\n\nWithout `webmiddens` the HTTP request happens as usual and all is good\n\n```{r}\nsome_fxn()\n```\n\nNow, with `webmiddens`\n\nrun `wm_configuration()` first to set the path where HTTP requests will be cached\n\n```{r}\nwm_configuration(\"foo1\")\n```\n\n```{r echo=FALSE}\nbb \u003c- midden_current()\nbb$destroy()\n```\n\nfirst request is a real HTTP request\n\n```{r}\nres1 \u003c- use_midden(some_fxn())\nres1\n```\n\nsecond request uses the cached response from the first request\n\n```{r}\nres2 \u003c- use_midden(some_fxn())\nres2\n```\n\n### the midden class\n\n```{r}\nx \u003c- midden$new()\nx # no path\n# Run $init() to set the path\nx$init(path = \"forest\")\nx\n```\n\nThe `cache` slot has a `hoardr` object which you can use to fiddle with\nfiles, see `?hoardr::hoard`\n\n```{r}\nx$cache\n```\n\nUse `expire()` to set the expire time (in seconds). You can set it through\npassing to `expire()` or through the environment variable `WEBMIDDENS_EXPIRY_SEC`\n\n```{r}\nx$expire()\nx$expire(5)\nx$expire()\nx$expire(reset = TRUE)\nx$expire()\nSys.setenv(WEBMIDDENS_EXPIRY_SEC = 35)\nx$expire()\nx$expire(reset = TRUE)\nx$expire()\n```\n\nFIXME: The below not working right now - figure out why\n\n```{r eval=FALSE}\nwm_enable()\ncon \u003c- crul::HttpClient$new(\"https://httpbin.org\")\n# first request is a real HTTP request\nx$r(con$get(\"get\", query = list(stuff = \"bananas\")))\n# following requests use the cached response\nx$r(con$get(\"get\", query = list(stuff = \"bananas\")))\n```\n\nverbose output\n\n```{r eval=FALSE}\nx \u003c- midden$new(verbose = TRUE)\nx$init(path = \"rainforest\")\nx$r(con$get(\"get\", query = list(stuff = \"bananas\")))\n```\n\nset expiration time\n\n```{r eval=FALSE}\nx \u003c- midden$new()\nx$init(path = \"grass\")\nx$expire(3)\nx\n```\n\nDelete all the files in your \"midden\" (the folder with cached files)\n\n```{r eval=FALSE}\nx$cleanup()\n```\n\nDelete the \"midden\" (the folder with cached files)\n\n```{r eval=FALSE}\nx$destroy()\n```\n\n## Meta\n\n* Please [report any issues or bugs](https://github.com/sckott/webmiddens/issues).\n* License: MIT\n* Get citation information for `webmiddens` in R doing `citation(package = 'webmiddens')`\n* Please note that this project is released with a [Contributor Code of Conduct][coc].\nBy participating in this project you agree to abide by its terms.\n\n[coc]: https://github.com/sckott/webmiddens/blob/master/CODE_OF_CONDUCT.md\n","funding_links":[],"categories":["R"],"sub_categories":[],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsckott%2Fwebmiddens","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fsckott%2Fwebmiddens","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsckott%2Fwebmiddens/lists"}