{"id":13706701,"url":"https://github.com/hrbrmstr/spiderbar","last_synced_at":"2025-10-17T21:11:54.817Z","repository":{"id":51342396,"uuid":"100300264","full_name":"hrbrmstr/spiderbar","owner":"hrbrmstr","description":"Lightweight R wrapper around rep-cpp for robot.txt (Robots Exclusion Protocol) parsing and path testing in R","archived":false,"fork":false,"pushed_at":"2024-07-26T17:37:09.000Z","size":91,"stargazers_count":10,"open_issues_count":1,"forks_count":2,"subscribers_count":4,"default_branch":"master","last_synced_at":"2025-02-27T13:18:38.613Z","etag":null,"topics":["r","r-cyber","robots-exclusion-protocol","robots-txt","rstats"],"latest_commit_sha":null,"homepage":null,"language":"C++","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"other","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/hrbrmstr.png","metadata":{"files":{"readme":"README.Rmd","changelog":"NEWS.md","contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2017-08-14T19:01:10.000Z","updated_at":"2023-02-07T14:07:35.000Z","dependencies_parsed_at":"2024-10-27T11:14:24.331Z","dependency_job_id":"b981680c-a1f4-4e51-a865-fe5c23781c8a","html_url":"https://github.com/hrbrmstr/spiderbar","commit_stats":{"total_commits":21,"total_committers":3,"mean_commits":7.0,"dds":0.1428571428571429,"last_synced_commit":"66a2723990b6e79cabe19ae608f3fcb0837e3efb"},"previous_names":[],"tags_count":1,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/hrbrmstr%2Fspiderbar","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/hrbrmstr%2Fspiderbar/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/hrbrmstr%2Fspiderbar/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/hrbrmstr%2Fspiderbar/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/hrbrmstr","download_url":"https://codeload.github.com/hrbrmstr/spiderbar/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":243830912,"owners_count":20354848,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["r","r-cyber","robots-exclusion-protocol","robots-txt","rstats"],"created_at":"2024-08-02T22:01:05.820Z","updated_at":"2025-10-17T21:11:49.752Z","avatar_url":"https://github.com/hrbrmstr.png","language":"C++","funding_links":[],"categories":["C++"],"sub_categories":[],"readme":"---\noutput: \n  rmarkdown::github_document:\n    df_print: kable\neditor_options: \n  chunk_output_type: console\n---\n```{r pkg-knitr-opts, include=FALSE}\nhrbrpkghelpr::global_opts()\n```\n\n```{r badges, results='asis', echo=FALSE, cache=FALSE}\nhrbrpkghelpr::stinking_badges()\n```\n\n```{r description, results='asis', echo=FALSE, cache=FALSE}\nhrbrpkghelpr::yank_title_and_description()\n```\n\n- [`rep-cpp`](https://github.com/seomoz/rep-cpp)\n- [`url-cpp`](https://github.com/seomoz/url-cpp)\n\n## What's Inside the Tin\n\nThe following functions are implemented:\n\n```{r ingredients, results='asis', echo=FALSE, cache=FALSE}\nhrbrpkghelpr::describe_ingredients()\n```\n\n## Installation\n\n```{r install-ex, results='asis', echo=FALSE, cache=FALSE}\nhrbrpkghelpr::install_block()\n```\n\n## Usage\n\n```{r message=FALSE, warning=FALSE, error=FALSE}\nlibrary(spiderbar)\nlibrary(robotstxt)\n\n# current verison\npackageVersion(\"spiderbar\")\n\n# use helpers from the robotstxt package\n\nrt \u003c- robxp(get_robotstxt(\"https://cdc.gov\"))\n\nprint(rt)\n\n# or \n\nrt \u003c- robxp(url(\"https://cdc.gov/robots.txt\"))\n\ncan_fetch(rt, \"/asthma/asthma_stats/default.htm\", \"*\")\n\ncan_fetch(rt, \"/_borders\", \"*\")\n\ngh_rt \u003c- robxp(robotstxt::get_robotstxt(\"github.com\"))\n\ncan_fetch(gh_rt, \"/humans.txt\", \"*\") # TRUE\n\ncan_fetch(gh_rt, \"/login\", \"*\") # FALSE\n\ncan_fetch(gh_rt, \"/oembed\", \"CCBot\") # FALSE\n\ncan_fetch(gh_rt, c(\"/humans.txt\", \"/login\", \"/oembed\"))\n\ncrawl_delays(gh_rt)\n\nimdb_rt \u003c- robxp(robotstxt::get_robotstxt(\"imdb.com\"))\n\ncrawl_delays(imdb_rt)\n\nsitemaps(imdb_rt)\n```\n\n## spiderbar Metrics\n\n```{r cloc, echo=FALSE}\ncloc::cloc_pkg_md()\n```\n\n## Code of Conduct\n\nPlease note that this project is released with a Contributor Code of Conduct. By participating in this project you agree to abide by its terms.","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fhrbrmstr%2Fspiderbar","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fhrbrmstr%2Fspiderbar","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fhrbrmstr%2Fspiderbar/lists"}