{"id":20685914,"url":"https://github.com/erykwalczak/prarulebook","last_synced_at":"2025-04-22T13:41:53.294Z","repository":{"id":89373156,"uuid":"192198090","full_name":"erykwalczak/PRArulebook","owner":"erykwalczak","description":"Scrape the Prudential Regulation Authority (PRA) rulebook","archived":false,"fork":false,"pushed_at":"2021-11-21T17:12:30.000Z","size":2316,"stargazers_count":3,"open_issues_count":0,"forks_count":3,"subscribers_count":1,"default_branch":"master","last_synced_at":"2025-03-29T15:34:56.587Z","etag":null,"topics":["r","regtech","regulation","scraper","suptech"],"latest_commit_sha":null,"homepage":"https://www.bankofengland.co.uk/working-paper/2019/the-language-of-rules-textual-complexity-in-banking-reforms","language":"R","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"gpl-3.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/erykwalczak.png","metadata":{"files":{"readme":"README.Rmd","changelog":null,"contributing":null,"funding":null,"license":"LICENSE.md","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2019-06-16T14:04:43.000Z","updated_at":"2023-12-28T02:08:49.000Z","dependencies_parsed_at":null,"dependency_job_id":"23dca7c8-0eac-4b12-a799-26535db1e8bc","html_url":"https://github.com/erykwalczak/PRArulebook","commit_stats":null,"previous_names":["erykwalczak/prarulebook","erzk/prarulebook"],"tags_count":1,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/erykwalczak%2FPRArulebook","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/erykwalczak%2FPRArulebook/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/erykwalczak%2FPRArulebook/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/erykwalczak%2FPRArulebook/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/erykwalczak","download_url":"https://codeload.github.com/erykwalczak/PRArulebook/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":250250528,"owners_count":21399655,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["r","regtech","regulation","scraper","suptech"],"created_at":"2024-11-16T22:29:25.682Z","updated_at":"2025-04-22T13:41:53.269Z","avatar_url":"https://github.com/erykwalczak.png","language":"R","funding_links":[],"categories":[],"sub_categories":[],"readme":"---\noutput: github_document\n---\n\n```{r, include = FALSE}\nknitr::opts_chunk$set(\n  collapse = TRUE,\n  comment = \"#\u003e\",\n  fig.path = \"man/figures/README-\",\n  out.width = \"100%\"\n)\n```\n# PRArulebook\n\n\u003c!-- badges: start --\u003e\n[![Project Status: Active – The project has reached a stable, usable state and is being actively developed.](http://www.repostatus.org/badges/latest/active.svg)](http://www.repostatus.org/#active)\n[![Travis-CI Build Status](https://travis-ci.org/erzk/PRArulebook.svg?branch=master)](https://travis-ci.org/erzk/PRArulebook)\n[![Lifecycle: maturing](https://img.shields.io/badge/lifecycle-maturing-blue.svg)](https://www.tidyverse.org/lifecycle/#maturing)\n[![CRAN status](https://www.r-pkg.org/badges/version/PRArulebook)](https://cran.r-project.org/package=PRArulebook)\n[![Codecov test coverage](https://codecov.io/gh/erzk/PRArulebook/branch/master/graph/badge.svg)](https://codecov.io/gh/erzk/PRArulebook?branch=master)\n[![DOI](https://zenodo.org/badge/192198090.svg)](https://zenodo.org/badge/latestdoi/192198090)\n\u003c!-- badges: end --\u003e\n\n`PRArulebook` is a package to scrape the PRA (Prudential Regulation Authority) [Rulebook](http://www.prarulebook.co.uk/) (the website containing the rules made and enforced by the PRA under powers conferred by the Financial Services and Markets Act 2000 (FSMA)).\n\nThe input to this package is the PRA Rulebook website. Outputs from this package are the rules published on the PRA Rulebook website in a format more amenable to text and network analysis.\n\n`PRArulebook` was developed while preparing:\n\n**Amadxarif, Z., Brookes, J., Garbarino, N., Patel, R., Walczak, E. (2019) *[The Language of Rules: Textual Complexity in Banking Reforms.](https://www.bankofengland.co.uk/working-paper/2019/the-language-of-rules-textual-complexity-in-banking-reforms)* Staff Working Paper No. 834. Bank of England.**\n\nAny use of this package with the PRA Rulebook must comply with the PRA Rulebook's [Terms of Use](http://www.prarulebook.co.uk/terms-of-use). These include, but are not limited to, restrictions on using content from the PRA Rulebook for commercial purposes without obtaining a licence from the PRA.\n\n## Installation\n\nYou can install the development version of `PRArulebook` from GitHub with:\n\n```{r, eval = FALSE}\ninstall.packages(\"devtools\")\ndevtools::install_github(\"erzk/PRArulebook\")\n```\n\n## Data\n\n`PRArulebook` scrapes two types of data: **structure** and **content**.\n\n* **Structure** - hierarchy of the PRA Rulebook. Includes URLs and names.\n\n* **Content**\n  * Text - can be used for text analysis\n  * Network - can be used for network analysis.\n\nThe next section shows how to extract these types of data.\n\n## Examples\n\nLoad the package\n\n```{r}\nlibrary(PRArulebook)\n```\n\n### Structure\n\nThe simplest way to extract a rulebook structure is to use `get_structure` function\n\n```{r, eval = FALSE}\n# get the structure of the rulebook down to the part-level\nparts \u003c-\n  get_structure(\"16-11-2007\",\n                layer = \"part\")\n# or chapter-level\n# warnings (410) are displayed for inactive sites\nchapters \u003c-\n  get_structure(\"18-06-2019\",\n                layer = \"chapter\")\n```\n\nThis will start scraping the PRA rulebook. Warnings ([410](https://en.wikipedia.org/wiki/List_of_HTTP_status_codes) code) will be displayed when a page is no longer active. Pulling data will take longer if you decide to pull more granular data. The rulebook has several layers and each of them can be passed to the `layer` argument of `get_structure` (in descending order):\n\n* `sector`\n* `part`\n* `chapter`\n\nThe output will be a data frame with information about the structure (i.e. URLs and names).\n\nScraping individual rules is much slower so another function should be used\n\n```{r, eval = FALSE}\n# extract all rules from the first three chapters\nrules \u003c- scrape_rule_structure(chapters[1:3,], \"18-06-2019\")\n```\n\n### Content\n\nOnce the structure URLs are scraped, they can be used to extract content. \n\n#### Text\n\nTo get content of the rulebook (text or links) use `get_content` function with a URL of a given chapter. \n\n```{r, eval = FALSE}\n# scrape text from a single chapter\nchapter_text \u003c- get_content(chapters$chapter_url[1])\n# or single rule\nrule_text \u003c- get_content(rules$rule_url[3], \"text\", \"yes\")\n```\n\nThis function can be applied on the entire rulebook in the following way:\n\n```{r, eval = FALSE}\nlibrary(purrr)\n\n# text: chapter-level\n# exception handling might be needed\nchapters_text \u003c-\n  map_df(chapters$chapter_url[1:5],\n                get_content)\n\n# text\nrules_text \u003c-\n  map_df(rules$rule_url[1:5],\n         get_content,\n         type = \"text\",\n         single_rule_selector = \"yes\")\n\n# links\nrules_links \u003c-\n  map_df(rules$rule_url[1:5],\n         get_content,\n         type = \"links\",\n         single_rule_selector = \"yes\")\n```\n\nThe output can be then joined to the information about the rulebook structure and aggregated at a higher level.\n\n#### Network\n\nTo scrape the links and create data set for network analysis `get_content` function can be used but with a `type` argument set to `\"links\"`. Like in the previous example, this call can also be parallelised.\n\n```{r, eval = FALSE}\nchapter_link \u003c- get_content(chapters$chapter_url[1], \"links\")\n\n# sequential\nparts_links \u003c-\n  purrr::map_df(parts$part_url[1:5],\n                get_content,\n                \"links\")\n```\n\nThe code above will return a data frame with *from/to url*, *text* used in a link, and a *type* of a link.\n\nScraped data containing information about the links can be used for network analysis (warning: further cleaning might be required).\n\n### Disclaimer \n\nThis package is an outcome of a research project. All errors are mine. All views expressed are personal views, not those of any employer.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ferykwalczak%2Fprarulebook","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Ferykwalczak%2Fprarulebook","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ferykwalczak%2Fprarulebook/lists"}