{"id":20685917,"url":"https://github.com/erykwalczak/ebarulebook","last_synced_at":"2026-05-20T16:40:12.657Z","repository":{"id":89373126,"uuid":"192205958","full_name":"erykwalczak/EBArulebook","owner":"erykwalczak","description":"Scrape the European Banking Authority (EBA) rulebook and Q\u0026As","archived":false,"fork":false,"pushed_at":"2021-03-06T13:10:43.000Z","size":2680,"stargazers_count":1,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"master","last_synced_at":"2025-01-17T16:21:29.412Z","etag":null,"topics":["banking","eba","r","regtech","regulations","rulebook","scraper","suptech"],"latest_commit_sha":null,"homepage":"","language":"HTML","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"gpl-3.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/erykwalczak.png","metadata":{"files":{"readme":"README.Rmd","changelog":null,"contributing":null,"funding":null,"license":"LICENSE.md","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2019-06-16T15:17:43.000Z","updated_at":"2023-12-28T03:07:06.000Z","dependencies_parsed_at":null,"dependency_job_id":"316cdf53-0e72-4947-90f2-0f1442099433","html_url":"https://github.com/erykwalczak/EBArulebook","commit_stats":null,"previous_names":["erykwalczak/ebarulebook"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/erykwalczak%2FEBArulebook","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/erykwalczak%2FEBArulebook/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/erykwalczak%2FEBArulebook/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/erykwalczak%2FEBArulebook/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/erykwalczak","download_url":"https://codeload.github.com/erykwalczak/EBArulebook/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":242940017,"owners_count":20209882,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["banking","eba","r","regtech","regulations","rulebook","scraper","suptech"],"created_at":"2024-11-16T22:29:27.155Z","updated_at":"2026-05-20T16:40:07.634Z","avatar_url":"https://github.com/erykwalczak.png","language":"HTML","funding_links":[],"categories":[],"sub_categories":[],"readme":"---\noutput: github_document\n---\n\n# EBArulebook\n\n\u003c!-- badges: start --\u003e\n[![Lifecycle: experimental](https://img.shields.io/badge/lifecycle-experimental-orange.svg)](https://lifecycle.r-lib.org/articles/stages.html#experimental)\n[![CRAN status](https://www.r-pkg.org/badges/version/EBArulebook)](https://cran.r-project.org/package=EBArulebook)\n\n\u003c!-- badges: end --\u003e\n\n`EBArulebook` is a package to scrape the [EBA Single Rulebook](https://www.eba.europa.eu/regulation-and-policy/single-rulebook/).\n\nThe input to this package is the Single Rulebook website. Outputs from this package are the rules and [Q\u0026As](https://www.eba.europa.eu/single-rule-book-qa) published on the Single Rulebook website in a format more amenable to text and network analysis.\n\nThe goal of `EBArulebook` is to scrape the [Single Rulebook](https://eba.europa.eu/regulation-and-policy/single-rulebook) and [Q\u0026As](https://eba.europa.eu/single-rule-book-qa) published by the European Banking Authority (EBA). This package was developed while working on a Staff Working Paper:\n\n**Amadxarif, Z., Brookes, J., Garbarino, N., Patel, R., Walczak, E. (2019) *[The Language of Rules: Textual Complexity in Banking Reforms.](https://www.bankofengland.co.uk/working-paper/2019/the-language-of-rules-textual-complexity-in-banking-reforms)* Staff Working Paper No. 834. Bank of England.**\n\nIf you use this package, then please cite the paper.\n\nAny use of this package with the Single Rulebook must comply with the Single Rulebook's [Terms of Use](https://www.eba.europa.eu/legal-notice).\n\n## Installation\n\nYou can install the development version of EBArulebook from [GitHub](https://github.com/) with:\n\n```{r, eval = FALSE}\n# install.packages(\"devtools\")\ndevtools::install_github(\"erzk/EBArulebook\")\n```\n## Examples\n\nThis packages makes it easy to obtain EBA data in a tabular form\n\n```{r example, eval = FALSE, message = FALSE}\nlibrary(EBArulebook)\n```\n\nTo get **EBA single rulebook** (tested only on Capital Requirements Regulation (*CRR*))\n\n```{r, eval = FALSE}\nall_eba_rules \u003c- scrape_EBA()\n\ndplyr::glimpse(all_eba_rules)\n```\n\nTo get **EBA single rulebook Q\u0026As**\n\n```{r, eval = FALSE}\nqa_df \u003c- scrape_EBA_QA()\n\ndplyr::glimpse(qa_df)\n```\n\n\n## European Banking Authority - The Single Rulebook\n\nCode in this repository was used to acquire and analyse data used in the forthcoming Staff Working Paper.\n\nCheck the [vignettes](https://github.com/erzk/EBArulebook/tree/master/vignettes) for more details and examples.\n\n### Get the full website\n\n* Install [phantomjs](https://github.com/ariya/phantomjs)\n\n* Write a simple [scraper](https://www.thedataschool.co.uk/brian-scally/web-scraping-javascript-content/) `scrape_EBA.js`:\n\nThis scraper was developed on Ubuntu 16.04. It was tested only on the Capital Requirements Regulation (CRR) but it should also work on other parts of the EBA rulebook.\n\nThe EBA rulebook is displayed dynamically so the first step is to scrape the text using a headless browser (phantomjs).\n\n```{r, eval = FALSE}\nvar url = 'https://eba.europa.eu/regulation-and-policy/single-rulebook/interactive-single-rulebook/-/interactive-single-rulebook/toc/504'; \n\nvar page = new WebPage(); \nvar fs = require('fs'); \n\npage.open(url, function (status) { \njust_wait(); }); \n\nfunction just_wait() { \nsetTimeout(function() { \nfs.write('website_phantom.html', \npage.content, 'w'); \nphantom.exit(); }, 2500); \n}\n```\n\n* Run the scraper from the command line with\n\n```{bash, eval = FALSE}\nphantomjs scrape_EBA.js\n```\n\nThis downloads the entire page which then needs to be cleaned.\n\n### Parse downloaded html pages\n\n* Analyse the output `website_phantom.html`\n\n* Use `rvest` to extract key data\n\nThe rulebook is using html to display the rules and is constructed in a simple way:\n\nhttps://eba.europa.eu/regulation-and-policy/single-rulebook/interactive-single-rulebook/-/interactive-single-rulebook/article-id/2002\n\nwhere the last number after '/' is an ID.\n\nTo solve this problem I extracted the relevant IDs by running regex on the scraped html file. Look for the IDs: 'article-id/[DIGIT]'. See `parse_EBA_page()` for details.\n\n## Q\u0026As\n\n### Scraper\n\nSearch for questions. There are 1757 Q\u0026As (1652 Final and 105 Rejected) as of 8 June 2019. Status: use 'All': both 'Final' and 'Rejected'.\n\nMaximum displayed Q\u0026As per page is 200 so 9 pages in total (see the final part the URL: 'cur=2'). \n\nUse `scrape_EBA_QA.js` from the command line \n\n```{bash, eval = FALSE}\nphantomjs scrape_EBA.js\n```\n\nEdit the .js file updating pages to scrape:\n\n* Page 1\n\nhttps://eba.europa.eu/single-rule-book-qa?p_p_id=questions_and_answers_WAR_questions_and_answersportlet\u0026p_p_lifecycle=0\u0026p_p_state=normal\u0026p_p_mode=view\u0026p_p_col_id=column-1\u0026p_p_col_pos=1\u0026p_p_col_count=2\u0026_questions_and_answers_WAR_questions_and_answersportlet_keywords=\u0026_questions_and_answers_WAR_questions_and_answersportlet_advancedSearch=false\u0026_questions_and_answers_WAR_questions_and_answersportlet_andOperator=true\u0026_questions_and_answers_WAR_questions_and_answersportlet_jspPage=%2Fhtml%2Fview.jsp\u0026_questions_and_answers_WAR_questions_and_answersportlet_statusSearch=All\u0026_questions_and_answers_WAR_questions_and_answersportlet_viewTab=1\u0026_questions_and_answers_WAR_questions_and_answersportlet_keyword=\u0026_questions_and_answers_WAR_questions_and_answersportlet_articleSearch=\u0026_questions_and_answers_WAR_questions_and_answersportlet_typeOfSubmitterSearch=\u0026_questions_and_answers_WAR_questions_and_answersportlet_publicIdSearch=\u0026_questions_and_answers_WAR_questions_and_answersportlet_startingDateSearch=\u0026_questions_and_answers_WAR_questions_and_answersportlet_endingDateSearch=\u0026_questions_and_answers_WAR_questions_and_answersportlet_applicableFromDate=\u0026_questions_and_answers_WAR_questions_and_answersportlet_applicableUntilDate=\u0026_questions_and_answers_WAR_questions_and_answersportlet_publicationFromDateSearch=\u0026_questions_and_answers_WAR_questions_and_answersportlet_publicationToDateSearch=\u0026_questions_and_answers_WAR_questions_and_answersportlet_currentTab=All\u0026_questions_and_answers_WAR_questions_and_answersportlet_resetCur=false\u0026_questions_and_answers_WAR_questions_and_answersportlet_delta=200\n\n* Page 2\n\nhttps://eba.europa.eu/single-rule-book-qa?p_p_id=questions_and_answers_WAR_questions_and_answersportlet\u0026p_p_lifecycle=0\u0026p_p_state=normal\u0026p_p_mode=view\u0026p_p_col_id=column-1\u0026p_p_col_pos=1\u0026p_p_col_count=2\u0026_questions_and_answers_WAR_questions_and_answersportlet_delta=200\u0026_questions_and_answers_WAR_questions_and_answersportlet_keywords=\u0026_questions_and_answers_WAR_questions_and_answersportlet_advancedSearch=false\u0026_questions_and_answers_WAR_questions_and_answersportlet_andOperator=true\u0026_questions_and_answers_WAR_questions_and_answersportlet_jspPage=%2Fhtml%2Fview.jsp\u0026_questions_and_answers_WAR_questions_and_answersportlet_statusSearch=All\u0026_questions_and_answers_WAR_questions_and_answersportlet_viewTab=1\u0026_questions_and_answers_WAR_questions_and_answersportlet_keyword=\u0026_questions_and_answers_WAR_questions_and_answersportlet_articleSearch=\u0026_questions_and_answers_WAR_questions_and_answersportlet_typeOfSubmitterSearch=\u0026_questions_and_answers_WAR_questions_and_answersportlet_publicIdSearch=\u0026_questions_and_answers_WAR_questions_and_answersportlet_startingDateSearch=\u0026_questions_and_answers_WAR_questions_and_answersportlet_endingDateSearch=\u0026_questions_and_answers_WAR_questions_and_answersportlet_applicableFromDate=\u0026_questions_and_answers_WAR_questions_and_answersportlet_applicableUntilDate=\u0026_questions_and_answers_WAR_questions_and_answersportlet_publicationFromDateSearch=\u0026_questions_and_answers_WAR_questions_and_answersportlet_publicationToDateSearch=\u0026_questions_and_answers_WAR_questions_and_answersportlet_currentTab=All\u0026_questions_and_answers_WAR_questions_and_answersportlet_resetCur=false\u0026_questions_and_answers_WAR_questions_and_answersportlet_cur=2\n\nThis will generate 9 html pages with max `200` Q\u0026As. These scraped pages need to be then processed.\n\n### Parsing\n\nExtract URL to the actual questions from the downloaded website using `parse_EBA_QA_page()`\n\n### Scraping\n\nScrape the actual Q\u0026As into a tabular form with `scrape_EBA_QA()`\n\n### Disclaimer \n\nThis package is an outcome of a research project. All errors are mine. All views expressed are personal views, not those of any employer.\nThe package is provided as-is and is not updated. Data provided here was scraped in 2019 so obtaining the latest data might require running the scraper again.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ferykwalczak%2Febarulebook","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Ferykwalczak%2Febarulebook","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ferykwalczak%2Febarulebook/lists"}