{"id":22538193,"url":"https://github.com/cfpb/crawl-cfgov","last_synced_at":"2026-02-14T16:33:09.783Z","repository":{"id":39333963,"uuid":"302103151","full_name":"cfpb/crawl-cfgov","owner":"cfpb","description":"Archive the HTML of consumerfinance.gov daily","archived":false,"fork":false,"pushed_at":"2026-02-14T06:09:55.000Z","size":22513155,"stargazers_count":10,"open_issues_count":3,"forks_count":11,"subscribers_count":22,"default_branch":"main","last_synced_at":"2026-02-14T14:08:45.743Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":"HTML","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"cc0-1.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/cfpb.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2020-10-07T16:56:00.000Z","updated_at":"2026-02-14T06:09:58.000Z","dependencies_parsed_at":"2025-06-04T07:21:24.594Z","dependency_job_id":"4e8889a4-a083-4318-82db-dae2d7abdf6d","html_url":"https://github.com/cfpb/crawl-cfgov","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/cfpb/crawl-cfgov","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/cfpb%2Fcrawl-cfgov","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/cfpb%2Fcrawl-cfgov/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/cfpb%2Fcrawl-cfgov/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/cfpb%2Fcrawl-cfgov/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/cfpb","download_url":"https://codeload.github.com/cfpb/crawl-cfgov/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/cfpb%2Fcrawl-cfgov/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":29449370,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-02-14T15:52:44.973Z","status":"ssl_error","status_checked_at":"2026-02-14T15:52:11.208Z","response_time":53,"last_error":"SSL_connect returned=1 errno=0 peeraddr=140.82.121.6:443 state=error: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-12-07T11:10:44.103Z","updated_at":"2026-02-14T16:33:09.764Z","avatar_url":"https://github.com/cfpb.png","language":"HTML","funding_links":[],"categories":[],"sub_categories":[],"readme":"# HTML Archive of consumerfinance.gov\n\n**Description**:\nThis repo contains an archive of consumerfinance.gov HTML, generated via web crawl.\nThis can serve as a resource when answering questions about how the site changes and when those changes happen.\n\n  - **Technology**: Uses `wget` to crawl the site and download HTML.\n  - **Status**:  See the [CHANGELOG](CHANGELOG.md).\n\n## The archive\n\nThe archive of consumerfinance.gov is contained in the `www.consumerfinance.gov` directory of this repo.\nIt contains an HTML download of each page it crawls.\nThe archive does not download any CSS, JavaScript, or images associated with the pages.\nIt also does not contain any PDF, CSV, or other supplementary files linked from consumerfinance.gov.\nThe HTML is [simplified](./postprocessing/) before being archived to prevent unimportant diffs.\n\n## Dependencies\n\nThis project uses [`wget`](https://www.gnu.org/software/wget/) to crawl consumerfinance.gov and download the HTML.\nYou can install it on a Mac using `brew install wget`.\n\n## Installation\n\nTo get a copy of the consumerfinance.gov archive or run a crawl on your computer, clone this repository.\n\n## Usage\n\n### Exploring the archive\n\nTo view the consumerfinance.gov archive, you can browse the history of this repo here on github.com, or clone this repository.\n\n#### Performing basic searches in a browser\n\nGitHub.com's search functionality can be used to perform basic searches for words or phrases.\nFor example,\n[searching for \"reverse  mortgage\"](https://github.com/cfpb/crawl-cfgov/search?q=%22reverse+mortgage%22)\nreturns all pages containing that term.\nUnfortunately, although GitHub does provide the ability to\n[customize search results](https://docs.github.com/en/free-pro-team@latest/github/searching-for-information-on-github/searching-code),\nit only supports basic querying and filtering.\nAdvanced searches can be more easily performed locally using shell commands after cloning this repository.\n\n#### Using shell commands to search locally\n\nOnce this repository has been cloned locally, the common shell command `grep` can be used for common searches.\nFor example, to list all instances of the case-insensitive phrase \"reverse mortgage\":\n\n```sh\ngrep -ri \"reverse mortgage\" www.consumerfinance.gov\n```\n\nTo list only matching filenames, use the `-l` option:\n\n```sh\ngrep -ril \"reverse mortgage\" www.consumerfinance.gov\n```\n\nYou may want to also sort the results alphabetically:\n\n```sh\ngrep -ril \"reverse mortgage\" www.consumerfinance.gov | sort\n```\n\nVersions of `grep` with support for extended regular expressions allow additional searches.\nFor example, to find all occurrences of a GovDelivery code like `USCFPB_12345`:\n\n```sh\ngrep -rE 'USCFPB_[0-9]+' www.consumerfinance.gov\n```\n\nThe results of a `grep` search can be piped to tools like `sed` to do further processing.\n\nFor example, let's say you want to check the value of all `aria-label` attributes on Spanish pages:\n\n```sh\n# Generate a list of Spanish pages, using the presence of \"Un sitio web\"\n# in the site header to distinguish them from English pages.\ngrep -rl \"Un sitio web\" www.consumerfinance.gov \u003e spanish-pages.txt\n\n# Grep only those files again to find all aria-label attributes.\ncat spanish-pages.txt | xargs grep aria-label \u003e spanish-aria-labels.txt\n\n# Used sed to extract the list of aria-labels, and show a sorted list of unique values.\ncat spanish-aria-labels.txt | sed -n 's/^.*aria-label=\"\\([^\"]*\\)\".*$/\\1/p' | sort | uniq\n```\n\nCommand line tools like\n[`grep`](https://linux.die.net/man/1/grep)\nand\n[`sed`](https://linux.die.net/man/1/sed)\nare very complex (and can vary depending on operating system),\nso reading their documentation can be helpful in creating searches.\n\n### Running the crawler locally\n\nTo run a crawl on your computer, `cd` into the root of this project and use the following command:\n\n```sh\n./crawl.sh https://www.consumerfinance.gov\n```\n\nA full crawl can take several hours. To limit the crawl depth:\n\n```sh\n./crawl.sh -d 4 https://www.consumerfinance.gov\n```\n\nOr, to start the crawl at a specific URL:\n\n```sh\n./crawl.sh https://www.consumerfinance.gov/start/crawl/here/\n```\n\n## Known issues\n\nThe crawl has some constraints and limitations.\n- The results intentionally only contain pages that share the same domain.\n- The crawl will not include any pages that are not linked to from any other page reachable from the site root.\n- The crawl records each page based on its url.\n  If we accidentally record a page with url parameters, it counts that as a separate page, which could result in duplication.\n- There are some pages on consumerfinance.gov that can only be found by paging through paginated lists of results.\n  We try to configure the crawl to find and download all of these pages, but it's possible there will be omissions.\n\n## Getting help\n\nIf you have questions, concerns, bug reports, etc, please file an issue in this repository's Issue Tracker.\n\n## Getting involved\n\nSee our [contributing guidelines](CONTRIBUTING.md).\n\n\n----\n\n## Open source licensing info\n1. [TERMS](TERMS.md)\n2. [LICENSE](LICENSE)\n3. [CFPB Source Code Policy](https://github.com/cfpb/source-code-policy/)\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcfpb%2Fcrawl-cfgov","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fcfpb%2Fcrawl-cfgov","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcfpb%2Fcrawl-cfgov/lists"}