{"id":22538113,"url":"https://github.com/cfpb/website-indexer","last_synced_at":"2025-04-09T20:15:13.168Z","repository":{"id":37849221,"uuid":"472686270","full_name":"cfpb/website-indexer","owner":"cfpb","description":"Website crawler for consumerfinance.gov 🪱","archived":false,"fork":false,"pushed_at":"2025-03-21T10:46:54.000Z","size":1423,"stargazers_count":6,"open_issues_count":4,"forks_count":6,"subscribers_count":11,"default_branch":"main","last_synced_at":"2025-04-09T20:15:07.822Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"cc0-1.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/cfpb.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2022-03-22T09:01:42.000Z","updated_at":"2025-03-21T10:46:58.000Z","dependencies_parsed_at":"2024-03-15T12:46:34.904Z","dependency_job_id":"0e65891b-e5c4-45ed-8d35-fe9145873720","html_url":"https://github.com/cfpb/website-indexer","commit_stats":null,"previous_names":["cfpb/website-indexer","cfpb/crawsqueal"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/cfpb%2Fwebsite-indexer","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/cfpb%2Fwebsite-indexer/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/cfpb%2Fwebsite-indexer/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/cfpb%2Fwebsite-indexer/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/cfpb","download_url":"https://codeload.github.com/cfpb/website-indexer/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248103872,"owners_count":21048245,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-12-07T11:10:12.293Z","updated_at":"2025-04-09T20:15:13.141Z","avatar_url":"https://github.com/cfpb.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# website-indexer 🪱\n\nCrawl a website and search its content.\n\nThis project consists of two components:\na **crawler** application to crawl the contents of a website and store its content in a database; and a **viewer** web application that allows for searching of that crawled content.\n\nBoth components require\n[Python 3.12](https://www.python.org/)\nto run and are built using the\n[Django](https://www.djangoproject.com/)\nweb application framework.\nThe crawler piece is built on top of the Archive Team's\n[ludios_wpull](https://github.com/ArchiveTeam/ludios_wpull)\nweb crawler.\n\n## Getting started\n\nThis project can be run\n[using Docker](#using-docker)\nor a local\n[Python virtual environment](#using-a-python-virtual-environment).\n\n### Using Docker\n\nTo build the Docker image:\n\n```sh\ndocker build -t website-indexer:main .\n```\n\n#### Viewing a sample crawl using Docker\n\nTo then run the viewer application using sample data:\n\n```sh\ndocker run -it \\\n    -p 8000:8000 \\\n    website-indexer:main\n```\n\nThe web application using sample data will be accessible at http://localhost:8000/.\n\n#### Crawling a website and viewing the crawl results using Docker\n\nTo crawl a website using the Docker image,\nstoring the result in a local SQLite database named `crawl.sqlite3`,\nfirst create the database file:\n\n```sh\ndocker run -it \\\n    -v `pwd`:/data \\\n    -e DATABASE_URL=sqlite:////data/crawl.sqlite3 \\\n    website-indexer:main \\\n    python manage.py migrate\n```\n\nand then run the crawl, storing results into that database file:\n\n```sh\ndocker run -it \\\n    -v `pwd`:/data \\\n    -e DATABASE_URL=sqlite:////data/crawl.sqlite3 \\\n    website-indexer:main \\\n    python manage.py crawl https://www.consumerfinance.gov\n```\n\nTo then run the viewer web application to view that crawler database:\n\n```sh\ndocker run -it \\\n    -p 8000:8000 \\\n    -v `pwd`:/data \\\n    -e DATABASE_URL=sqlite:////data/crawl.sqlite3 \\\n    website-indexer:main\n```\n\nThe web application with the crawl results will be accessible at http://localhost:8000/.\n\n### Using a Python virtual environment\n\nCreate a Python virtual environment and install required packages:\n\n```sh\npython3.12 -m venv venv\nsource venv/bin/activate\npip install -r requirements/base.txt\n```\n\nFrom the repo's root, compile frontend assets:\n\n```sh\nyarn\nyarn build\n```\n\nAlternatively, to continuously watch the frontend assets and rebuild as necessary:\n\n```\nyarn\nyarn watch\n```\n\n#### Viewing a sample crawl using a Python virtual environment\n\nRun the viewer application using sample data:\n\n```sh\n./manage.py runserver\n```\n\nThe web application using sample data will be accessible at http://localhost:8000/.\n\n#### Crawling a website and viewing the crawl results using a Python virtual environment\n\nTo crawl a website and store the result in a local SQLite database named `crawl.sqlite3`:\n\n```sh\nDATABASE_URL=sqlite:///crawl.sqlite3 /manage.py crawl https://www.consumerfinance.gov\n```\n\nTo then run the viewer web application to view that crawler database:\n\n```sh\nDATABASE_URL=sqlite:///crawl.sqlite3 ./manage.py runserver\n```\n\nThe web application with the crawl results will be accessible at http://localhost:8000/\n\n### Managing crawls in the database\n\nThe `./manage.py manage_crawls` command can be used to list, delete, and cleanup old crawls (assuming `DATABASE_URL` is set appropriately).\n\nCrawls in the database have a `status` field which can be one of `Started`, `Finished`, or `Failed`.\n\n#### Listing crawls\n\nTo list crawls in the database:\n\n```sh\n./manage.py manage_crawls list\n```\n\nThis will list crawls in the database, including each crawl's unique ID.\n\n#### Deleting crawls\n\nTo delete an existing crawl, for example one with ID `123`:\n\n```sh\n./manage.py manage_crawls delete 123\n```\n\n`--dry-run` can be added to the `delete` command to preview its output\nwithout modifying the database.\n\n#### Cleaning crawls\n\nTo clean old crawls, leaving behind one crawl of each status:\n\n```sh\n./manage.py manage_crawls clean\n```\n\nTo modify the number of crawls left behind, for example leaving behind two of each status:\n\n```sh\n./manage.py manage_crawls clean --keep=2\n```\n\n`--dry-run` can also be added to the `clean` command to preview its output\nwithout modifying the database.\n\n## Configuration\n\n### Database configuration\n\nThe `DATABASE_URL` environment variable can be used to specify the database\nused for crawl results by the viewer application.\nThis project makes use of the\n[dj-database-url](https://github.com/jazzband/dj-database-url)\nproject to convert that variable into a Django database specification.\n\nFor example, to use a SQLite file at `/path/to/db.sqlite`:\n\n```sh\nexport DATABASE_URL=sqlite:////path/to/db.sqlite\n```\n\n(Note use of four slashes when referring to an absolute path;\nonly three are needed when referring to a relative path.)\n\nTo point to a PostgreSQL database instead:\n\n```sh\nexport DATABASE_URL=postgres://username:password@localhost/dbname\n```\n\nPlease see\n[the dj-database-url documentation](https://github.com/jazzband/dj-database-url)\nfor additional examples.\n\nIf the `DATABASE_URL` environment variable is left unset, the\n[sample SQLite database file](#sample-test-data)\nwill be used.\n\n### Google Tag Manager\n\nTo enable Google Tag Manager on all pages on the viewer application,\ndefine the `GOOGLE_TAG_ID` environment variable.\n\n### Using files as an alternative to environment variables\n\nAs alternative to configuration via environment variable,\nvalues may be provided in files instead.\n\nSet the `PATCH_ENVIRON_PATH` environment variable to the location of a directory\ncontaining files named for environment variables to set,\neach containing the desired value for that variable.\n\nWhen configured this way,\nthe application environment will be populated with those file-based values.\nValues that already exist in the environment will not be overwritten.\n\n## Development\n\n### Sample test data\n\nThis repository includes a sample database file for testing purposes at `sample/sample.sqlite3`.\n\nThe sample database file is used by the viewer application when no other crawl\ndatabase file has been specified.\n\nThe source website content used to generate this file is included in this repository\nunder the `sample/src` subdirectory.\n\nTo regenerate the same database file, first delete it:\n\n```sh\nrm ./sample/sample.sqlite3\n```\n\nThen, start a Python webserver to serve the sample website locally:\n\n```sh\ncd ./sample/src \u0026\u0026 python -m http.server\n```\n\nThis starts the sample website running at http://localhost:8000.\n\nThen, in another terminal, recreate the database file:\n\n```sh\n./manage.py migrate\n```\n\nFinally, perform the crawl against the locally running site:\n\n```sh\n./manage.py crawl http://localhost:8000/\n```\n\nThese commands assume use of a local Python virtual environment;\nalternatively consider\n[using Docker](#crawling-a-website-and-viewing-the-crawl-results-using-docker).\n\nThis command will receate the sample database file\n`sample/sample.sqlite3`\nwith a fresh crawl.\nTo write to a different database, use\n[the `DATABASE_URL` environment variable](#database-configuration).\n\nFor consistency, the\n[Python test fixture](#testing)\nshould be updated at the same time as the sample database.\n\n### Testing\n\nTo run Python unit tests, first install the test dependencies in your virtual environment:\n\n```sh\npip install -r requirements/test.txt\n```\n\nTo run the tests:\n\n```sh\npytest\n```\n\nThe Python tests make use of a test fixture generated from\n[the sample database](#sample-test-data).\n\nTo recreate this test fixture:\n\n```sh\n./manage.py dumpdata --indent=4 crawler \u003e crawler/fixtures/sample.json\n```\n\n### Code formatting\n\nThis project uses [Black](https://github.com/psf/black) as a Python code formatter.\n\nTo check if your changes to project code match the desired coding style:\n\n```sh\nblack . --check\n```\n\nYou can fix any problems by running:\n\n```sh\nblack .\n```\n\nThis project uses [Prettier](https://prettier.io/) as a code formatter\nfor JavaScript, CSS, and HTML templates.\n\nTo check if your changes to project code match the desired coding style:\n\n```sh\nyarn prettier\n```\n\nYou can fix any problems by running:\n\n```sh\nyarn prettier:fix\n```\n\n## Deployment\n\nFor information on how this project is deployed at the CFPB,\nemployees and contractors should refer to the internal\n[CFGOV/crawler-deploy](https://github.local/CFGOV/crawler-deploy/) 🔒\nrepository.\n\n## Open source licensing info\n\n1. [TERMS](TERMS.md)\n2. [LICENSE](LICENSE)\n3. [CFPB Source Code Policy](https://github.com/cfpb/source-code-policy/)\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcfpb%2Fwebsite-indexer","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fcfpb%2Fwebsite-indexer","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcfpb%2Fwebsite-indexer/lists"}