{"id":17177482,"url":"https://github.com/sangaline/scrapy-wayback-machine","last_synced_at":"2025-04-09T15:09:32.741Z","repository":{"id":53406909,"uuid":"87251546","full_name":"sangaline/scrapy-wayback-machine","owner":"sangaline","description":"A Scrapy middleware for scraping time series data from Archive.org's Wayback Machine.","archived":false,"fork":false,"pushed_at":"2024-02-18T19:23:17.000Z","size":147,"stargazers_count":113,"open_issues_count":5,"forks_count":29,"subscribers_count":5,"default_branch":"master","last_synced_at":"2025-04-09T15:09:27.765Z","etag":null,"topics":["archive-dot-org","middleware","python","scrapy","scrapy-extension","wayback-machine","web-scraping"],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"isc","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/sangaline.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE.txt","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2017-04-05T01:21:25.000Z","updated_at":"2025-03-17T16:30:43.000Z","dependencies_parsed_at":"2024-06-20T22:12:51.156Z","dependency_job_id":null,"html_url":"https://github.com/sangaline/scrapy-wayback-machine","commit_stats":{"total_commits":48,"total_committers":3,"mean_commits":16.0,"dds":0.0625,"last_synced_commit":"22d525decd7e334907812611dca092365311c443"},"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sangaline%2Fscrapy-wayback-machine","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sangaline%2Fscrapy-wayback-machine/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sangaline%2Fscrapy-wayback-machine/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sangaline%2Fscrapy-wayback-machine/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/sangaline","download_url":"https://codeload.github.com/sangaline/scrapy-wayback-machine/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248055282,"owners_count":21040157,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["archive-dot-org","middleware","python","scrapy","scrapy-extension","wayback-machine","web-scraping"],"created_at":"2024-10-15T00:04:12.304Z","updated_at":"2025-04-09T15:09:32.717Z","avatar_url":"https://github.com/sangaline.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"![The Scrapy Wayback Machine Logo](img/logo.png)\n\n# Scrapy Wayback Machine Middleware\n\nThis project provides a [Scrapy](https://scrapy.org) middleware for scraping archived snapshots of webpages as they appear on [archive.org](http://archive.org)'s [Wayback Machine](https://archive.org/web/).\nThis can be useful if you're trying to scrape a site that has scraping measures that make direct scraping impossible or prohibitively slow.\nIt's also useful if you want to scrape a website as it appeared at some point in the past or to scrape information that changes over time.\n\nIf you're just just interested in mirroring page content or would like to parse the HTML content in a language other than python then you should check out [the Wayback Machine Scraper](https://github.com/sangaline/wayback-machine-scraper).\nIt's a command-line utility that uses the middleware provided here to crawl through historical snapshots of a website and save them to disk.\nIt's highly configurable in terms of what it scrapes but it only saves the unparsed content of the pages on the site.\nThis may or may not suit your needs.\n\nIf you're using [Scrapy](https://scrapy.org) already or interested in parsing the data that is crawled then this `WaybackMachineMiddleware` is probably what you want.\nThis middleware handles all of the tricky parts and passes normal `response` objects to your [Scrapy](https://scrapy.org) spiders with archive timestamp information attached.\nThe middleware is very unobtrusive and should work seamlessly with existing [Scrapy](https://scrapy.org) middlewares, extensions, and spiders.\n\n## Installation\n\nThe package can be installed using `pip`.\n\n```bash\npip install scrapy-wayback-machine\n```\n\n## Usage\n\nTo enable the middleware you simply have to add\n\n```python\nDOWNLOADER_MIDDLEWARES = {\n    'scrapy_wayback_machine.WaybackMachineMiddleware': 5,\n}\n\nWAYBACK_MACHINE_TIME_RANGE = (start_time, end_time)\n```\n\nto your [Scrapy](https://scrapy.org) settings.\nThe start and end times can be specified as `datetime.datetime` objects, Unix timestamps, `YYYYmmdd` timestamps, or `YYYYmmddHHMMSS` timestamps.\nThe type will be automatically inferred from the content and the ranges will limit the range of snapshots to crawl.\nYou can also pass a single time if you would like to scrape pages as they appeared at that time.\n\nAfter configuration, responses will be passed to your spiders as they normally would.\nBoth `response.url` and all links within `response.body` point to the unarchived content so your parsing code should work the same regardless of whether or not the middleware is enabled.\nIf you need to access either the time of the snapshot or the [archive.org](http://archive.org) URL for a response then this information is easily available as metadata attached to the response.\nNamely, `response.meta['wayback_machine_time']` contains a `datetime.datetime` corresponding to the time of the crawl and `response.meta['wayback_machine_url']` contains the actual URL that was requested.\nUnless you're scraping a single point in time, you will almost certainly want to include the timestamp in the items that your spiders produce to differentiate items scraped from the same URL.\n\n### Examples\n\n[The Wayback Machine Scraper](https://github.com/sangaline/wayback-machine-scraper) command-line utility is a good example of how to use the middleware.\nThe necessary settings are defined in [\\_\\_main\\_\\_.py](https://github.com/sangaline/wayback_machine_scraper/scraper/__main__.py) and the handling of responses is done in [mirror_spider.py](https://github.com/sangaline/wayback_machine_scraper/scraper/mirror_spider.py).\nThe `MirrorSpider` class simply uses the `response.meta['wayback_machine_time']` information attached to each response to construct the snapshot filenames and is otherwise a fairly generic spider.\n\nThere's also an article [Internet Archeology: Scraping time series data from Archive.org](http://sangaline.com/post/wayback-machine-scraper/) that discusses the development of the middleware and includes an example of scraping time series data from [Reddit](https://reddit.com).\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsangaline%2Fscrapy-wayback-machine","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fsangaline%2Fscrapy-wayback-machine","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsangaline%2Fscrapy-wayback-machine/lists"}