{"id":32481781,"url":"https://github.com/overbrowsing/wasteback-machine","last_synced_at":"2026-07-06T17:31:33.188Z","repository":{"id":320462903,"uuid":"1049160904","full_name":"overbrowsing/wasteback-machine","owner":"overbrowsing","description":"JavaScript library for analysing archived web pages.","archived":false,"fork":false,"pushed_at":"2026-06-12T16:14:15.000Z","size":164,"stargazers_count":6,"open_issues_count":1,"forks_count":1,"subscribers_count":1,"default_branch":"main","last_synced_at":"2026-06-12T17:23:02.645Z","etag":null,"topics":["internet-archive","memento","wayback-machine","web-archiving","website-analysis"],"latest_commit_sha":null,"homepage":"","language":"JavaScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/overbrowsing.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2025-09-02T15:15:37.000Z","updated_at":"2026-06-12T16:14:19.000Z","dependencies_parsed_at":"2025-10-24T00:12:02.179Z","dependency_job_id":"4414b9ef-3c9b-4035-b76a-15d1b632bb1e","html_url":"https://github.com/overbrowsing/wasteback-machine","commit_stats":null,"previous_names":["overbrowsing/wasteback-machine"],"tags_count":1,"template":false,"template_full_name":null,"purl":"pkg:github/overbrowsing/wasteback-machine","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/overbrowsing%2Fwasteback-machine","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/overbrowsing%2Fwasteback-machine/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/overbrowsing%2Fwasteback-machine/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/overbrowsing%2Fwasteback-machine/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/overbrowsing","download_url":"https://codeload.github.com/overbrowsing/wasteback-machine/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/overbrowsing%2Fwasteback-machine/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":35201088,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-26T15:22:16.424Z","status":"online","status_checked_at":"2026-07-06T02:00:07.184Z","response_time":106,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["internet-archive","memento","wayback-machine","web-archiving","website-analysis"],"created_at":"2025-10-27T02:58:39.710Z","updated_at":"2026-07-06T17:31:32.750Z","avatar_url":"https://github.com/overbrowsing.png","language":"JavaScript","funding_links":[],"categories":["Software"],"sub_categories":["Command-line Clients"],"readme":"# Wasteback Machine\n\n[![NPM version](https://img.shields.io/npm/v/@overbrowsing/wasteback-machine.svg)](https://www.npmjs.com/package/@overbrowsing/wasteback-machine)\n[![npm](https://img.shields.io/npm/dm/@overbrowsing/wasteback-machine.svg)](https://www.npmtrends.com/@overbrowsing/wasteback-machine)\n[![License](https://img.shields.io/npm/l/@overbrowsing/wasteback-machine.svg)](https://tldrlegal.com/license/apache-license-2.0-(apache-2.0))\n\n## Overview\n\nWasteback Machine is a JavaScript library for analysing archived web pages, measuring their size and composition to enable retrospective, quantitative web research.\n\n## Features\n\n- **Multi-archive support:** [Supports 20+ web archives](#supported-web-archives) and is [extensible to additional archives](#adding-web-archives).\n- **Aggregate mementos:** Retrieve memento-datetimes for a target URL from an archive’s CDX server.\n- **Analyse page composition:** Break down archived web pages by resource type, including HTML, stylesheets, scripts, images, etc.\n- **Calculate size metrics:** Compute total and per-type sizes, including counts and bytes.\n- **Generate resource inventory:** Optionally produce an inventory of all resources with metadata.\n- **Completeness scoring:** Assess how fully an archived web page was retrieved.\n- **CLI utility:** Analyse archived web pages [directly from the command line](#cli).\n\n## Installation\n\n```sh\nnpm i @overbrowsing/wasteback-machine\n```\n\n## Usage\n\nWasteback Machine provides two functions:\n\n1. **`getMementos`**: Fetch all memento-datetimes from the CDX server of a [supported web archive](#supported-web-archives) for a given URL.\n2. **`analyseMemento`**: Analyses the size and composition of an archived web page from a [supported web archive](#supported-web-archives).\n\n### 1. Fetch Available Memento-datetimes (`getMementos`)\n\n  Fetch all memento-datetimes for https://nytimes.com, from the Internet Archive [(🆔 = ia)](#supported-web-archives).\n\n  ```javascript\n  import { getMementos } from \"@overbrowsing/wasteback-machine\";\n\n  const mementos = await getMementos(\n    \"ia\", // Web archive ID (🆔 = ia, Internet Archive)\n    \"https://nytimes.com\", // Target URL\n  );\n\n  console.log(mementos);\n  ```\n\n  #### Example Output\n\n  ```javascript\n  [\n    '19961112181513', '19961121230155', '19961219002950', '19961220073509',\n    '19961226135029', '19961228014508', '19961230230427', '19970209220858',\n    '19970303103041', '19970414192930', '19970414210143', '19970415180120',\n    ... 688983 more items\n  ]\n  ```\n\n### 2. Analyse An Archived Web Page (`analyseMemento`)\n\nAnalyse the archived snapshot of https://nytimes.com, November 12, 1996, from the Internet Archive [(🆔 = ia)](#supported-web-archives).\n\n\u003e [!TIP]\n\u003e If you provide a full 14-digit datetime (`YYYYMMDDHHMMSS`) using [`getMementos`](#1-fetch-available-memento-datetimes-getmementos), Wasteback Machine skips the TimeGate (URI-G) lookup, improving performance.\n\n```javascript\nimport { analyseMemento } from \"@overbrowsing/wasteback-machine\";\n\nconst mementoData = await analyseMemento(\n  \"ia\", // Web archive ID (🆔 = ia, Internet Archive)\n  \"https://nytimes.com\", // Target URL\n  \"19961112\", // Target memento-datetime (YYYYMMDDhhmmss); minimum input: YYYY\n  { includeResources: true } // Resource list (true/false)\n);\n\nconsole.log(mementoData);\n```\n\n#### Example Output\n\n```javascript\n{\n  target: {\n    url: 'https://nytimes.com', \n    datetime: '19961112'\n  },\n  memento: {\n    url: 'https://web.archive.org/web/19961112181513if_/https://nytimes.com',\n    datetime: '19961112181513',\n  },\n  archive: {\n    name: 'Internet Archive (Wayback Machine)',\n    organisation: 'Internet Archive',\n    country: 'United States of America',\n    continent: 'North America',\n    url: 'https://web.archive.org',\n  },\n  sizes: {\n    html: { bytes: 1653, count: 1 },\n    stylesheet: { bytes: 0, count: 0 },\n    script: { bytes: 0, count: 0 },\n    image: { bytes: 46226, count: 2 },\n    video: { bytes: 0, count: 0 },\n    audio: { bytes: 0, count: 0 },\n    font: { bytes: 0, count: 0 },\n    flash: { bytes: 0, count: 0 },\n    plugin: { bytes: 0, count: 0 },\n    data: { bytes: 0, count: 0 },\n    document: { bytes: 0, count: 0 },\n    other: { bytes: 0, count: 0 },\n    total: { bytes: 47879, count: 3 }\n  },\n  completeness: '100%',\n  resources: [\n    {\n      url: 'https://web.archive.org/web/19961112181513im_/http://www.nytimes.com/index.gif',\n      type: 'image',\n      size: 45259\n    },\n    {\n      url: 'https://web.archive.org/web/19961112181513im_/http://www.nytimes.com/free-images/marker.gif',\n      type: 'image',\n      size: 967\n    }\n  ]\n}\n```\n\n## Supported Web Archives\n\nEach supported web archive has a unique web archive ID (🆔) required for API calls. The table also indicates which functions each archive supports.\n\n| Web Archive                                                                                                                                | Organisation                                      | 🆔                                           | [`getMementos`](#1-fetch-available-memento-datetimes-getmementos) | [`analyseMemento`](#2-analyse-an-archived-web-page-analysememento) |\n|--------------------------------------------------------------------------------------------------------------------------------------------|---------------------------------------------------|---------------------------------------------|------------------------------------------------------------------ |--------------------------------------------------------------------|\n| [Arquivo.pt](https://arquivo.pt)                                                                                                           | 🇵🇹 FCCN/FCT                                        | [arq](/src/archives/arq/arq.js)             | ✓                                                                | ✓                                                                  |\n| [National Library and Archives of Quebec (BAnQ) Web Archiving](https://www2.banq.qc.ca/collections/collections_patrimoniales/archives_web) | 🇨🇦 National Library and Archives of Quebec (BAnQ)  | [banq](/src/archives/banq/banq.js)          | ✕                                                                | ✓                                                                  |\n| [Columbia University Libraries Web Archives](https://library.columbia.edu/collections/web-archives)                                        | 🇺🇸 Columbia University Libraries                   | [cul](/src/archives/cul/cul.js)             | ✓                                                                | ✓                                                                  |\n| [Webarchiv](https://webarchiv.cz)                                                                                                          | 🇨🇿 National Library of the Czech Republic          | [cz](/src/archives/cz/cz.js)                | ✓                                                                | ✓                                                                  |\n| [European Union Web Archive](https://op.europa.eu/en/web/euwebarchive)                                                                     | 🇪🇺 European Union                                  | [euwa](/src/archives/euwa/euwa.js)          | ✓                                                                | ✓                                                                  |\n| [Estonian Web Archive](https://veebiarhiiv.digar.ee)                                                                                       | 🇪🇪 National Library of Estonia                     | [ewa](/src/archives/ewa/ewa.js)             | ✓                                                                | ✓                                                                  |\n| [Government of Canada Web Archive](https://webarchiveweb.bac-lac.canada.ca)                                                                | 🇨🇦 Library and Archives Canada                     | [gcwa](/src/archives/gcwa/gcwa.js)          | ✓                                                                | ✓                                                                  |\n| [Croatian Web Archives (HAW)](https://haw.nsk.hr)                                                                                          | 🇭🇷 National and University Library in Zagreb       | [haw](/src/archives/haw/haw.js)             | ✓                                                                | ✓                                                                  |\n| [Internet Archive (Wayback Machine)](https://web.archive.org)                                                                              | 🇺🇸 Internet Archive                                | [ia](/src/archives/ia/ia.js)                | ✓                                                                | ✓                                                                  |\n| [Icelandic Web Archive (Vefsafn.is)](https://vefsafn.is)                                                                                   | 🇮🇸 National and University Library of Iceland      | [iwa](/src/archives/iwa/iwa.js)             | ✓                                                                | ✓                                                                  |\n| [Library of Congress Web Archive](https://loc.gov/web-archives)                                                                            | 🇺🇸 Library of Congress                             | [loc](/src/archives/loc/loc.js)             | ✕                                                                | ✓                                                                  |\n| [National Library of Ireland Web Archive](https://nli.ie/collections/our-collections/web-archive)                                          | 🇮🇪 National Library of Ireland                     | [nliwa](/src/archives/nliwa/nliwa.js)       | ✓                                                                | ✓                                                                  |\n| [National Library of Medicine](https://archive-it.org/organizations/350)                                                                   | 🇺🇸 National Library of Medicine                    | [nlm](/src/archives/nlm/nlm.js)             | ✓                                                                | ✓                                                                  |\n| [National Records of Scotland Web Archive](https://webarchive.nrscotland.gov.uk)                                                           | 🏴󠁧󠁢󠁳󠁣󠁴󠁿 National Records of Scotland                    | [nrs](/src/archives/nrs/nrs.js)             | ✓                                                                | ✓                                                                  |\n| [Norwegian Web Archive](https://nb.no/en/collection/web-archive)                                                                           | 🇳🇴 National Library of Norway                      | [nwa](/src/archives/nwa/nwa.js)             | ✓                                                                | ✓                                                                  |\n| [New Zealand Web Archive](https://webarchive.natlib.govt.nz)                                                                               | 🇳🇿 National Library of New Zealand                 | [nzwa](/src/archives/nzwa/nzwa.js)          | ✓                                                                | ✓                                                                  |\n| [The Web Archive of Catalonia (Padicat)](https://padicat.cat)                                                                              | 🇪🇸 Library of Catalonia                            | [padicat](/src/archives/padicat/padicat.js) | ✓                                                                | ✓                                                                  |\n| [PRONI Web Archive](https://webarchive.proni.gov.uk)                                                                                       | 🇬🇧 The Public Record Office of Northern Ireland    | [proni](/src/archives/proni/proni.js)       | ✓                                                                | ✓                                                                  |\n| [Smithsonian Institution Archives](https://siarchives.si.edu)                                                                              | 🇺🇸 Smithsonian Libraries and Archives              | [sia](/src/archives/sia/sia.js)             | ✓                                                                | ✓                                                                  |\n| [Spletni Arhiv](https://arhiv.nuk.uni-lj.si)                                                                                               | 🇸🇮 National and University Library of Slovenia     | [slo](/src/archives/slo/slo.js)             | ✕                                                                | ✓                                                                  |\n| [Australia Web Archive (Trove)](https://webarchive.nla.gov.au)                                                                             | 🇦🇺 National Library of Australia                   | [trove](/src/archives/trove/trove.js)       | ✕                                                                | ✓                                                                  |\n| [UK Government Web Archive (UKGWA)](https://nationalarchives.gov.uk/webarchive)                                                            | 🇬🇧 The National Archives                           | [ukgwa](/src/archives/ukgwa/ukgwa.js)       | ✓                                                                | ✓                                                                  |\n| [University of North Texas Web Archives](https://digital.library.unt.edu/explore/collections/untweb)                                       | 🇺🇸 University of North Texas University Libraries  | [untwa](/src/archives/untwa/untwa.js)       | ✓                                                                | ✓                                                                  |\n| [York University Digital Library](https://digital.library.yorku.ca)                                                                        | 🇨🇦 York University Libraries                       | [yudl](/src/archives/yudl/yudl.js)          | ✓                                                                | ✓                                                                  |\n\n### Adding Web Archives\n\nWasteback Machine can support additional web archives if they meet the following criteria:\n\n1. Provide a CDX server API (required for [`getMementos`](#1-fetch-available-memento-datetimes-getmementos)).\n2. Support the [Memento Protocol (RFC7089)](https://datatracker.ietf.org/doc/html/rfc7089) (required for [`analyseMemento`](#2-analyse-an-archived-web-page-analysememento)).\n3. Support replay state modifiers ([URL Rewrite Type Modifier](https://pywb.readthedocs.io/en/master/manual/rewriter.html?highlight=modifiers#url-rewrite-type-modifier)) endpoints for both:\n    -\tRaw content ([see example](https://web.archive.org/web/20131001001332id_/https://www.bbc.co.uk)).\n    -\tNavigational toolbars suppressed ([see example](https://web.archive.org/web/20131001001332if_/https://www.bbc.co.uk)).\n\nTo request support for an archive that meets these criteria, [submit an issue](https://github.com/overbrowsing/wasteback-machine/issues/new?template=add-new-web-archive.md) using the template.\n\n## CLI\n\nWasteback Machine CLI lets you analyse an archived web page to view its size, composition, and estimated emissions using [CO2.js](https://developers.thegreenwebfoundation.org/co2js/overview) and the [Sustainable Web Design Model](https://sustainablewebdesign.org/estimating-digital-emissions).\n\n### Quick Start\n\nAfter [installation](#installation), start the CLI:\n\n```sh\nnpx cli\n```\n\n### CLI Prompts\n\n```sh\n1. Enter web archive ID ('help' to list archives or [Enter ↵] = Internet Archive):\n2. Enter URL to analyse:\n3. Enter target year (YYYY):\n4. Enter target month (MM or [Enter ↵] = 01):\n5. Enter target day (DD or [Enter ↵] = 01):\n```\n\n#### Example Output\n\n```sh\n________________________________________________________\n\nMEMENTO INFO\n\n  Memento URL:    https://web.archive.org/web/19961112181513if_/https://nytimes.com\n  Web Archive:    Internet Archive (Wayback Machine)\n  Organisation:   Internet Archive\n  Website:        https://web.archive.org\n\n________________________________________________________\n\nPAGE SIZE\n\n  Data:           46.76 KB\n  Emissions:      0.014 g CO₂e\n  Completeness:   100%\n\n________________________________________________________\n\nPAGE COMPOSITION\n\n  HTML\n      Count:      1\n      Data:       1653 bytes (3.5%)\n      Emissions:  0.000 g CO₂e\n\n  IMAGE\n      Count:      2\n      Data:       46226 bytes (96.5%)\n      Emissions:  0.013 g CO₂e\n\n________________________________________________________\n```\n\n## Credits\n\nDeveloped by the [Overbrowsing Research Group](https://overbrowsing.com) at the [Institute for Design Informatics, The University of Edinburgh](https://designinformatics.org/portfolio-item/wasteback-machine), with support in part from the [European Association for Digital Humanities (EADH)](https://eadh.org).\n\n## Citing\n\nResults generated with Wasteback Machine may be freely cited, quoted, analysed, or republished with attribution to 'Wasteback Machine'. No special permission is required for academic, journalistic, or personal use.\n\nA publication related to this project appeared in the Proceedings of iConference 2026 ([view PDF](https://publicera.kb.se/ir/article/view/64185/51902)). Please cite as:\n\n\u003e Mahoney, D. (2026). Wasteback Machine: a method for quantitative measurement of the archived web. Information Research an International Electronic Journal, 31 (iConf), 448–464. https://doi.org/10.47989/ir31iConf64185\n\n```bib\n@article{Mahoney_2026,\n  author  = {Mahoney, David},\n  title   = {Wasteback Machine: a method for quantitative measurement of the archived web},\n  journal = {Information Research: An International Electronic Journal},\n  volume  = {31},\n  number  = {iConf},\n  pages   = {448-464},\n  year    = {2026},\n  month   = {Mar},\n  url     = {https://publicera.kb.se/ir/article/view/64185},\n  doi     = {10.47989/ir31iConf64185}\n}\n```\n\n## Licenses\n\nWasteback Machine is licensed under [Apache 2.0](https://tldrlegal.com/license/apache-license-2.0-(apache-2.0)). For full licensing details, see the [LICENSE](/LICENSE) file.\n\nUse of Wasteback Machine is subject to the terms, policies and licenses of each respective [supported web archive](#supported-web-archives).\n\n## Terms\n\nAll results generated by Wasteback Machine are provided \"as-is\" without warranties of any kind, express or implied, including but not limited to accuracy, completeness, or reliability. The authors and contributors accept no liability for any errors, omissions, or consequences arising from the use of this software or the results it produces.","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Foverbrowsing%2Fwasteback-machine","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Foverbrowsing%2Fwasteback-machine","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Foverbrowsing%2Fwasteback-machine/lists"}