https://github.com/overbrowsing/wasteback-machine
JavaScript library for analysing archived web pages.
https://github.com/overbrowsing/wasteback-machine
internet-archive memento wayback-machine web-archiving website-analysis
Last synced: 18 days ago
JSON representation
JavaScript library for analysing archived web pages.
- Host: GitHub
- URL: https://github.com/overbrowsing/wasteback-machine
- Owner: overbrowsing
- License: apache-2.0
- Created: 2025-09-02T15:15:37.000Z (11 months ago)
- Default Branch: main
- Last Pushed: 2026-06-12T16:14:15.000Z (about 1 month ago)
- Last Synced: 2026-06-12T17:23:02.645Z (about 1 month ago)
- Topics: internet-archive, memento, wayback-machine, web-archiving, website-analysis
- Language: JavaScript
- Homepage:
- Size: 160 KB
- Stars: 6
- Watchers: 1
- Forks: 1
- Open Issues: 1
-
Metadata Files:
- Readme: README.md
- License: LICENSE
Awesome Lists containing this project
- awesome-memento - src - machine)) - JavaScript library for analyzing archived web pages, measuring their size and composition to support retrospective, quantitative web research (Software / Command-line Clients)
README
# Wasteback Machine
[](https://www.npmjs.com/package/@overbrowsing/wasteback-machine)
[](https://www.npmtrends.com/@overbrowsing/wasteback-machine)
[](https://tldrlegal.com/license/apache-license-2.0-(apache-2.0))
## Overview
Wasteback Machine is a JavaScript library for analysing archived web pages, measuring their size and composition to enable retrospective, quantitative web research.
## Features
- **Multi-archive support:** [Supports 20+ web archives](#supported-web-archives) and is [extensible to additional archives](#adding-web-archives).
- **Aggregate mementos:** Retrieve memento-datetimes for a target URL from an archiveโs CDX server.
- **Analyse page composition:** Break down archived web pages by resource type, including HTML, stylesheets, scripts, images, etc.
- **Calculate size metrics:** Compute total and per-type sizes, including counts and bytes.
- **Generate resource inventory:** Optionally produce an inventory of all resources with metadata.
- **Completeness scoring:** Assess how fully an archived web page was retrieved.
- **CLI utility:** Analyse archived web pages [directly from the command line](#cli).
## Installation
```sh
npm i @overbrowsing/wasteback-machine
```
## Usage
Wasteback Machine provides two functions:
1. **`getMementos`**: Fetch all memento-datetimes from the CDX server of a [supported web archive](#supported-web-archives) for a given URL.
2. **`analyseMemento`**: Analyses the size and composition of an archived web page from a [supported web archive](#supported-web-archives).
### 1. Fetch Available Memento-datetimes (`getMementos`)
Fetch all memento-datetimes for https://nytimes.com, from the Internet Archive [(๐ = ia)](#supported-web-archives).
```javascript
import { getMementos } from "@overbrowsing/wasteback-machine";
const mementos = await getMementos(
"ia", // Web archive ID (๐ = ia, Internet Archive)
"https://nytimes.com", // Target URL
);
console.log(mementos);
```
#### Example Output
```javascript
[
'19961112181513', '19961121230155', '19961219002950', '19961220073509',
'19961226135029', '19961228014508', '19961230230427', '19970209220858',
'19970303103041', '19970414192930', '19970414210143', '19970415180120',
... 688983 more items
]
```
### 2. Analyse An Archived Web Page (`analyseMemento`)
Analyse the archived snapshot of https://nytimes.com, November 12, 1996, from the Internet Archive [(๐ = ia)](#supported-web-archives).
> [!TIP]
> If you provide a full 14-digit datetime (`YYYYMMDDHHMMSS`) using [`getMementos`](#1-fetch-available-memento-datetimes-getmementos), Wasteback Machine skips the TimeGate (URI-G) lookup, improving performance.
```javascript
import { analyseMemento } from "@overbrowsing/wasteback-machine";
const mementoData = await analyseMemento(
"ia", // Web archive ID (๐ = ia, Internet Archive)
"https://nytimes.com", // Target URL
"19961112", // Target memento-datetime (YYYYMMDDhhmmss); minimum input: YYYY
{ includeResources: true } // Resource list (true/false)
);
console.log(mementoData);
```
#### Example Output
```javascript
{
target: {
url: 'https://nytimes.com',
datetime: '19961112'
},
memento: {
url: 'https://web.archive.org/web/19961112181513if_/https://nytimes.com',
datetime: '19961112181513',
},
archive: {
name: 'Internet Archive (Wayback Machine)',
organisation: 'Internet Archive',
country: 'United States of America',
continent: 'North America',
url: 'https://web.archive.org',
},
sizes: {
html: { bytes: 1653, count: 1 },
stylesheet: { bytes: 0, count: 0 },
script: { bytes: 0, count: 0 },
image: { bytes: 46226, count: 2 },
video: { bytes: 0, count: 0 },
audio: { bytes: 0, count: 0 },
font: { bytes: 0, count: 0 },
flash: { bytes: 0, count: 0 },
plugin: { bytes: 0, count: 0 },
data: { bytes: 0, count: 0 },
document: { bytes: 0, count: 0 },
other: { bytes: 0, count: 0 },
total: { bytes: 47879, count: 3 }
},
completeness: '100%',
resources: [
{
url: 'https://web.archive.org/web/19961112181513im_/http://www.nytimes.com/index.gif',
type: 'image',
size: 45259
},
{
url: 'https://web.archive.org/web/19961112181513im_/http://www.nytimes.com/free-images/marker.gif',
type: 'image',
size: 967
}
]
}
```
## Supported Web Archives
Each supported web archive has a unique web archive ID (๐) required for API calls. The table also indicates which functions each archive supports.
| Web Archive | Organisation | ๐ | [`getMementos`](#1-fetch-available-memento-datetimes-getmementos) | [`analyseMemento`](#2-analyse-an-archived-web-page-analysememento) |
|--------------------------------------------------------------------------------------------------------------------------------------------|---------------------------------------------------|---------------------------------------------|------------------------------------------------------------------ |--------------------------------------------------------------------|
| [Arquivo.pt](https://arquivo.pt) | ๐ต๐น FCCN/FCT | [arq](/src/archives/arq/arq.js) | โ | โ |
| [National Library and Archives of Quebec (BAnQ) Web Archiving](https://www2.banq.qc.ca/collections/collections_patrimoniales/archives_web) | ๐จ๐ฆ National Library and Archives of Quebec (BAnQ) | [banq](/src/archives/banq/banq.js) | โ | โ |
| [Columbia University Libraries Web Archives](https://library.columbia.edu/collections/web-archives) | ๐บ๐ธ Columbia University Libraries | [cul](/src/archives/cul/cul.js) | โ | โ |
| [Webarchiv](https://webarchiv.cz) | ๐จ๐ฟ National Library of the Czech Republic | [cz](/src/archives/cz/cz.js) | โ | โ |
| [European Union Web Archive](https://op.europa.eu/en/web/euwebarchive) | ๐ช๐บ European Union | [euwa](/src/archives/euwa/euwa.js) | โ | โ |
| [Estonian Web Archive](https://veebiarhiiv.digar.ee) | ๐ช๐ช National Library of Estonia | [ewa](/src/archives/ewa/ewa.js) | โ | โ |
| [Government of Canada Web Archive](https://webarchiveweb.bac-lac.canada.ca) | ๐จ๐ฆ Library and Archives Canada | [gcwa](/src/archives/gcwa/gcwa.js) | โ | โ |
| [Croatian Web Archives (HAW)](https://haw.nsk.hr) | ๐ญ๐ท National and University Library in Zagreb | [haw](/src/archives/haw/haw.js) | โ | โ |
| [Internet Archive (Wayback Machine)](https://web.archive.org) | ๐บ๐ธ Internet Archive | [ia](/src/archives/ia/ia.js) | โ | โ |
| [Icelandic Web Archive (Vefsafn.is)](https://vefsafn.is) | ๐ฎ๐ธ National and University Library of Iceland | [iwa](/src/archives/iwa/iwa.js) | โ | โ |
| [Library of Congress Web Archive](https://loc.gov/web-archives) | ๐บ๐ธ Library of Congress | [loc](/src/archives/loc/loc.js) | โ | โ |
| [National Library of Ireland Web Archive](https://nli.ie/collections/our-collections/web-archive) | ๐ฎ๐ช National Library of Ireland | [nliwa](/src/archives/nliwa/nliwa.js) | โ | โ |
| [National Library of Medicine](https://archive-it.org/organizations/350) | ๐บ๐ธ National Library of Medicine | [nlm](/src/archives/nlm/nlm.js) | โ | โ |
| [National Records of Scotland Web Archive](https://webarchive.nrscotland.gov.uk) | ๐ด๓ ง๓ ข๓ ณ๓ ฃ๓ ด๓ ฟ National Records of Scotland | [nrs](/src/archives/nrs/nrs.js) | โ | โ |
| [Norwegian Web Archive](https://nb.no/en/collection/web-archive) | ๐ณ๐ด National Library of Norway | [nwa](/src/archives/nwa/nwa.js) | โ | โ |
| [New Zealand Web Archive](https://webarchive.natlib.govt.nz) | ๐ณ๐ฟ National Library of New Zealand | [nzwa](/src/archives/nzwa/nzwa.js) | โ | โ |
| [The Web Archive of Catalonia (Padicat)](https://padicat.cat) | ๐ช๐ธ Library of Catalonia | [padicat](/src/archives/padicat/padicat.js) | โ | โ |
| [PRONI Web Archive](https://webarchive.proni.gov.uk) | ๐ฌ๐ง The Public Record Office of Northern Ireland | [proni](/src/archives/proni/proni.js) | โ | โ |
| [Smithsonian Institution Archives](https://siarchives.si.edu) | ๐บ๐ธ Smithsonian Libraries and Archives | [sia](/src/archives/sia/sia.js) | โ | โ |
| [Spletni Arhiv](https://arhiv.nuk.uni-lj.si) | ๐ธ๐ฎ National and University Library of Slovenia | [slo](/src/archives/slo/slo.js) | โ | โ |
| [Australia Web Archive (Trove)](https://webarchive.nla.gov.au) | ๐ฆ๐บ National Library of Australia | [trove](/src/archives/trove/trove.js) | โ | โ |
| [UK Government Web Archive (UKGWA)](https://nationalarchives.gov.uk/webarchive) | ๐ฌ๐ง The National Archives | [ukgwa](/src/archives/ukgwa/ukgwa.js) | โ | โ |
| [University of North Texas Web Archives](https://digital.library.unt.edu/explore/collections/untweb) | ๐บ๐ธ University of North Texas University Libraries | [untwa](/src/archives/untwa/untwa.js) | โ | โ |
| [York University Digital Library](https://digital.library.yorku.ca) | ๐จ๐ฆ York University Libraries | [yudl](/src/archives/yudl/yudl.js) | โ | โ |
### Adding Web Archives
Wasteback Machine can support additional web archives if they meet the following criteria:
1. Provide a CDX server API (required for [`getMementos`](#1-fetch-available-memento-datetimes-getmementos)).
2. Support the [Memento Protocol (RFC7089)](https://datatracker.ietf.org/doc/html/rfc7089) (required for [`analyseMemento`](#2-analyse-an-archived-web-page-analysememento)).
3. Support replay state modifiers ([URL Rewrite Type Modifier](https://pywb.readthedocs.io/en/master/manual/rewriter.html?highlight=modifiers#url-rewrite-type-modifier)) endpoints for both:
- Raw content ([see example](https://web.archive.org/web/20131001001332id_/https://www.bbc.co.uk)).
- Navigational toolbars suppressed ([see example](https://web.archive.org/web/20131001001332if_/https://www.bbc.co.uk)).
To request support for an archive that meets these criteria, [submit an issue](https://github.com/overbrowsing/wasteback-machine/issues/new?template=add-new-web-archive.md) using the template.
## CLI
Wasteback Machine CLI lets you analyse an archived web page to view its size, composition, and estimated emissions using [CO2.js](https://developers.thegreenwebfoundation.org/co2js/overview) and the [Sustainable Web Design Model](https://sustainablewebdesign.org/estimating-digital-emissions).
### Quick Start
After [installation](#installation), start the CLI:
```sh
npx cli
```
### CLI Prompts
```sh
1. Enter web archive ID ('help' to list archives or [Enter โต] = Internet Archive):
2. Enter URL to analyse:
3. Enter target year (YYYY):
4. Enter target month (MM or [Enter โต] = 01):
5. Enter target day (DD or [Enter โต] = 01):
```
#### Example Output
```sh
________________________________________________________
MEMENTO INFO
Memento URL: https://web.archive.org/web/19961112181513if_/https://nytimes.com
Web Archive: Internet Archive (Wayback Machine)
Organisation: Internet Archive
Website: https://web.archive.org
________________________________________________________
PAGE SIZE
Data: 46.76 KB
Emissions: 0.014 g COโe
Completeness: 100%
________________________________________________________
PAGE COMPOSITION
HTML
Count: 1
Data: 1653 bytes (3.5%)
Emissions: 0.000 g COโe
IMAGE
Count: 2
Data: 46226 bytes (96.5%)
Emissions: 0.013 g COโe
________________________________________________________
```
## Credits
Developed by the [Overbrowsing Research Group](https://overbrowsing.com) at the [Institute for Designย Informatics, The University of Edinburgh](https://designinformatics.org/portfolio-item/wasteback-machine), with support in part from the [European Association for Digital Humanities (EADH)](https://eadh.org).
## Citing
Results generated with Wasteback Machine may be freely cited, quoted, analysed, or republished with attribution to 'Wasteback Machine'. No special permission is required for academic, journalistic, or personal use.
A publication related to this project appeared in the Proceedings of iConference 2026 ([view PDF](https://publicera.kb.se/ir/article/view/64185/51902)). Please cite as:
> Mahoney, D. (2026). Wasteback Machine: a method for quantitative measurement of the archived web. Information Research an International Electronic Journal, 31 (iConf), 448โ464. https://doi.org/10.47989/ir31iConf64185
```bib
@article{Mahoney_2026,
author = {Mahoney, David},
title = {Wasteback Machine: a method for quantitative measurement of the archived web},
journal = {Information Research: An International Electronic Journal},
volume = {31},
number = {iConf},
pages = {448-464},
year = {2026},
month = {Mar},
url = {https://publicera.kb.se/ir/article/view/64185},
doi = {10.47989/ir31iConf64185}
}
```
## Licenses
Wasteback Machine is licensed under [Apache 2.0](https://tldrlegal.com/license/apache-license-2.0-(apache-2.0)). For full licensing details, see the [LICENSE](/LICENSE) file.
Use of Wasteback Machine is subject to the terms, policies and licenses of each respective [supported web archive](#supported-web-archives).
## Terms
All results generated by Wasteback Machine are provided "as-is" without warranties of any kind, express or implied, including but not limited to accuracy, completeness, or reliability. The authors and contributors accept no liability for any errors, omissions, or consequences arising from the use of this software or the results it produces.