{"id":51433620,"url":"https://github.com/ssskay/internet-historian","last_synced_at":"2026-07-05T06:00:27.351Z","repository":{"id":369180772,"uuid":"1288734323","full_name":"ssskay/internet-historian","owner":"ssskay","description":"Quietly preserve the web things you love, forever — a patient Internet-Archive-only web archiver for macOS, with a Claude Code skill. Built to save ちいかわ (Chiikawa) pages.","archived":false,"fork":false,"pushed_at":"2026-07-04T00:21:20.000Z","size":48,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2026-07-04T02:08:16.608Z","etag":null,"topics":["chiikawa","claude-code","digital-preservation","internet-archive","launchd","macos","wayback-machine","web-archiving"],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/ssskay.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2026-07-03T23:04:39.000Z","updated_at":"2026-07-04T00:21:24.000Z","dependencies_parsed_at":null,"dependency_job_id":null,"html_url":"https://github.com/ssskay/internet-historian","commit_stats":null,"previous_names":["ssskay/internet-historian"],"tags_count":null,"template":false,"template_full_name":null,"purl":"pkg:github/ssskay/internet-historian","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ssskay%2Finternet-historian","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ssskay%2Finternet-historian/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ssskay%2Finternet-historian/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ssskay%2Finternet-historian/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/ssskay","download_url":"https://codeload.github.com/ssskay/internet-historian/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ssskay%2Finternet-historian/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":35144845,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-26T15:22:16.424Z","status":"online","status_checked_at":"2026-07-05T02:00:06.290Z","response_time":100,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["chiikawa","claude-code","digital-preservation","internet-archive","launchd","macos","wayback-machine","web-archiving"],"created_at":"2026-07-05T06:00:23.240Z","updated_at":"2026-07-05T06:00:26.458Z","avatar_url":"https://github.com/ssskay.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Internet Historian 🏛️\n\n[![tests](https://github.com/ssskay/internet-historian/actions/workflows/test.yml/badge.svg)](https://github.com/ssskay/internet-historian/actions/workflows/test.yml)\n\u003c!-- TODO: once published to PyPI, add PyPI version + supported-Python-versions badges here:\n     [![PyPI](https://img.shields.io/pypi/v/internet-historian.svg)](https://pypi.org/project/internet-historian/)\n     [![Python versions](https://img.shields.io/pypi/pyversions/internet-historian.svg)](https://pypi.org/project/internet-historian/) --\u003e\n\n\n**Quietly preserve the web things you love, forever.**\n\nInternet Historian is a tiny, patient tool that saves the web pages you care about into the\n[Internet Archive's Wayback Machine](https://web.archive.org/) — and keeps trying until each\none is safely preserved. You hand it URLs (or just name a thing you love); it runs quietly in\nthe background on your Mac and makes sure they don't disappear.\n\nIt was built for one purpose: **saving [ちいかわ (Chiikawa)](https://en.wikipedia.org/wiki/Chiikawa)\npages before they vanish** — official sites, the anime, the shops, the wikis. But it works for\nanything: a band, a webcomic, a favorite blog, a fandom, a single irreplaceable page.\n\n\u003e It optimizes for **never losing a URL**, not for speed. The Wayback Machine throttles when\n\u003e it's busy — Internet Historian treats that as normal weather, waits, and tries again. And\n\u003e again. For as long as it takes.\n\n---\n\n## Why this exists\n\nWeb pages die. Fan sites go offline, shops close, links rot. The Internet Archive can preserve\nalmost any public page — but only if someone asks it to, at the right time, and keeps asking\nwhen the Archive is too busy to answer. That \"keep patiently asking\" part is tedious to do by\nhand. Internet Historian does it for you, forever, in the background.\n\n- 🗃️ **A patient queue, not a scraper.** Add URLs once; it preserves them and remembers what's done.\n- 🔁 **Throttle-proof.** Built around the Internet Archive's real rate limits. Being throttled is expected, not an error.\n- 💀 **Knows dead from busy.** A 404 or a vanished domain gets flagged as dead — but only after real confirmation. A busy Archive never counts against a page.\n- 🧠 **Talks to you (optionally).** Ships with a [Claude Code](https://claude.ai/code) skill so you can just say *\"archive Chiikawa stuff\"* and it finds the real pages itself.\n- 🔒 **Your keys stay yours.** API keys live in your macOS Keychain, never in this repo.\n\n## Requirements\n\n- **macOS** (uses the Keychain and `launchd`) — see [Windows / Linux](#windows--linux) below.\n- **Python 3.11+** (for `tomllib`). `python3 --version` to check.\n- A **free [archive.org](https://archive.org) account** and its S3-style API keys (setup walks you through this).\n- One dependency: [`requests`](https://pypi.org/project/requests/).\n\n## Quickstart\n\nInstall with [pipx](https://pipx.pypa.io/) — it keeps the tool isolated and puts the\n`internet-historian` command on your PATH:\n\n```bash\npipx install internet-historian\ninternet-historian setup\n```\n\n(Plain `pip install internet-historian` works too, e.g. inside a virtualenv.)\n\n`setup` will walk you through getting your free API keys (it opens the right page), store them\nin your Keychain, verify them, and install the background job that does the archiving. That's it.\n\n`setup` gets you to the finish line. You never have to run anything on a schedule — a `launchd`\njob wakes up every 10 minutes, preserves whatever's queued, and goes back to sleep.\n\n## Use it: just talk to it\n\nThe nicest way to run Internet Historian is through its [Claude Code](https://claude.ai/code)\nskill — no commands to remember. Install it once:\n\n```bash\ninternet-historian install-skill\n```\n\nThen, in any Claude Code session, just say what you want preserved:\n\n\u003e **You:** archive Chiikawa stuff\n\u003e\n\u003e **Claude:** *searches the web for the real, official ちいかわ pages, checks they're live,\n\u003e shows you the list, and queues them to a `chiikawa` collection — then the background job\n\u003e preserves them, patiently, on its own.*\n\n\u003e **You:** how's my archive doing?\n\u003e\n\u003e **Claude:** *reads back what's preserved, what's still queued, and anything that's stuck.*\n\n\u003e **You:** why hasn't that shop page saved yet?\n\u003e\n\u003e **Claude:** *tells you plainly: just throttled by a busy Archive (leave it — it'll retry) or a\n\u003e genuinely dead link.*\n\nYou name a thing; it finds the **real, primary pages** (official sites, Wikipedia, press) over\nfan wikis unless you ask otherwise, confirms the list with you, and queues them. That's the whole\npoint — you become a proper internet historian without lifting a finger.\n\n## Prefer the terminal?\n\nEvery skill action is just a CLI call you can run yourself:\n\n```bash\ninternet-historian add https://www.anime-chiikawa.jp/ --collection chiikawa\ninternet-historian status\n```\n\n(A shorter `historian` alias is installed too.)\n\n| Command | What it does |\n|---------|--------------|\n| `setup` | Connect your Archive account + install the background job (run once) |\n| `add URL [URL ...] [--collection NAME]` | Queue URLs for preservation |\n| `add --file urls.txt [--collection NAME]` | Queue a whole text file of URLs |\n| `status` | See what's preserved, queued, or dead |\n| `diagnose` | Plain-English \"why isn't this archived yet?\" — throttled vs. genuinely dead |\n| `pause` / `resume` | Stop/restart a single URL or a whole `--collection` |\n| `check` | Raw Internet Archive capacity right now |\n\nCollections are just tags — group your Chiikawa pages, your webcomics, your blogs. No setup needed.\n\n## How it works (the 60-second version)\n\n```\n  you ──add──▶ queue.db (SQLite) ◀──drain── launchd (every 10 min)\n                                              │\n                                              ▼\n                                   Internet Archive \"Save Page Now\"\n```\n\n- **`historian.py`** is the whole engine: a queue + an Internet Archive client, in one file.\n- **`queue.db`** remembers every URL and its state (queued → submitted → archived / dead).\n- **`launchd`** is the heartbeat: it runs `internet-historian drain` on a timer so you don't have to.\n- Captures use server-side dedup (`if_not_archived_within=30d`), so already-saved pages aren't\n  needlessly recaptured — they're just recorded as preserved.\n\nFailures are classified carefully: rate-limits and timeouts retry forever with backoff; only a\npage whose own server keeps answering badly (404, dead DNS, blocked) is marked dead, and only\nafter 3 confirmations spaced a day apart. Being throttled by a busy Archive **never** kills a URL.\n\n## What it does NOT do (by design, for now)\n\n- **One backend: the Internet Archive.** No archive.today, no local copies yet. (Local\n  snapshots via [`monolith`](https://github.com/Y2Z/monolith) are a planned future backend.)\n- **No auto-discovery.** You add pages (or ask the skill to find them); it doesn't crawl the\n  web hunting for new ones on its own.\n- **Social media is best-effort.** X/Twitter and Instagram hide content behind login walls, so\n  the Archive often captures a login page instead. Internet Historian will still try, but flags\n  these — it's expected, not a bug.\n\n## Related projects\n\nInternet Historian's niche is being a **persistent, patient background queue** with a\n**conversational control surface**: you add things once (or just ask the skill), and it keeps\nworking for weeks without you. Other excellent tools in this space, and how they differ:\n\n- **[agude/wayback-machine-archiver](https://github.com/agude/wayback-machine-archiver)** — submits the URLs in a sitemap to the Wayback Machine. A one-shot batch script; no persistent queue, backoff, or state that survives between runs.\n- **[overcast07/wayback-machine-spn-scripts](https://github.com/overcast07/wayback-machine-spn-scripts)** — capable Bash scripts around Save Page Now (outlinks, retries, cross-platform). You run and babysit a session; Internet Historian runs itself indefinitely via `launchd`.\n- **[Mearman/mcp-wayback-machine](https://github.com/Mearman/mcp-wayback-machine)** — an MCP server exposing archive/retrieve/search as on-demand tool calls. Ideal for one-off actions inside an agent; it has no durable queue that keeps retrying throttled captures for you.\n- **[internetarchive/internet-archive-skills](https://github.com/internetarchive/internet-archive-skills)** — the Internet Archive's official Claude Code skill for uploading, downloading, and searching archive.org *items*. Complementary: it manages Archive items, not patient web-page (SPN2) capture with backoff.\n- **[bellingcat/auto-archiver](https://github.com/bellingcat/auto-archiver)** — heavy-duty archiving of links (including media) from a spreadsheet to many backends, built for OSINT/evidence workflows. Far more powerful, and far more to set up; Internet Historian is deliberately tiny, IA-only, and personal.\n\n## Windows / Linux\n\nNot yet — but it's close. The **engine** (`historian.py`: the queue, the SQLite store, the\nArchive client) is pure Python and already cross-platform. Only **two** pieces are macOS-specific:\n\n1. **Key storage** uses the macOS Keychain (via the `security` command). *There's already an\n   env-var fallback* — set `IA_ACCESS_KEY` / `IA_SECRET_KEY` and the engine works anywhere.\n2. **The background heartbeat** uses `launchd`. On Linux you'd use a systemd timer or cron; on\n   Windows, Task Scheduler — each just needs to run `internet-historian drain`\n   (equivalently `python -m historian drain`) on a timer.\n\nPorting means swapping those two. **Contributions very welcome** — that's the whole to-do list.\n\n## Configuration\n\nKnobs live in [`config.toml`](config.toml) (no secrets there). Sensible defaults ship in the\nbox: 10-minute drain interval, 2 capture slots left free for your own browsing, 30-day dedup\nwindow, and a conservative per-URL daily attempt cap. Tweak if you like; the defaults are fine.\n\n**Periodic recapture (opt-in).** By default a page is archived once and then left alone. To keep\na collection *fresh* — re-snapshotting pages that change over time, like a shop or a news hub —\ngive it a refresh window:\n\n```toml\n[collections]\nchiikawa = { refresh_days = 30 }\n```\n\nAny archived page in that collection older than `refresh_days` is quietly re-queued on the next\ndrain; its last good snapshot stays on record until a new one lands. Collections you don't list\nnever recapture. The Internet Archive's own 30-day dedup still applies, so a very short\n`refresh_days` just re-confirms the existing snapshot rather than spamming captures.\n\n## License\n\n[MIT](LICENSE) — do what you like. Preserve the web things *you* love.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fssskay%2Finternet-historian","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fssskay%2Finternet-historian","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fssskay%2Finternet-historian/lists"}