{"id":51552811,"url":"https://github.com/akesson/htmlarc","last_synced_at":"2026-07-10T01:01:35.640Z","repository":{"id":362488423,"uuid":"1259248409","full_name":"akesson/htmlarc","owner":"akesson","description":null,"archived":false,"fork":false,"pushed_at":"2026-06-19T15:18:20.000Z","size":848,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2026-06-19T15:29:28.441Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Rust","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"agpl-3.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/akesson.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":"NOTICE","maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2026-06-04T10:20:27.000Z","updated_at":"2026-06-19T15:18:45.000Z","dependencies_parsed_at":null,"dependency_job_id":null,"html_url":"https://github.com/akesson/htmlarc","commit_stats":null,"previous_names":["akesson/htmlarc"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/akesson/htmlarc","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/akesson%2Fhtmlarc","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/akesson%2Fhtmlarc/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/akesson%2Fhtmlarc/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/akesson%2Fhtmlarc/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/akesson","download_url":"https://codeload.github.com/akesson/htmlarc/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/akesson%2Fhtmlarc/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":35316840,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-26T15:22:16.424Z","status":"online","status_checked_at":"2026-07-09T02:00:07.329Z","response_time":57,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2026-07-10T01:01:34.795Z","updated_at":"2026-07-10T01:01:35.618Z","avatar_url":"https://github.com/akesson.png","language":"Rust","funding_links":[],"categories":[],"sub_categories":[],"readme":"# htmlarc\n\nA toolkit for analysing **large corpora of HTML** efficiently. It stores many pre-parsed\nHTML documents in one file (`.htmlarc`) and lets you CSS-query them **without re-parsing** —\nuseful when you have gigabytes of HTML (e.g. a Wiktionary dump) split across hundreds of\nthousands of small files.\n\n\u003e [!NOTE]\n\u003e **Stable, but not under active development.** Dual-licensed: **[AGPL-3.0](LICENSE)** for\n\u003e open-source use, or a **[commercial license](COMMERCIAL.md)** to use it in closed-source or\n\u003e SaaS software without AGPL obligations. There is no free support or roadmap, but **I'm\n\u003e available for paid contract work** to extend, integrate, or maintain it for your use case —\n\u003e reach me at [@akesson](https://github.com/akesson).\n\n## Why\n\n- **No filesystem overhead.** Hundreds of thousands of tiny `.html` files waste a huge amount\n  of space and inodes, and are slow to traverse. One archive avoids all of that.\n- **No HTML parsing at query time.** Documents are parsed once into a compact, flat binary DOM\n  and stored with [rkyv]. A packed `.htmlarc` is queried **zero-copy** straight from a\n  memory-map — no HTML parse, and no per-node deserialization — so repeated CSS queries over the\n  whole corpus are fast. (Per-document **text** payloads are zstd-compressed and inflate lazily on\n  read; topology and selector matching stay zero-copy, so a pure selector sweep never decompresses.)\n- **A real CSS3 selector engine** runs over that DOM (compound/complex/relative selectors,\n  `:has()`, `nth-*`, attribute operators), at speeds comparable to a pointer-tree DOM.\n\n## Quick start\n\n```sh\ncargo build --release\n\n# Every command takes a \u003csource\u003e, which is one of:\n#   • a .htmlarc archive file   (loaded directly, no parsing)\n#   • a single .html/.htm file  (parsed into a one-entry archive)\n#   • a directory of *.html     (each parsed, keyed by file name)\n\n# Pack a folder of HTML into one archive (parse once):\nhtmlarc pack ./pages -o pages.htmlarc\n\n# List the documents (keys), optionally filtered:\nhtmlarc list pages.htmlarc\nhtmlarc list pages.htmlarc -i 'css: section \u003e h1'\nhtmlarc list page.html                     # a single loose file works too\n\n# Count CSS-selector matches across the whole corpus:\nhtmlarc probe pages.htmlarc -p 'section \u003e h1 =\u003e HtmlFmt[id][class^=mw]@words'\n\n# Diff two sources by per-document checksum (fast):\nhtmlarc diff old.htmlarc new.htmlarc\n```\n\nRun `htmlarc --help` (or `htmlarc \u003ccommand\u003e --help`) for the full options.\n\n## Ingesting documents (`htmlarc-convert`)\n\nThe workspace also ships `htmlarc-convert`, a companion that turns a **ZIM** file (the\nKiwix/Wikipedia offline format), a **WARC** web-crawl file (e.g. Common Crawl, plain or\n`.warc.gz`), or a **directory** of saved HTML files into a `.htmlarc` archive. The format is\ninferred from the input path (override with `--format zim|warc|dir`):\n\n```sh\n# Convert every HTML document from a source into one archive:\nhtmlarc-convert convert wikipedia.zim wikipedia.htmlarc\nhtmlarc-convert convert crawl.warc.gz crawl.htmlarc\nhtmlarc-convert convert ./saved-pages/ pages.htmlarc\n\n# Only convert documents whose key is in a list (one per line):\nhtmlarc-convert convert wikipedia.zim subset.htmlarc --list words.txt\n\n# Sample a huge input:\nhtmlarc-convert convert crawl.warc.gz sample.htmlarc --limit 50000\n\n# Inspect a source without converting:\nhtmlarc-convert list wikipedia.zim                 # one document key per line\nhtmlarc-convert extract wikipedia.zim 'Some Title' # print one document's HTML\n```\n\nThen query the result with `htmlarc` as usual (`htmlarc probe wikipedia.htmlarc -p '…'`).\n\n- ZIM is read via the pure-Rust [`zim`] crate (MIT/Apache), so **no system libzim** is\n  required — but building it does need a **C compiler** (`zstd-sys`/`lzma-sys` compile bundled\n  C) and pulls in ~110 transitive crates. This is isolated to the `htmlarc-convert` binary; the\n  three core `htmlarc-*` crates stay pure-Rust with no C dependencies.\n- We depend on a **fork** of `zim` 0.4: the upstream crate is unmaintained and fails to open\n  any current Kiwix dump (modern ZIMs omit the legacy title pointer list, which upstream slices\n  unconditionally → `OutOfBounds`). The fork guards that sentinel.\n- Document keys are the ZIM title (URL slug if the title is empty), the WARC `WARC-Target-URI`,\n  or the file path relative to the directory root. `extract` matches a key **exactly**.\n- The WARC reader currently loads selected documents into memory; for very large crawls use\n  `--limit`. (Parser error-recovery has since landed — see *Honest limitations* — so converting\n  general web HTML now drops only ~0.0005% of documents, all genuine capacity overflow rather than\n  malformed markup.)\n\n[`zim`]: https://github.com/akesson/zim\n\n## How it works\n\nThe DOM (`htmlarc-dom`) is **structure-of-arrays**: each node is a fixed-stride record in a\nsingle `Vec\u003cu8\u003e`, with parent/sibling/child links packed as adaptive `u16` or `u24` indices\n(17- or 22-byte records); text, attributes and classes are interned into side tables. Documents\nare built at `u24` width and **down-packed to `u16`** on save when they fit comfortably under\nthe `u16` ceiling. The whole thing derives `rkyv::Archive`, so it serializes to disk and is read\nback **without pointer fix-ups or per-node allocation** — traversal is array indexing into hot,\ncontiguous memory rather than chasing heap pointers.\n\nAn archive (`htmlarc-archive`) is a **bundle-segmented, footer-indexed** container: documents are\ngrouped into bundles (up to 1,000) and written with rkyv, with a sorted key index in the footer.\n`get` binary-searches that index by key, and `MmapArchive` reads the archived topology **zero-copy**\nfrom a memory-map. Each document's text/comment payload is relocated into a per-bundle block and\nstored as an independent zstd frame, inflated lazily per document on first text read — so\nselector/topology queries, which touch none of it, stay fully zero-copy.\n\n## Honest limitations\n\n- **~16.7M nodes per document.** Node links are packed as `u24` (the public `NodeIndex` is a\n  `u32`), so the per-document ceiling is `2^24 − 1` nodes. The design still targets many *small*\n  documents (dictionary entries, etc.); documents that fit are stored at half the link width\n  (`u16`).\n- **Spec-compliant tokenizer, pragmatic tree builder.** Tokenization uses [html5gum] (a WHATWG\n  spec-compliant tokenizer); the tree builder is an in-house stack-based builder with HTML5 **error\n  recovery** (stray/mismatched end tags, foreign SVG/MathML, optional tags) rather than the full\n  WHATWG tree-construction algorithm (no adoption-agency / foster-parenting). On a 2.04M-document\n  Common Crawl corpus it parses **99.9995%** of `text/html` documents; the ~0.0005% dropped are\n  genuine capacity overflow (the node ceiling above), not malformed markup.\n\n## Workspace layout\n\n```\ncrates/\n  htmlarc-dom/      flat, rkyv-archivable HTML DOM + parser + CSS3 selector engine\n  htmlarc-archive/  the single-file .htmlarc archive (build / open / query / diff)\ncli/\n  htmlarc/          the `htmlarc` binary (pack / list / probe / diff)\n  htmlarc-convert/  converts ZIM / WARC / a directory of HTML into a .htmlarc archive\n```\n\n## Building \u0026 testing\n\n```sh\ncargo build --workspace\ncargo nextest run            # tests use cargo-nextest (process-per-test)\n```\n\nTests are run with [cargo-nextest]. Some snapshot tests use insta's `glob!` over the same\nfixtures and rely on nextest's process-per-test isolation; running them with the default\n`cargo test` (in-process threads) can produce spurious failures.\n\n### Git hooks (one-time setup)\n\nThis repo ships git hooks in `.githooks/` that mirror the CI checks: `pre-commit` runs\n`cargo fmt --check` + clippy, and `pre-push` additionally runs the test suite (nextest).\nGit does **not** pick them up automatically — activate them once per clone with:\n\n```sh\nmake setup\n```\n\nThe other `make` targets mirror the CI jobs so you can reproduce a failure locally:\n`make fmt`, `make lint`, `make test`, `make bench`, or `make ci` for all of them. Run\n`make` to list them.\n\n## Example\n\nA runnable, end-to-end example of using htmlarc as a library — the same **step-wise reduction**\nthat powers a Wiktionary pipeline (raw MediaWiki page → cleaned `\u003chead\u003e` → article content only),\nwith a `.htmlarc` archive as the checkpoint between each pass:\n\n```sh\ncargo run -p htmlarc-archive --example wiktionary\n```\n\nIt parses real Swedish Wiktionary pages, mutates each DOM (strip chrome, lift the article body),\nre-packs each stage into its own archive, then reopens the result zero-copy and CSS-queries the\ndefinitions out — printing the size shrink at every step. See\n[`crates/htmlarc-archive/examples/wiktionary.rs`](crates/htmlarc-archive/examples/wiktionary.rs).\n\n## License\n\nDual-licensed:\n\n- **[GNU AGPL-3.0](LICENSE)** — free for open-source use. If you distribute htmlarc (or a work\n  based on it), or offer it to users over a network, the AGPL requires you to make the\n  corresponding source available under the AGPL.\n- **[Commercial license](COMMERCIAL.md)** — to use htmlarc in closed-source or SaaS software\n  without AGPL obligations. Contact [@akesson](https://github.com/akesson).\n\nTest/benchmark fixtures under `crates/htmlarc-dom/src/**` and `cli/htmlarc/src/testdata/` are\nreal Wikimedia (Wiktionary) pages, licensed CC BY-SA — see [NOTICE](NOTICE).\n\n[rkyv]: https://rkyv.org\n[cargo-nextest]: https://nexte.st\n[html5gum]: https://github.com/untitaker/html5gum\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fakesson%2Fhtmlarc","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fakesson%2Fhtmlarc","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fakesson%2Fhtmlarc/lists"}