{"id":27629476,"url":"https://github.com/indexzero/_all_docs","last_synced_at":"2026-02-02T07:57:29.976Z","repository":{"id":280700516,"uuid":"939748074","full_name":"indexzero/_all_docs","owner":"indexzero","description":null,"archived":false,"fork":false,"pushed_at":"2025-04-22T16:13:17.000Z","size":116,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-04-22T17:32:30.009Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"JavaScript","has_issues":false,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/indexzero.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2025-02-27T03:29:13.000Z","updated_at":"2025-04-03T18:46:16.000Z","dependencies_parsed_at":"2025-03-04T21:27:56.308Z","dependency_job_id":"63bd6610-31d1-4e93-be20-1c8baff163cb","html_url":"https://github.com/indexzero/_all_docs","commit_stats":null,"previous_names":["indexzero/_all_docs"],"tags_count":3,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/indexzero%2F_all_docs","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/indexzero%2F_all_docs/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/indexzero%2F_all_docs/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/indexzero%2F_all_docs/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/indexzero","download_url":"https://codeload.github.com/indexzero/_all_docs/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":250289551,"owners_count":21406001,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2025-04-23T15:17:35.855Z","updated_at":"2026-02-02T07:57:29.970Z","avatar_url":"https://github.com/indexzero.png","language":"JavaScript","funding_links":[],"categories":[],"sub_categories":[],"readme":"# @_all_docs\n\n\u003e Stability: NaN – `Array(16).join(\"wat\" - 1) + \" Batman!\"`\n\nFetch \u0026 cache :origin/_all_docs using a set of lexographically sorted keys. High-performance, partition-tolerant system for fetching and caching npm registry data at scale\n\n**[Quick Start](#quick-start)**\n·\n**[Features](#features)**\n·\n**[Documentation](#-more-documentation)**\n·\n**[Architecture](#architecture)**\n·\n**[Contributing](#contributing)**\n\n## Quick Start\n\n```bash\n# Install the CLI globally\nnpm install -g @_all_docs/cli\n\n# Fetch npm registry partitions\nnpx _all_docs partition refresh --pivots ./pivots.js\n\n# Fetch package documents\nnpx _all_docs packument fetch express\n```\n\n## Features\n\n* 🛋️ Relax! Use the `start_key` and `end_key` CouchDB APIs to harness the power of partition-tolerance from the b-tree\n* 🔑 Accepts a set of lexographically sorted pivots to use as B-tree partitions\n* 🦿 Run map-reduce operations on `_all_docs` and `packument` entries by key range or cache partition\n* 🏁 Checkpoint system tracks processing progress across partition sets\n* ☁️ Parallel processing across multiple edge runtimes\n* 🔜 ~🕸️⚡️🐢🦎🦀 Lightning fast partition-tolerant edge read-replica for `cache-control: immutable` \"Pouch-like\" `[{ _id, _rev, ...doc }*]` JSON documents out of the box!~\n\n## Usage\n\n### Create Partition Pivots\n\n```javascript\n// pivots.js\nmodule.exports = [\n  '0', '1', '2', '3', '4', '5', '6', '7', '8', '9',\n  'a', 'b', 'c', 'd', 'e', 'f', 'g', 'h', 'i', 'j',\n  'k', 'l', 'm', 'n', 'o', 'p', 'q', 'r', 's', 't',\n  'u', 'v', 'w', 'x', 'y', 'z'\n];\n```\n\n### Fetch Registry Data (from CLI)\n\n```bash\n# Refresh all partitions\nnpx _all_docs partition refresh --pivots ./pivots.js\n\n# Fetch specific packages\nnpx _all_docs packument fetch express react vue\n\n# Fetch packages from a file (with automatic checkpoint/resume)\nnpx _all_docs packument fetch-list ./packages.json\n\n# Create cache index\nnpx _all_docs cache create-index \u003e index.txt\n```\n\n### Bulk Fetch with Checkpoints\n\nThe `packument fetch-list` command fetches packuments from a JSON array or newline-delimited text file. Checkpoints are enabled by default, making large fetches resumable:\n\n```bash\n# Fetch from JSON array (checkpoint enabled by default)\nnpx _all_docs packument fetch-list ./packages.json\n\n# Check progress\nnpx _all_docs packument fetch-list ./packages.json --status\n\n# List any failed packages\nnpx _all_docs packument fetch-list ./packages.json --list-failed\n\n# Start fresh (delete existing checkpoint)\nnpx _all_docs packument fetch-list ./packages.json --fresh\n\n# Disable checkpoint for one-off fetches\nnpx _all_docs packument fetch-list ./packages.json --no-checkpoint\n```\n\nInput file formats:\n- **JSON array**: `[\"lodash\", \"express\", \"@babel/core\"]`\n- **Text file**: One package name per line, `#` comments supported\n\nCheckpoints track per-package progress and automatically resume on re-run. Failed packages retry up to 3 times. Progress saves every 100 packages and on Ctrl+C.\n\n### Fetch Registry Data (from code)\n\n```javascript\nimport { PartitionClient } from '@_all_docs/partition';\nimport { PackumentClient } from '@_all_docs/packument';\n\n// Fetch partition data\nconst partitionClient = new PartitionClient({\n  env: { RUNTIME: 'node', CACHE_DIR: './cache' }\n});\n\nconst partition = await partitionClient.request({\n  startKey: 'express',\n  endKey: 'express-z'\n});\n\n// Fetch package document\nconst packumentClient = new PackumentClient({\n  env: { RUNTIME: 'node', CACHE_DIR: './cache' }\n});\n\nconst packument = await packumentClient.request('express');\n```\n\n## 📚 More Documentation\n- [Getting Started Guide](./doc/getting-started.md) - Quick tutorial and common use cases\n- [Architecture Overview](./doc/architecture.md) - System design and technical details\n- [CLI Reference](./doc/cli-reference.md) - Complete command documentation\n- [API Reference](./doc/api.md) - Programmatic usage and package APIs\n\n## Development Setup\n\n```bash\n# Clone and install\ngit clone https://github.com/indexzero/_all_docs.git\ncd _all_docs\npnpm install\n\n# Run tests\npnpm test\n\n# Start development worker\npnpm dev\n```\n\n## License\n\nApache-2.0 © 2024 Charlie Robbins\n\n## Thanks\n\nMany thanks to [bmeck], [guybedford], [mylesborins], [mikeal], [jhs], [jchris], [darcyclarke], [isaacs], \u0026 [mcollina] for all the code, docs, \u0026 past conversations that contributed to this technique working so well, 10 years later ❤️\n\n[bmeck]: https://github.com/bmeck\n[guybedford]: https://github.com/guybedford\n[mylesborins]: https://github.com/mylesborins\n[mikeal]: https://github.com/mikeal\n[jhs]: https://github.com/jhs\n[jchris]: https://github.com/jchris\n[darcyclarke]: https://github.com/darcyclarke\n[isaacs]: https://github.com/isaacs\n[mcollina]: https://github.com/mcollina\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Findexzero%2F_all_docs","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Findexzero%2F_all_docs","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Findexzero%2F_all_docs/lists"}