{"id":51440070,"url":"https://github.com/alexstep/doc-extract","last_synced_at":"2026-07-05T10:30:46.251Z","repository":{"id":361390523,"uuid":"1253889049","full_name":"alexstep/doc-extract","owner":"alexstep","description":"Fast native text extraction from PDF, Office, EPUB \u0026 more. Rust N-API for Node.js and Bun","archived":false,"fork":false,"pushed_at":"2026-05-30T12:29:20.000Z","size":273,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2026-05-30T13:14:03.316Z","etag":null,"topics":["ai-preprocessing","docparser","textract"],"latest_commit_sha":null,"homepage":"","language":"Rust","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/alexstep.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2026-05-29T23:12:25.000Z","updated_at":"2026-05-30T12:30:04.000Z","dependencies_parsed_at":null,"dependency_job_id":null,"html_url":"https://github.com/alexstep/doc-extract","commit_stats":null,"previous_names":["alexstep/doc-extract"],"tags_count":3,"template":false,"template_full_name":null,"purl":"pkg:github/alexstep/doc-extract","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/alexstep%2Fdoc-extract","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/alexstep%2Fdoc-extract/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/alexstep%2Fdoc-extract/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/alexstep%2Fdoc-extract/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/alexstep","download_url":"https://codeload.github.com/alexstep/doc-extract/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/alexstep%2Fdoc-extract/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":35151638,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-26T15:22:16.424Z","status":"online","status_checked_at":"2026-07-05T02:00:06.290Z","response_time":100,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["ai-preprocessing","docparser","textract"],"created_at":"2026-07-05T10:30:46.165Z","updated_at":"2026-07-05T10:30:46.228Z","avatar_url":"https://github.com/alexstep.png","language":"Rust","funding_links":[],"categories":[],"sub_categories":[],"readme":"# doc-extract\n\n**Text extraction library for RAG pipelines, LLM apps, and AI agents** — turn uploaded documents into clean plain text before chunking, embedding, or indexing.\n\nNative addon for **Node.js** and **Bun** (Rust + N-API): fast, in-process, no subprocesses. PDF, Office, EPUB, calendars, contacts, Apple Wallet passes, and more.\n\n## Why doc-extract\n\n- **Fast** - Rust parsers run natively\n- **Small footprint** - ~6 MB native addon per platform\n- **Non-blocking** - `extractText()` returns a Promise; CPU work runs on a Rust thread pool\n- **Backpressure** - built-in concurrency limit (default 32) via `setMaxConcurrent`\n- **Flexible limits** - global, per-instance, or per-call file size in megabytes\n- **Large files** - file paths and ZIP-based formats read from disk; buffers above threshold auto-spill to temp\n- **No subprocesses** - PDF and Office handled in-process\n\n## Install\n\n```bash\nnpm install @alexstep/doc-extract\n# or\nbun add @alexstep/doc-extract\n```\n\nRequires **Node.js ≥ 18** or **Bun ≥ 1.3**.\n\n## Quick start\n\n```javascript\nimport docExtract from '@alexstep/doc-extract'\n\n// Global defaults\ndocExtract.setMaxConcurrent(4)\ndocExtract.setMaxFilesizeMB(42)\ndocExtract.setInMemoryThresholdMB(64)\n\nconst text = await docExtract.extractText('./report.pdf')\nconst fromUrl = await docExtract.extractText('https://example.com/file.pdf')\nconst pass = await docExtract.parsePkPass('./ticket.pkpass')\n\n// Isolated instance\nconst custom = new docExtract({ maxConcurrent: 4, maxFileSizeMB: 200 })\n\n// Buffer: auto-detect by magic bytes (%PDF, ZIP/docx, etc.)\nconst buffer = await Bun.file('./report.pdf').bytes()\nconst doc = await custom.extractText(Buffer.from(buffer))\n\n// Explicit format when magic is ambiguous (e.g. plain CSV bytes)\nconst csv = await custom.extractText(someBuffer, 'csv')\n```\n\n### Auto-detect\n\n| Input | Hint source |\n|-------|-------------|\n| File path | extension hint + magic bytes from file head |\n| URL | extension from pathname + magic bytes |\n| `Buffer` | **magic bytes only** (no filename) |\n\nWorks without a second argument for PDF (`%PDF`), Office ZIP (docx/xlsx/pptx), ICS/VCF, JSON, HTML, and similar.\n\nFor buffers, pass `format` explicitly when the content has no clear signature (e.g. legacy `.doc`, ambiguous plain text):\n\n```javascript\nawait docExtract.extractText(buffer, 'docx')\nawait docExtract.extractText(buffer, { format: 'pdf', debug: true })\nawait docExtract.extractText(buffer, { unknown: 'reject' }) // strict: no text heuristic\n```\n\n### Unknown content policy\n\nWhen format cannot be determined confidently:\n\n| `unknown` | Behavior |\n|-----------|----------|\n| `text-if-likely` (default) | Treat as `txt` only if bytes look like text (UTF-8/UTF-16/BOM, low control-byte ratio) |\n| `reject` | Return `\"\"` (unsupported) |\n| `text-lossy` | Try `txt` unless bytes are obviously binary |\n\n**Detection** uses explicit `format` first, then combines extension hints and magic bytes. Magic bytes override conflicting extensions when possible (e.g. `%PDF` vs a `.zip` path, ICS content vs a `.txt` name). Unknown bytes fall back according to `unknown` policy.\n\nPath-based text heuristics read up to **32 KB** of file head; magic/ZIP sniffing uses the first **4 KB**.\n\n## API\n\n| Method | Description |\n|--------|-------------|\n| `docExtract.extractText(input, format?)` | Extract text. `input` = `Buffer`, file path, or `http(s)://` URL. Optional `format` or `{ format, unknown, maxFileSizeMB, inMemoryThresholdMB, tempDir }`. |\n| `docExtract.parsePkPass(input, options?)` | Parse `.pkpass` → `{ pass, localization?, stripImage? }`. |\n| `docExtract.setMaxConcurrent(n)` | Global parallel parse limit (`n === 0` or negative = no-op). |\n| `docExtract.setMaxFilesizeMB(n)` | Global max input size in MB (default **42**). `0` = unlimited. |\n| `docExtract.setInMemoryThresholdMB(n)` | Above this size, paths/URLs skip JS heap; buffers spill to temp (default **64**). |\n| `docExtract.setMaxWorkingSetMB(n)` | Optional cap on total in-flight parse memory (`0` = disabled). |\n| `docExtract.setTempDir(dir)` | Directory for auto-spill temp files. |\n| `new docExtract({ maxConcurrent, maxFileSizeMB, inMemoryThresholdMB, tempDir, debug })` | Instance with its own limits and optional debug logging. |\n\n**Environment:** `DOCEXTRACT_MAX_CONCURRENT`, `DOCEXTRACT_MAX_FILESIZE_MB`, `DOCEXTRACT_MAX_BYTES`, `DOCEXTRACT_IN_MEMORY_THRESHOLD_MB`, `DOCEXTRACT_MAX_WORKING_SET_MB`, `DOCEXTRACT_TMPDIR`, `DOCEXTRACT_DEBUG`.\n\n### Error handling\n\n`extractText` resolves to a string — no `try/catch` needed for content issues:\n\n| Situation | Result |\n|-----------|--------|\n| Empty PDF (image scan), unsupported format, parse failure | `\"\"` |\n| File too large, missing file, URL fetch error | **throws** — use `.catch()` |\n\n```javascript\nconst text = await docExtract.extractText('./scan.pdf') // \"\" if image-only PDF\n\nawait docExtract.extractText('https://example.com/huge.pdf').catch((err) =\u003e {\n  // size limit, network, missing file\n})\n\nawait docExtract.extractText('./scan.pdf', { debug: true })\n// console.debug: PDF has no text layer (likely image-only scan)\n```\n\n`parsePkPass` still returns `null` for invalid passes (unchanged).\n\n### Large files\n\ndoc-extract is built for batch imports and server-side pipelines, not only small uploads.\n\n| Input | Behavior |\n|-------|----------|\n| **File path** | Rust opens the file directly; ZIP formats (docx, xlsx, pptx, epub, odt, pkpass) read entries from disk |\n| **URL** | Streamed to a temp file, then parsed via path pipeline |\n| **Buffer ≤ threshold** | Passed to native code in memory (default threshold 64 MB) |\n| **Buffer \u003e threshold** | Auto-written to temp, parsed from disk, temp removed |\n\n**Policy vs memory:**\n\n- `maxFileSizeMB` — reject files larger than N (`0` = no limit)\n- `inMemoryThresholdMB` — when to avoid keeping full payload in the JS heap\n\n**Batch tip:** for many large files (e.g. 20 × 200 MB), lower `maxConcurrent` (2–4) and optionally set `setMaxWorkingSetMB` to cap total in-flight memory. Peak RAM for ZIP formats is roughly `concurrency × parser working set`, not `concurrency × file size`.\n\n### pkpass\n\n```javascript\nconst text = await docExtract.extractText('./ticket.pkpass') // formatted text for AI\nconst json = await docExtract.parsePkPass('./ticket.pkpass') // structured pass.json\n```\n\n## Supported formats\n\n| Group | Extensions |\n|-------|------------|\n| Office | `pdf`, `docx`, `docm`, `xlsx`, `xls`, `ods`, `pptx`, `pptm`, `odt`, `rtf` |\n| Books | `epub`, `fb2` |\n| Calendar / contacts | `ics`, `ifb`, `ical`, `vcf`, `vcard` |\n| Data | `json`, `jsonl`, `ndjson`, `csv`, `tsv` |\n| Web / text | `html`, `htm`, `xhtml`, `xml`, `txt`, `md`, `markdown`, `log` |\n| Wallet | `pkpass` (auto-detected from ZIP + `pass.json`) |\n\n## Limitations \u0026 alternatives\n\ndoc-extract targets **in-process text extraction**: text-layer PDF, Office Open XML, EPUB, calendars, and similar. No OCR, no subprocesses, no Docker sidecar.\n\n**Not supported (returns `\"\"` or needs another tool):**\n\n- Image-only / scanned PDF (no text layer) — see OCR below\n- Legacy **`.doc`** (binary Word), **`.msg`**, PostScript\n- Images with text: PNG, JPEG, TIFF (needs Tesseract OCR)\n- Audio: mp3, wav\n\nFor those cases, a HTTP sidecar such as [textract-docker](https://github.com/floleuerer/textract-docker) is a practical fallback. It wraps Python [textract](https://github.com/deanmalmgren/textract) with Tesseract OCR, `antiword` for `.doc`, and many other backends behind a simple REST API.\n\nTypical integration pattern:\n\n1. Try `docExtract.extractText()` first (fast, in-process).\n2. If result is `\"\"` or format is unsupported — call textract-docker (or your existing docparser service).\n\n## Performance\n\ndoc-extract runs **inside the Node/Bun process** — no HTTP, base64 encoding, or Docker hop per request. That makes it a better fit for high-throughput paths (batch imports) where latency and concurrency matter.\n\n[textract-docker](https://github.com/floleuerer/textract-docker) adds network and Python/subprocess overhead on each call, but covers **OCR and legacy formats** doc-extract deliberately skips.\n\n## Build from source\n\n```bash\ngit clone https://github.com/alexstep/doc-extract.git\ncd doc-extract\nbun install\nbun run build   # Rust ≥ 1.88\nbun test\n```\n\n## License\n\nMIT\n\n---\n\n## Stats\n\n| | |\n|---|---|\n| Native addon size | ~6 MB per platform |\n| Default max input | 42 MB (`setMaxFilesizeMB`, `0` = unlimited) |\n| In-memory threshold | 64 MB (`setInMemoryThresholdMB`) |\n| Zip entry cap | 64 MB per entry — exceeds limit throws (not silent truncate) |\n| Supported extensions | 30+ |\n| Default concurrency | 32 (`DOCEXTRACT_MAX_CONCURRENT`) |\n| Runtime | Node.js ≥ 18, Bun ≥ 1.3 |\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Falexstep%2Fdoc-extract","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Falexstep%2Fdoc-extract","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Falexstep%2Fdoc-extract/lists"}