{"id":51423867,"url":"https://github.com/andenick/hdarp","last_synced_at":"2026-07-05T01:30:29.444Z","repository":{"id":368464946,"uuid":"1226136672","full_name":"andenick/hdarp","owner":"andenick","description":"HDARP: Hybrid Direct Agent Reading Protocol — Production-grade PDF extraction for AI agent pipelines (95-98% accuracy)","archived":false,"fork":false,"pushed_at":"2026-06-30T16:49:13.000Z","size":46,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":0,"default_branch":"master","last_synced_at":"2026-06-30T18:23:08.070Z","etag":null,"topics":["ai-agents","document-processing","ocr","pdf-extraction"],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/andenick.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":"CITATION.cff","codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2026-05-01T02:39:49.000Z","updated_at":"2026-06-30T16:49:17.000Z","dependencies_parsed_at":null,"dependency_job_id":null,"html_url":"https://github.com/andenick/hdarp","commit_stats":null,"previous_names":["andenick/hdarp"],"tags_count":null,"template":false,"template_full_name":null,"purl":"pkg:github/andenick/hdarp","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/andenick%2Fhdarp","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/andenick%2Fhdarp/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/andenick%2Fhdarp/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/andenick%2Fhdarp/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/andenick","download_url":"https://codeload.github.com/andenick/hdarp/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/andenick%2Fhdarp/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":35141083,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-26T15:22:16.424Z","status":"online","status_checked_at":"2026-07-04T02:00:05.987Z","response_time":113,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["ai-agents","document-processing","ocr","pdf-extraction"],"created_at":"2026-07-05T01:30:24.211Z","updated_at":"2026-07-05T01:30:29.410Z","avatar_url":"https://github.com/andenick.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# HDARP — Hybrid Direct Agent Reading Protocol\n\n**Production-grade PDF extraction for AI agent pipelines. High body-text accuracy through multi-engine OCR consensus.**\n\n\u003e **Snapshot note**: This repository is a frozen snapshot of the HDARP **v5.1** OCR-consensus layer (released 2026-05-01). The protocol has since evolved; this snapshot is preserved as a self-contained reference implementation of the multi-engine consensus approach, not as the current production version.\n\u003e\n\u003e **On the numbers**: Accuracy and cost figures in this README and `docs/` are **indicative** — illustrative ranges and worked examples drawn from development use, *not* results from a published, reproducible benchmark dataset. No formal benchmark corpus is released with this repo. Treat them as order-of-magnitude guidance, not measured claims.\n\n---\n\n## What HDARP Does\n\nHDARP solves the fundamental problem of getting structured data out of scanned PDFs at scale when you're working with AI agents. It combines two complementary approaches:\n\n1. **DARP (Direct Agent Reading)** for structured content: Uses Claude's vision API to extract tables → CSV (high accuracy), equations → LaTeX, and figures → markdown descriptions.\n\n2. **Sraffa 3.0 Multi-Engine OCR Consensus** for body text: A 3-engine ensemble (PaddleOCR, EasyOCR, Tesseract) with a 6-rule adjudication hierarchy that is, in development use, materially more accurate than any single engine.\n\n\u003e **Version note**: This repo implements the Sraffa 3.0 consensus engine. The current production system uses **Sraffa 4.0**, which adds document-adaptive routing (digital pages → PyMuPDF instant extraction, scanned pages → EasyOCR GPU with agent QA, QA failures → Chandra 2 NF4 fallback). The Sraffa 3.0 consensus engine remains the core OCR adjudication layer within Sraffa 4.0.\n\nThe result is a hybrid system that uses expensive agent vision only where it matters (tables, equations) and free local OCR where it's sufficient (body text), with intelligent consensus to maximize accuracy.\n\n---\n\n## Key Innovations\n\n### 1. Density-Aware Chunking\n\nNot all PDFs are created equal. A 100-page text-heavy academic paper behaves very differently from a 100-page image-heavy annual report. HDARP calculates the actual MB/page density of each PDF and selects the optimal chunking strategy:\n\n| Density | MB/page | Strategy | Rationale |\n|---------|---------|----------|-----------|\n| LOW | \u003c 0.05 | PAGE_FIRST | Text-heavy: maximize pages per chunk (up to 10) |\n| MEDIUM | 0.05-0.10 | SIZE_FIRST | Mixed: balance pages and size |\n| HIGH | \u003e 0.10 | SIZE_FIRST | Image-heavy: strict size limits with retry |\n\nThe splitter retries with progressively fewer pages if a chunk exceeds the size limit, and accepts oversized single pages with a warning flag rather than failing.\n\n### 2. Six-Rule OCR Consensus\n\nWhen three OCR engines look at the same text, how do you decide which one is right? HDARP applies six rules in priority order:\n\n| # | Rule | Confidence | When It Fires |\n|---|------|------------|---------------|\n| 1 | **Perfect Agreement** | 0.95-0.99 | All engines produce identical text. Confidence: `1 - (1-p₁)(1-p₂)(1-p₃)` |\n| 2 | **Majority Agreement** | 0.85-0.95 | 2+ engines agree. 5% confidence boost for consensus. |\n| 3 | **High-Confidence Unilateral** | 0.85-0.90 | One engine \u003e95% confidence, others \u003c50%. Trust the confident one. |\n| 4 | **Column-Type Validation** | 0.85-0.92 | NUMERIC context catches O→0, I→1 substitutions. Semantic validation. |\n| 5 | **Character Similarity** | 0.80-0.90 | \u003e80% character overlap. Weighted by engine priority. |\n| 6 | **Default to Primary** | 0.60-0.85 | Fallback to PaddleOCR (highest-priority engine). |\n\nThe probabilistic confidence combination in Rule 1 is mathematically rigorous: if PaddleOCR is 90% confident and EasyOCR is 85% confident and Tesseract is 80% confident, the probability all three are wrong simultaneously is (0.10)(0.15)(0.20) = 0.003 — giving us 99.7% confidence in perfect agreement.\n\n### 3. Quality Scoring Framework\n\nEvery extraction is scored on a 27-point weighted scale:\n\n| Component | Max Points | What It Measures |\n|-----------|-----------|-----------------|\n| Tables | 8 | CSV formatting, header detection, cell accuracy |\n| Text | 4 | Character accuracy, word completion, paragraph structure |\n| Equations | 3 | LaTeX validity, symbol recognition |\n| Figures | 3 | Description completeness, reference accuracy |\n| OCR Confidence | 2 | Mean confidence across consensus results |\n| Formatting | 3 | Section structure, whitespace, encoding |\n| Metadata | 4 | Page numbers, headers/footers, cross-references |\n\n### 4. Batch Processing Architecture\n\nHDARP processes documents in batches with a parallel validation pattern:\n\n```\nBatch 1 → [Processors extract] → Complete\n                                    ↓\nBatch 2 → [Processors extract] → [Validator checks Batch 1] → Complete\n                                                                  ↓\nBatch 3 → [Processors extract] → [Validator checks Batch 2] → Complete\n```\n\nThis previous-batch validation design eliminates race conditions and enables true parallel execution. The validator always works on a completed batch while processors handle the current one.\n\n**Automatic continuation** (v5.1): After completing a batch, the system automatically advances to the next PREPARED batch and continues until no more remain. Use `--single` for one-batch-at-a-time processing.\n\n### 5. Zero-Fabrication Guarantee\n\nHDARP implements a strict policy: **never fabricate content**. If a region cannot be read with sufficient confidence, it is marked as a gap rather than filled with plausible-looking text. This is critical for scholarly and regulatory use cases where false positives are worse than missing data.\n\n---\n\n## Installation\n\n```bash\ngit clone https://github.com/andenick/hdarp.git\ncd hdarp\npip install -r requirements.txt\n```\n\n### OCR Engine Dependencies\n\nHDARP requires three OCR engines. Install them separately:\n\n```bash\n# PaddleOCR (primary engine)\npip install paddlepaddle paddleocr\n\n# EasyOCR (secondary engine)\npip install easyocr\n\n# Tesseract (tertiary engine)\n# Windows: Download from https://github.com/UB-Mannheim/tesseract/wiki\n# macOS: brew install tesseract\n# Linux: sudo apt install tesseract-ocr\npip install pytesseract\n\n# PDF handling\npip install PyPDF2 PyMuPDF\n```\n\n---\n\n## Quick Start\n\n### Chunk a PDF\n\n```python\nfrom hdarp import PDFSplitterOrchestrator\n\nsplitter = PDFSplitterOrchestrator(\n    max_chunk_size_mb=1.0,\n    max_chunk_pages=10\n)\n\n# Assess density first\nreport = splitter.assess_pdf_density(\"large_document.pdf\")\nprint(f\"Density: {report.density_category} ({report.avg_density:.3f} MB/page)\")\nprint(f\"Strategy: {report.recommended_strategy}\")\nprint(f\"Estimated chunks: {report.estimated_chunks}\")\n\n# Chunk with intelligent strategy\nresult = splitter.chunk_pdf_intelligent(\"large_document.pdf\", \"output/chunks/\")\nprint(f\"Created {len(result.chunks)} chunks\")\n```\n\n### Run OCR Consensus\n\n```python\nfrom hdarp import Sraffa30ConsensusEngine\n\nengine = Sraffa30ConsensusEngine()\n\nresult = engine.adjudicate(\n    paddle_text=\"Total Revenue: $1,234,567\",\n    paddle_conf=0.92,\n    easyocr_text=\"Total Revenue: $1,234,567\",\n    easyocr_conf=0.88,\n    tesseract_text=\"Total Revenue: $l,234,567\",  # Common OCR error: 1→l\n    tesseract_conf=0.75,\n    column_type=\"TEXT\"\n)\n\nprint(f\"Winner: {result.text}\")           # \"Total Revenue: $1,234,567\"\nprint(f\"Confidence: {result.confidence}\")  # 0.987 (perfect agreement between 2 engines)\nprint(f\"Rule: {result.rule_applied}\")      # \"majority_agreement\"\nprint(f\"Engines: {result.winning_engines}\")# [\"paddle\", \"easyocr\"]\n```\n\n### Score Extraction Quality\n\n```python\nfrom hdarp import QualityScorer\n\nscorer = QualityScorer()\nscore = scorer.score_extraction(extraction_result)\nprint(f\"Quality: {score.total}/27 ({score.grade})\")\n```\n\n---\n\n## Architecture\n\n```\n┌─────────────────────────────────────────────────────┐\n│                    HDARP Pipeline                     │\n├─────────────────────────────────────────────────────┤\n│                                                      │\n│  PDF Input                                           │\n│    │                                                 │\n│    ▼                                                 │\n│  ┌──────────────────────┐                           │\n│  │ Density Assessment   │ ← Classify LOW/MED/HIGH   │\n│  │ (splitter.py)        │                           │\n│  └──────────┬───────────┘                           │\n│             │                                        │\n│             ▼                                        │\n│  ┌──────────────────────┐                           │\n│  │ Intelligent Chunking │ ← PAGE_FIRST or SIZE_FIRST│\n│  │ (splitter.py)        │   with retry on oversize  │\n│  └──────────┬───────────┘                           │\n│             │                                        │\n│     ┌───────┴───────┐                               │\n│     │               │                                │\n│     ▼               ▼                                │\n│  ┌────────┐  ┌───────────────┐                      │\n│  │  DARP  │  │  Sraffa 3.0   │                      │\n│  │ Tables │  │  OCR Consensus│                      │\n│  │ Eqns   │  │  (3 engines)  │                      │\n│  │ Figs   │  │  (6 rules)    │                      │\n│  └────┬───┘  └──────┬────────┘                      │\n│       │              │                               │\n│       └──────┬───────┘                               │\n│              ▼                                       │\n│  ┌──────────────────────┐                           │\n│  │ Quality Scoring      │ ← 27-point framework      │\n│  │ (quality_scorer.py)  │                           │\n│  └──────────┬───────────┘                           │\n│             │                                        │\n│             ▼                                        │\n│  ┌──────────────────────┐                           │\n│  │ Batch Orchestration  │ ← Parallel processing     │\n│  │ (orchestrator.py)    │   with auto-continuation  │\n│  └──────────────────────┘                           │\n│                                                      │\n└─────────────────────────────────────────────────────┘\n```\n\n---\n\n## Module Reference\n\n| Module | LOC | Purpose |\n|--------|-----|---------|\n| `splitter.py` | 582 | Density-aware PDF chunking with retry logic |\n| `ocr_engines.py` | 627 | PaddleOCR, EasyOCR, Tesseract wrappers with unified interface |\n| `consensus.py` | 590 | 6-rule consensus adjudication engine |\n| `processor.py` | 356 | Multi-engine OCR processing orchestration |\n| `orchestrator.py` | 756 | Batch pipeline with state management and auto-continuation |\n| `quality_scorer.py` | 640 | 27-point quality scoring framework |\n\n---\n\n## Performance\n\nThe figures below are **indicative**, not benchmark results — illustrative ranges observed across academic papers, books, and regulatory documents during development. No formal benchmark dataset, ground-truth definition, or reproducible eval harness is published with this repo, so treat these as order-of-magnitude guidance rather than measured claims:\n\n| Metric | Single Engine | HDARP Consensus |\n|--------|--------------|-----------------|\n| Text accuracy (clean scans) | 88-92% | 95-98% |\n| Text accuracy (degraded) | 70-80% | 85-92% |\n| Table extraction | N/A (OCR only) | high (agent vision) |\n| Processing speed | ~1 sec/page | ~3-5 sec/page |\n| False positives | Common | Near-zero (gap marking) |\n\n---\n\n## Design Philosophy\n\nHDARP was built for a specific use case: enabling AI agents to reliably extract structured data from large document corpora for scholarly research and regulatory analysis. Three principles guide its design:\n\n1. **Accuracy over speed**: Three OCR engines are slower than one, but the consensus mechanism catches errors that no single engine would. In scholarly contexts, a missed decimal point or a fabricated number can invalidate an entire analysis.\n\n2. **Transparency over opacity**: Every extraction includes a full audit trail — which engine won, which rule applied, what the confidence was, and what the alternatives were. A researcher can inspect any result and understand why it was chosen.\n\n3. **Safety over convenience**: The zero-fabrication guarantee means HDARP will never produce plausible-looking text that wasn't in the source document. Missing data is marked as missing, not filled in.\n\n---\n\n## Use with Claude Code\n\nHDARP was designed to work inside Claude Code agent pipelines. Drop a `CLAUDE.md` in your project root:\n\n```markdown\n# HDARP Project\n\nThis project uses HDARP for PDF extraction.\n\n## Quick Start\n- Chunk PDFs: `python -c \"from hdarp import PDFSplitterOrchestrator; ...\"`\n- Run OCR: See examples/ for usage patterns\n- Check quality: Use QualityScorer on extraction results\n\n## Key Files\n- hdarp/splitter.py — PDF chunking\n- hdarp/consensus.py — OCR adjudication (the interesting part)\n- hdarp/quality_scorer.py — Extraction quality scoring\n```\n\n---\n\n## License\n\nMIT\n\n---\n\n## Citation\n\nIf you use HDARP in academic work:\n\n```bibtex\n@software{hdarp2026,\n  title = {HDARP: Hybrid Direct Agent Reading Protocol},\n  author = {Anderson, Nicholas},\n  version = {5.1},\n  year = {2026},\n  url = {https://github.com/andenick/hdarp}\n}\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fandenick%2Fhdarp","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fandenick%2Fhdarp","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fandenick%2Fhdarp/lists"}