{"id":50795340,"url":"https://github.com/daboss2003/decant","last_synced_at":"2026-06-12T14:01:43.373Z","repository":{"id":362351455,"uuid":"1257480569","full_name":"daboss2003/decant","owner":"daboss2003","description":"Turn messy, real-world financial documents into **validated, structured data that's trustworthy enough for a financial workflow — and that knows when it can't be trusted and asks a human","archived":false,"fork":false,"pushed_at":"2026-06-03T19:51:29.000Z","size":3063,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2026-06-03T21:18:26.685Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"TypeScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/daboss2003.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2026-06-02T18:08:28.000Z","updated_at":"2026-06-03T19:51:33.000Z","dependencies_parsed_at":null,"dependency_job_id":null,"html_url":"https://github.com/daboss2003/decant","commit_stats":null,"previous_names":["daboss2003/decant"],"tags_count":null,"template":false,"template_full_name":null,"purl":"pkg:github/daboss2003/decant","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/daboss2003%2Fdecant","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/daboss2003%2Fdecant/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/daboss2003%2Fdecant/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/daboss2003%2Fdecant/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/daboss2003","download_url":"https://codeload.github.com/daboss2003/decant/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/daboss2003%2Fdecant/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":34247461,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-26T15:22:16.424Z","status":"online","status_checked_at":"2026-06-12T02:00:06.859Z","response_time":109,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2026-06-12T14:01:42.082Z","updated_at":"2026-06-12T14:01:43.356Z","avatar_url":"https://github.com/daboss2003.png","language":"TypeScript","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Decant\n\nTurn messy, real-world financial documents into **validated, structured data that's trustworthy enough for a financial workflow — and that knows when it can't be trusted and asks a human.**\n\nAnyone can prompt a vision model to read a receipt. The differentiator here is **calibrated confidence + safe failure**: every field gets a confidence score, low-confidence fields route to a human review queue, and every extraction/correction is recorded in an audit trail.\n\n## How it works\n\n```\nupload (image | PDF → rasterized + text layer) → Classify (Gemini Flash-Lite, batched) → segment\n       → Extract  (born-digital text → model directly, no vision; else image → vision;\n                   optional N-sample self-consistency fan-out)\n       → Validate (schema + domain/reconciliation rules)\n       → Confidence (fuse signals incl. self-consistency) → Route (auto-approve | needs review)\n       → persist + Audit trail → human review (web UI / MCP elicitation)\n```\n\n**Multi-format ingestion.** Born-digital documents — **PDF** (text layer via mupdf), **Markdown, HTML, XML, SVG, TXT, CSV, JSON, YAML** — have their exact text read **directly (no OCR, no vision model)** and fed to **both classify and extract** (cheaper + character-exact; markup is stripped to text). No image is created just to give the model — a text doc never touches the vision path; the vision model is the fallback only for scanned PDFs and raster images. (Deliberately *not* LangChain — the loaders are ~80 lines with minimal deps; LangChain's loaders are thin wrappers around the same libs and would add a large, fast-moving dependency that overlaps the existing Zod-typed pipeline.) Extraction can also run **N samples and measure self-consistency** (`--samples N`) — agreement across samples is a model-internal confidence signal that routes shaky fields to review. The pipeline runs in-process by default or as durable **BullMQ/Redis** jobs (`REDIS_URL`).\n\nA document classifies to a **registered type** (receipt/invoice, bank statement, …) → full typed schema + domain rules + per-type confidence; anything else falls back to a **generic, low-trust** extractor that always routes to review. The strongest confidence signal is **reconciliation** — receipt totals must add up; a bank statement's running balance must walk row by row — which localizes errors to the exact field/row.\n\n## Packages\n\nEvery package and app has its own `README.md` — follow the link for its exact API, usage, and gotchas.\n\n| Package | What it is |\n|---|---|\n| [`packages/schemas`](packages/schemas/README.md) | Zod single source of truth (drives Gemini structured output, validation, types, MCP/elicitation) |\n| [`packages/core`](packages/core/README.md) | Transport-agnostic domain core: registry, segmentation, rules, confidence, routing, the pipeline orchestrator |\n| [`packages/gemini`](packages/gemini/README.md) | `@google/genai`-backed Classify + Extract services (the SDK behind a mockable interface) |\n| [`packages/ocr`](packages/ocr/README.md) | tesseract.js `OcrProvider` → per-field bbox provenance (fuzzy-aligned to each value, independent of the model's claim) |\n| [`packages/enrich`](packages/enrich/README.md) | the **MCP client role** + a **pluggable verification adapter** (`makeVerifier` — add a source by implementing one `lookup`); FX enrichment + company-registry verification, with bundled demo and real (open.er-api/GLEIF) servers |\n| [`packages/ingest`](packages/ingest/README.md) | Multi-format ingestion: `toPages` (PDF rasterize + text layer via mupdf; md/html/xml/svg/… read as exact text), `persistPageImages`, `FsPageImageStore` |\n| [`packages/queue`](packages/queue/README.md) | the async-pipeline seam: a `JobQueue` with an in-process default (dev) + a Redis/BullMQ adapter (retries/backoff/concurrency), picked by `REDIS_URL` |\n| [`packages/db`](packages/db/README.md) | Prisma + SQLite persistence, the audit-trail-writing `ReviewService`, and the runtime calibrator loader (`loadCalibration`) |\n| [`packages/eval`](packages/eval/README.md) | Offline gold scoring + success metrics (field accuracy, reliability/ECE/Brier, safe-failure rate, threshold sweep) — not a runtime dependency |\n| [`packages/calibrate`](packages/calibrate/README.md) | Python offline sidecar (scikit-learn): fits Platt/isotonic calibrators from eval results; never on the request path |\n| [`apps/cli`](apps/cli/README.md) | Run extraction / eval against real Gemini; PDF rasterization (mupdf) + born-digital text-layer extraction |\n| [`apps/api`](apps/api/README.md) | NestJS **REST adapter** — results / review-queue / corrections + async `POST /uploads` over the same core + db (rate-limited; optional bearer auth) |\n| [`apps/mcp`](apps/mcp/README.md) | **MCP server adapter** (stdio + bearer-guarded HTTP) — tools/resources with elicitation-as-review |\n| [`apps/web`](apps/web/README.md) | Next.js human-in-the-loop review UI (review queue, multi-page scan + bbox overlays, upload, optional login) |\n\n**Design principle (plan §8): one domain core, many thin adapters.** The CLI, the web app, the MCP server, and the **NestJS REST API** are all thin adapters over `packages/core` + `packages/db` — so a correction made via the web UI, MCP elicitation, or `POST /documents/:id/corrections` writes the *identical* audit event (proven by `apps/api/src/api.e2e.test.ts`).\n\n```bash\n# REST API (NestJS) — results / review-queue / corrections over the same core + db\nPORT=3001 pnpm --filter @decant/api run start        # optional: API_AUTH_TOKEN=… for bearer auth\n#   GET /review-queue · GET /documents/:id · GET /documents/:id/audit · POST /documents/:id/corrections\n```\n\n## Quick start\n\n```bash\npnpm install\npnpm --filter @decant/db run db:generate \u0026\u0026 pnpm --filter @decant/db run db:push\necho 'GEMINI_API_KEY=...' \u003e packages/gemini/.env   # for the live demos\n\npnpm test          # 152 unit/integration tests\npnpm run typecheck # all packages\n```\n\n### Demos\n\n```bash\n# 1. Extract a document (real Gemini)\npnpm --filter @decant/cli run gen-sample              # writes a synthetic receipt PNG\npnpm --filter @decant/cli run extract sample-receipt.png\npnpm --filter @decant/cli run extract sample-receipt.png --save   # also push it into the review queue\npnpm --filter @decant/cli run extract sample-receipt.png --save --ocr   # + bbox provenance via Tesseract\n\n# 2. Eval over the gold set (real Gemini) — accuracy, ECE, safe-failure, threshold sweep\npnpm --filter @decant/cli run eval                       # full generated set (48 docs across 3 types)\npnpm --filter @decant/cli run eval --render-only         # render the gold images only (no API calls)\npnpm --filter @decant/cli run eval --type receipt --limit 8   # cost-controlled subset\npnpm --filter @decant/cli run eval --gold-dir gold-samples    # REAL (redacted) labeled docs from a directory\n\n# 3. Human-in-the-loop review UI (+ upload form at /upload → posts to the REST API)\npnpm --filter @decant/web run seed\npnpm --filter @decant/web run dev    # http://localhost:3000\n#   WEB_PASSWORD=… → require login · NEXT_PUBLIC_API_URL=… → point /upload at the API (default :3001)\n#   API: open + rate-limited (RATE_LIMIT_RPM, default 120/IP/min; keyed on the socket peer —\n#        set TRUST_PROXY=1 only behind a real proxy to honor X-Forwarded-For); uploads capped\n#        at 20 MB/file × 20 files; page images are served from UPLOADS_DIR (default the web\n#        app's public/uploads) so scans show in review.\n```\n\n**Postgres (managed cloud).** SQLite is the dev/test default; for a cloud Postgres set `DATABASE_URL` and switch the Prisma datasource provider (it must be a literal, so it's scripted):\n\n```bash\npnpm --filter @decant/db run use-postgres   # flips provider sqlite→postgresql (use-sqlite reverts)\npnpm --filter @decant/db run db:generate \u0026\u0026 pnpm --filter @decant/db exec prisma db push\n```\n\n## Calibration\n\nThe eval harness **measures** calibration (ECE / reliability); the offline Python sidecar (`packages/calibrate` — the only non-TS component, batch-only) **fits** a calibrator so \"0.9\" becomes a real 90%. It reads the harness's `results.json`, fits **Platt + isotonic** with scikit-learn, picks the best by ECE, and emits `calibration.json` + a before/after **reliability diagram**. The TS runtime applies the fitted params (`applyCalibration` in `@decant/core`), **parity-tested** against the sidecar.\n\n```bash\npython3 -m venv packages/calibrate/.venv\npackages/calibrate/.venv/bin/pip install -e packages/calibrate\n# pnpm --filter @decant/cli run eval  writes reports/eval/results.json, then:\npackages/calibrate/.venv/bin/python -m calibrate.fit --in reports/eval/results.json --out reports/eval/\n```\n\nThe labeled gold set is **generated** (`@decant/eval` `generateGoldSet` — deterministic/seeded, PII-free) across all three registered types, with per-type renderers + image degradation (blur/rotate/low-quality JPEG) so the model's confidence actually varies. **Real redacted documents** can be scored too: drop `\u003cname\u003e.\u003cext\u003e` + `\u003cname\u003e.gold.json` pairs into a directory and run `eval --gold-dir \u003cdir\u003e` (see `apps/cli/gold-samples/` for the format and redacted examples) — sources ingest through the same multi-format path. The Gemini client retries transient 429(per-minute)/5xx/network errors with backoff and **fails fast on a per-day quota**.\n\nOn a synthetic overconfident set the sidecar halves ECE (**0.245 → 0.101**). A real run over 16 generated receipts scored 100% field accuracy / 0% silent-error at ECE 0.067 — but a statistically meaningful **per-type** reliability diagram needs the full multi-type set, which exceeds the **Gemini free tier's 20 `gemini-2.5-flash` requests/day**; run `pnpm --filter @decant/cli run eval` on a paid key (or accumulate across days) to produce it.\n\n## MCP server\n\nDecant exposes its capabilities over the **Model Context Protocol** — `apps/mcp` is a thin adapter over the same `core` + `db` the web UI uses, so a correction made via MCP writes the **identical** audit trail. It runs over **stdio** or a **bearer-guarded Streamable HTTP** transport.\n\n- **Tools:** `extract_document`, `list_review_queue`, `get_document`, `correct_field`, and **`review_document`** — the human-review step, delivered via **MCP elicitation**.\n- **Resources:** `decant://documents/{id}`, `decant://audit/{id}`.\n\n```bash\n# drive it with the bundled demo client (stdio)\npnpm --filter @decant/mcp run client                 # list tools + review queue\npnpm --filter @decant/mcp run client \u003cdocumentId\u003e    # read its resource + review it\n\n# or register the stdio server with an MCP host (e.g. Claude Code)\nclaude mcp add decant -- pnpm --filter @decant/mcp run serve\n\n# HTTP transport (bearer-guarded, loopback): server then client\nMCP_TRANSPORT=http MCP_AUTH_TOKEN=$(openssl rand -hex 16) pnpm --filter @decant/mcp run serve\nMCP_SERVER_URL=http://127.0.0.1:3333/mcp MCP_AUTH_TOKEN=… pnpm --filter @decant/mcp run client\n```\n\nThe marquee: when `review_document` hits a flagged field it **elicits** a structured correction from the human and records it through the same `ReviewService` as the web UI — proven by headless integration tests over **both** transports (`server.test.ts` stdio; `http-auth.test.ts` HTTP) that drive the server, auto-answer the elicitation, and assert the audit trail.\n\n**HTTP auth \u0026 hardening** (`apps/mcp/src/auth.ts`, `http-transport.ts`): every request needs `Authorization: Bearer \u003cMCP_AUTH_TOKEN\u003e`, compared in **constant time** (SHA-256 + `timingSafeEqual`); missing/invalid tokens get `401` + a correct `WWW-Authenticate` challenge. The server **fails closed** (refuses to start without a token), binds to `127.0.0.1`, enables the SDK's **DNS-rebinding protection** (Host/Origin allow-lists), uses crypto-random session ids, caps the request body (1 MiB → `413`) and concurrent sessions, and reaps idle/disconnected sessions. A static shared secret is a deliberate simplification of the MCP OAuth 2.1 authorization framework — appropriate for a local tool. The transport choices were derived from the SDK source and the implementation passed an adversarial security review.\n\n### MCP client role (consuming other servers)\n\nDecant is also an MCP **client**: after extraction it connects OUT to external MCP servers to **enrich** and **verify** data (`packages/enrich`, bundled deterministic demo servers under `src/demo/`).\n\n- **FX** — convert money fields into a base currency (enrichment).\n- **Verification** — cross-check an extracted field against an authority. The built-in **company registry** verifier compares the registered name to the extracted `companyName`; a **mismatch routes the field to human review** — an external-source *safe failure* that feeds the same trust loop. Verdicts: `verified` (found, value matches, **and** in good standing), `mismatch`, `not_found`, `inactive` (found but dissolved), `unavailable` (source unreachable) — each with its own signal so \"couldn't verify\" is never mistaken for \"verified\". A verified match records a `\u003cverifier\u003eVerified` corroboration plus the answering `source` and an anchoring reference (e.g. a GLEIF LEI). Surfaced in the review UI (\"External verification\" panel + an honest \"Why\") and persisted to `Document.enrichment` with an `enriched` audit event.\n\n```bash\npnpm --filter @decant/cli run extract sample-receipt.png --save --enrich        # deterministic demo servers\npnpm --filter @decant/cli run extract sample-receipt.png --save --enrich-live   # REAL servers (see egress below)\n```\n\n**Live mode \u0026 data egress (opt-in, OFF by default).** `--enrich-live` (or `buildEnrichment({ live: true })`) swaps the deterministic demo servers for **real** ones — `open.er-api.com` (FX, free/no-key) and **GLEIF** (`api.gleif.org`, the free global legal-entity registry). Plain `--enrich` and all tests/CI stay fully local/deterministic. What leaves the process when live:\n\n| Provider | Sent | NOT sent |\n|---|---|---|\n| FX `open.er-api.com` | the source ISO **currency code** only | the amount (converted locally), the date |\n| Registry `api.gleif.org` | the extracted company **legal name** | the RC number (reaches the local registry child but isn't forwarded to GLEIF's name-based API), the image, amounts, and any secrets (no `process.env`/`GEMINI_API_KEY` is forwarded to spawned children) |\n\n**Add a verification source — implement one function.** Verification is a generic, pluggable adapter — **not** registry- or country-specific. Decant owns the machinery (compare → verdict → route → audit); you supply a `lookup` and which field it checks. The same primitive covers a company registry in *any* jurisdiction (NG CAC, UK Companies House, SEC, GLEIF, …), a **tax ID**, a **bank account name**, **sanctions screening**, an **address** — anything where an extracted field can be checked against an authority:\n\n```ts\nimport { makeVerifier, fieldValue, EnrichmentService } from '@decant/enrich';\n\n// company registry (any country): id field → value field\nconst ukRegistry = makeVerifier({\n  name: 'companiesHouse',\n  field: 'companyName',\n  applies: (doc) =\u003e !!fieldValue(doc, 'companyNumber'),\n  lookup: async (doc) =\u003e {\n    const rec = await myRegistryApi(String(fieldValue(doc, 'companyNumber')));   // ← the only thing you write\n    return rec ? { value: rec.name, standing: rec.status, reference: rec.number, source: 'companies-house' } : null;\n    // null ⇒ not_found · throw ⇒ unavailable (both route to review — never silently dropped)\n  },\n});\n\n// a NON-registry check, same primitive: exact-match a tax ID against an authority\nconst taxId = makeVerifier({ name: 'taxId', field: 'taxId', compare: (a, b) =\u003e (a === b ? 1 : 0), lookup: myTaxLookup });\n\nnew EnrichmentService([ukRegistry, taxId /*, … */]);\n```\n\n`makeVerifier` handles the verdict, the standing gate, the trust-loop routing, the UI signal, and the audit entry. An MCP-backed source plugs in the same way via `mcpLookup(client, …)`.\n\n`registryVerifier(lookup, opts?)` is thin sugar over `makeVerifier` for the common id→value registry shape — **provider- and jurisdiction-agnostic**. The defaults (`rcNumber` → `companyName`) match the bundled CAC example; override `idField`/`valueField`/`name` for any other registry:\n\n```ts\nimport { registryVerifier } from '@decant/enrich';\n\nregistryVerifier(myCacLookup);                                          // NG CAC (defaults)\nregistryVerifier(myUkLookup, { name: 'companiesHouse', idField: 'companyNumber' });\n```\n\nEnrichment is **best-effort** (an unreachable server never sinks an extraction); external output is zod-validated at the boundary, the FX figure is recomputed locally from the validated rate, GLEIF results are gated on entity standing and anchored to a LEI, and connects fail fast. Integration tests spawn the demo servers over stdio (`enrich.test.ts`); network-gated tests exercise the real adapters (`live.test.ts`). The implementation passed two adversarial reviews (correctness, resource safety, security/privacy).\n\n## License\n\n[MIT](./LICENSE) © 2026 Samson Oluwafemi\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdaboss2003%2Fdecant","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fdaboss2003%2Fdecant","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdaboss2003%2Fdecant/lists"}