{"id":51131478,"url":"https://github.com/dignite-projects/vault-extract","last_synced_at":"2026-07-15T12:02:59.810Z","repository":{"id":352554078,"uuid":"1215527817","full_name":"dignite-projects/vault-extract","owner":"dignite-projects","description":"A channel layer that turns any content requiring IDP — scans, photos, image PDFs, Office files, digital-born docs — into trustworthy structured data: OCR + Markdown + metadata + optional field extraction, exposed via REST / EventBus / MCP (Webhook planned) to downstream RAG platforms, business systems, and AI clients. Built on ABP.","archived":false,"fork":false,"pushed_at":"2026-07-10T06:40:59.000Z","size":8628,"stargazers_count":6,"open_issues_count":12,"forks_count":1,"subscribers_count":0,"default_branch":"main","last_synced_at":"2026-07-10T07:14:03.352Z","etag":null,"topics":["abp","abp-framework","azure-document-intelligence","document-digitization","document-processing","dotnet","idp","llm","markdown","mcp","mcp-server","multi-tenant","ocr","paddleocr","rag"],"latest_commit_sha":null,"homepage":"https://dignite.com/vault-extract","language":"C#","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"lgpl-3.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/dignite-projects.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":"CODE_OF_CONDUCT.md","threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":"SECURITY.md","support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":"AGENTS.md","dco":null,"cla":null}},"created_at":"2026-04-20T02:18:12.000Z","updated_at":"2026-07-10T06:41:14.000Z","dependencies_parsed_at":null,"dependency_job_id":"33ef8c95-1c24-4b4e-b8be-27539ccad51d","html_url":"https://github.com/dignite-projects/vault-extract","commit_stats":null,"previous_names":["dignite-projects/dignite-paperbase","dignite-projects/document-ai","dignite-projects/dignite-extract","dignite-projects/vault-extract"],"tags_count":3,"template":false,"template_full_name":null,"purl":"pkg:github/dignite-projects/vault-extract","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dignite-projects%2Fvault-extract","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dignite-projects%2Fvault-extract/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dignite-projects%2Fvault-extract/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dignite-projects%2Fvault-extract/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/dignite-projects","download_url":"https://codeload.github.com/dignite-projects/vault-extract/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dignite-projects%2Fvault-extract/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":35503621,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-26T15:22:16.424Z","status":"online","status_checked_at":"2026-07-15T02:00:06.706Z","response_time":131,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["abp","abp-framework","azure-document-intelligence","document-digitization","document-processing","dotnet","idp","llm","markdown","mcp","mcp-server","multi-tenant","ocr","paddleocr","rag"],"created_at":"2026-06-25T13:00:39.155Z","updated_at":"2026-07-15T12:02:59.805Z","avatar_url":"https://github.com/dignite-projects.png","language":"C#","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Dignite Vault Extract\n\n\u003e **Dignite Vault Extract = any content requiring IDP (Intelligent Document Processing) — scans / photos / PDF images / Office files / digital-born documents → trustworthy structured data.**\n\u003e A **channel layer**, not an end-product. It doesn't consume, doesn't own, doesn't dive into business — it hands Markdown + structured metadata to downstream RAG platforms, business systems, and AI clients via REST / EventBus / MCP server / Webhook (planned).\n\nFor the full positioning, architecture rules, OUT-of-scope list, Markdown-first contract, multi-stage ETO event contract, and security covenant, see [CLAUDE.md](./CLAUDE.md). It is the truth source — this README only stages the operational entry points.\n\n## Data flow\n\n```\ncontent requiring IDP: scans / photos / PDF images / Office files / digital-born documents\n    ↓\n[Dignite Vault Extract channel]: OCR + Markdown + system metadata + type-bound field extraction\n    ↓ (REST / EventBus / MCP server / Webhook — planned)\n    ├─→ downstream RAG platform\n    ├─→ business systems (finance / CLM / HR / ERP)\n    ├─→ AI clients (Claude Desktop / Cursor / any MCP client)\n    └─→ any consumer (build your own subscriber)\n```\n\n## Solution structure\n\n```\nextract/\n├── core/      # Channel implementation — ABP layers (Abstractions / Domain.Shared / Domain / Application / EntityFrameworkCore / HttpApi / Mcp)\n├── host/      # Host application — provider wiring (OCR + AI) and middleware (ASP.NET Core API)\n├── angular/   # Angular SPA (operator UI)\n└── docs/      # Operator-facing documentation (design decisions go to GitHub Issues, not here)\n```\n\nBusiness modules (contract management / invoice management / HR records / etc.) are **not** in this repo — they belong on the downstream consumer side per the channel philosophy.\n\n## Prerequisites\n\n| Requirement | Minimum version | Notes |\n|-------------|----------------|-------|\n| [.NET SDK](https://dotnet.microsoft.com/download/dotnet) | 10.0 | |\n| [Node.js](https://nodejs.org) | 20 | Required for the Angular frontend (Angular 21 needs Node 20.19+ / 22.12+) |\n| SQL Server | 2019+ | LocalDB works for development; production runs full SQL Server |\n| [Docker Desktop](https://www.docker.com/products/docker-desktop) | any recent | Optional but recommended — runs the PaddleOCR sidecar and the local OpenTelemetry dashboard |\n\n## Getting started (local development)\n\n### 1. Start the PaddleOCR sidecar (only if you enable the PaddleOCR provider)\n\nThe host currently wires the **Vision LLM** OCR provider by default (see [Choosing an OCR provider](#choosing-an-ocr-provider)), which needs no sidecar — it reuses the `Extract` AI-provider configuration below. If you switch the host to the PaddleOCR provider, start its Docker container first:\n\n```bash\ncd host\ndocker compose up -d paddleocr\n```\n\nFirst run downloads ~600 MB of model weights and takes 30–60 seconds. Subsequent starts are instant.\n\n### 2. Configure the database and the AI provider\n\nCreate `host/src/appsettings.Development.json` with your local SQL Server connection string and an LLM provider key:\n\n```json\n{\n  \"Serilog\": { \"MinimumLevel\": { \"Default\": \"Debug\" } },\n  \"ConnectionStrings\": {\n    \"Default\": \"Server=YOUR_DB_SERVER;Database=Extract-Dev;User ID=YOUR_USER;Password=YOUR_PASSWORD;TrustServerCertificate=true\"\n  },\n  \"StringEncryption\": {\n    \"DefaultPassPhrase\": \"any-random-string-here\"\n  },\n  \"Vault\": {\n    \"Extract\": {\n      \"Endpoint\": \"https://api.openai.com/v1\",\n      \"ApiKey\": \"YOUR_REAL_API_KEY\",\n      \"ChatModelId\": \"gpt-4o-mini\",\n      \"VisionOcrModelId\": \"gpt-4o-mini\"\n    }\n  }\n}\n```\n\n\u003e This file is git-ignored. In Development mode, the application automatically generates temporary OpenIddict certificates — no `.pfx` file is needed. For LocalDB, the committed `appsettings.json` default (`Server=(LocalDb)\\MSSQLLocalDB;...`) already works without any override.\n\nAn LLM provider is **mandatory** — classification and field extraction have no non-LLM fallback, and the host fails fast at startup while `Vault:Extract:ApiKey` is still the committed placeholder. Any OpenAI-compatible endpoint works; with the default Vision LLM OCR provider, `VisionOcrModelId` must point at a vision-capable model. See [AI provider](./docs/en/configuration/ai-provider.md).\n\n### 3. Install client-side libraries\n\n```bash\ncd host/src\nabp install-libs\n```\n\n### 4. Initialize the database (first run only)\n\n```bash\ncd host/src\ndotnet run --migrate-database\n```\n\nThis creates the database schema and seeds the admin account (`admin` / `1q2w3E*`). Only needed once per fresh database.\n\n### 5. Run the backend\n\n```bash\ncd host/src\ndotnet run\n```\n\nAPI: `https://localhost:44348`. Swagger: `https://localhost:44348/swagger`.\n\n### 6. Install frontend dependencies and run Angular\n\nThe Angular SPA lives in the repository-root `angular/` directory (an Nx workspace):\n\n```bash\ncd angular\nnpm install\nnpm start\n```\n\nSPA: `http://localhost:4200`. Default seeded credentials: `admin` / `1q2w3E*`.\n\n## Choosing an OCR provider\n\nDignite Vault Extract ships three OCR providers; the host enables exactly one (`[DependsOn(...)]` in `host/src/ExtractHostModule.cs` + the matching `ProjectReference` in `host/src/Dignite.Vault.Extract.Host.csproj`):\n\n* **Vision LLM** — the host's current default (#259). Sends images / rasterized PDF pages to a vision-capable `IChatClient` model; the strongest option for phone photos, thermal receipts, and image-only PDFs. No sidecar — only a vision model id. See [docs/en/text-extraction/ocr-vision-llm.md](./docs/en/text-extraction/ocr-vision-llm.md).\n* **PaddleOCR** — local Docker sidecar (PP-StructureV3, CPU); data never leaves the network. See [docs/en/text-extraction/ocr-paddleocr.md](./docs/en/text-extraction/ocr-paddleocr.md).\n* **Azure Document Intelligence** — cloud option (`prebuilt-layout`, high accuracy) when data is allowed to leave the network. See [docs/en/text-extraction/ocr-azure-document-intelligence.md](./docs/en/text-extraction/ocr-azure-document-intelligence.md). **Not yet validated against a live Azure resource — community testing welcome ([#327](https://github.com/dignite-projects/vault-extract/issues/327)).**\n\nFull selection guidance, configuration, and resource footprint: see [docs/en/text-extraction/text-extraction.md](./docs/en/text-extraction/text-extraction.md).\n\n## Deploying to production\n\nFor database connection strings, OpenIddict signing certificate, string-encryption key, and the Docker layout, see [docs/en/deployment/deployment.md](./docs/en/deployment/deployment.md). For per-release smoke tests, see [docs/en/deployment/deployment-checklist.md](./docs/en/deployment/deployment-checklist.md).\n\n## Documentation\n\nStart at the **[documentation index](./docs/en/index.md)**. Feature docs are grouped to follow the channel's data flow:\n\n**Get started**\n\n* [Local development setup](./docs/en/get-started/local-development.md) — prerequisites, Docker sidecars, configuration, troubleshooting\n\n**Text extraction** (OCR + Markdown)\n\n* [Text extraction](./docs/en/text-extraction/text-extraction.md) — Markdown-first contract, the two extraction paths, OCR provider comparison\n* [PaddleOCR](./docs/en/text-extraction/ocr-paddleocr.md) — local OCR sidecar (PP-StructureV3, CPU); model choice and resource footprint\n* [Azure Document Intelligence](./docs/en/text-extraction/ocr-azure-document-intelligence.md) — cloud OCR (`prebuilt-layout`); resource setup and F0 tier limits\n* [Vision-LLM OCR](./docs/en/text-extraction/ocr-vision-llm.md) — multimodal-`IChatClient` OCR for photos / thermal receipts / image-only PDFs\n\n**Pipeline**\n\n* [Classification](./docs/en/pipeline/classification.md) — document-type pipeline and prompt tuning\n* [Reprocessing](./docs/en/pipeline/reprocessing.md) — bulk re-run of classification / field extraction over existing documents after a config change\n* [Pipeline runs](./docs/en/pipeline/pipeline-runs.md) — run history and review-UI payloads\n\n**Egress**\n\n* [Export templates](./docs/en/egress/export-templates.md) — per-tenant CSV / XLSX file egress: field projection, rename, ordering — zero business transformation\n* [MCP server](./docs/en/egress/mcp-server.md) — document resources + structured search tool over Streamable HTTP, OpenIddict Bearer auth\n\n**Configuration**\n\n* [AI provider](./docs/en/configuration/ai-provider.md) — provider wiring for the two keyed chat clients (title generator + structured)\n\n**Deployment \u0026 operations**\n\n* [Deployment](./docs/en/deployment/deployment.md) — DB, certificate, Docker\n* [Deployment checklist](./docs/en/deployment/deployment-checklist.md) — per-release smoke tests\n* [Observability](./docs/en/deployment/observability.md) — OpenTelemetry pipeline, aspire-dashboard for local dev, switching OTLP backends\n\nExternal references:\n\n* [ABP Framework Documentation](https://abp.io/docs/latest)\n* [Application (Single Layer) Startup Template](https://abp.io/docs/latest/solution-templates/application-single-layer)\n* [Configuring OpenIddict for Production](https://abp.io/docs/latest/Deployment/Configuring-OpenIddict#production-environment)\n\n## License\n\nDignite Vault Extract is licensed under the [GNU Lesser General Public License v3.0](./LICENSE).\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdignite-projects%2Fvault-extract","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fdignite-projects%2Fvault-extract","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdignite-projects%2Fvault-extract/lists"}