{"id":50558080,"url":"https://github.com/dlcs/text-services","last_synced_at":"2026-06-04T09:01:30.742Z","repository":{"id":361825823,"uuid":"1195197959","full_name":"dlcs/text-services","owner":"dlcs","description":null,"archived":false,"fork":false,"pushed_at":"2026-06-01T11:27:12.000Z","size":1070,"stargazers_count":1,"open_issues_count":5,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2026-06-01T13:17:21.313Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"C#","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/dlcs.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2026-03-29T11:11:57.000Z","updated_at":"2026-05-28T16:26:59.000Z","dependencies_parsed_at":null,"dependency_job_id":null,"html_url":"https://github.com/dlcs/text-services","commit_stats":null,"previous_names":["dlcs/text-services"],"tags_count":null,"template":false,"template_full_name":null,"purl":"pkg:github/dlcs/text-services","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dlcs%2Ftext-services","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dlcs%2Ftext-services/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dlcs%2Ftext-services/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dlcs%2Ftext-services/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/dlcs","download_url":"https://codeload.github.com/dlcs/text-services/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dlcs%2Ftext-services/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":33897568,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-26T15:22:16.424Z","status":"online","status_checked_at":"2026-06-04T02:00:06.755Z","response_time":64,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2026-06-04T09:01:30.079Z","updated_at":"2026-06-04T09:01:30.732Z","avatar_url":"https://github.com/dlcs.png","language":"C#","funding_links":[],"categories":[],"sub_categories":[],"readme":"# TextServices\n\nIIIF Text Services — a .NET 10 solution that indexes the text content of IIIF Manifests and\nprovides IIIF Content Search, Autocomplete, and a suite of text-derived annotation endpoints.\n\n## What it does\n\n1. The **Builder API** accepts a IIIF Manifest (or an explicit list of pages) and builds a binary\n   text index — a compact map of every word, its bounding box, and its position on every canvas.\n   It also produces several stored derivatives from that index.\n2. The **Search API** serves those artefacts through IIIF-standard endpoints and returns a\n   decorated version of the original Manifest that is ready to load directly in a IIIF viewer.\n\n## Architecture\n\n```\n caller                  Builder API              Search API\n  |                         |                           |\n  |-- POST /textbuilder  --\u003e|  fetches Manifest +       |\n  |\u003c-- 202 + Location ------|  text files; builds \u0026     |\n  |                         |  stores index artefacts   |\n  |-- GET /textbuilder/id -\u003e|                           |\n  |\u003c-- job status ----------|                           |\n  |                         |                           |\n  |-- GET /text-augmented/v3/id -----------------------\u003e|\n  |\u003c-- decorated Manifest ------------------------------|\n  |                                                     |\n  |-- GET /search/v2/id?q=term ------------------------\u003e|\n  |\u003c-- IIIF Search v2 AnnotationPage -------------------|\n```\n\nThe two services share a storage backend (filesystem or S3). The Builder API writes artefacts\nonce; the Search API reads them. They can be deployed and scaled independently.\n\n---\n\n## Supported text formats\n\nThe Builder API selects a text-format provider automatically based on the `profile`, `format`,\nand `label` metadata of each `seeAlso` or `annotations` link in the Manifest.\n\n| Format | Detection |\n|---|---|\n| **METS-ALTO** (v2 / v3) | `seeAlso` profile contains `alto` (case-insensitive), or label contains `ALTO` / `METS-ALTO` |\n| **hOCR** | `seeAlso` profile contains `hocr`, or label contains `hOCR` |\n| **WebVTT** | format `text/vtt`, profile contains `vtt`, or label contains `vtt` / `webvtt` / `transcript` |\n| **W3C Annotations** | External `AnnotationPage` with an `id` (fetched from the canvas `annotations` array) |\n\nWhen neither `profile` nor `label` is present on a `seeAlso` entry, ALTO is assumed (the most\ncommon case for unattributed XML).\n\n---\n\n## What gets built\n\nWhat the Builder API can produce depends on the text format supplied.\n\n### METS-ALTO\n\nALTO provides per-word bounding boxes, so every word is indexed with a spatial position on the\ncanvas. The builder also extracts `ComposedBlock` elements (tables, illustrations, figures) when\nthey are present.\n\nProduced artefacts:\n\n| Artefact | Always? | Condition |\n|---|---|---|\n| Text index (word positions + search) | Yes | — |\n| AutoComplete index | Yes | — |\n| Plain text | Yes | — |\n| PDF | Yes | Requires at least one canvas with an image URL in the Manifest's painting annotations |\n| Manifest-level line annotations | Yes | — |\n| Figures / tables / illustrations | **Only if present in source** | ALTO `\u003cComposedBlock\u003e` elements with non-zero dimensions |\n\n### hOCR\n\nhOCR also provides per-word bounding boxes. No `ComposedBlock` equivalent exists in hOCR, so\nfigures are never extracted.\n\n| Artefact | Always? |\n|---|---|\n| Text index (word positions + search) | Yes |\n| AutoComplete index | Yes |\n| Plain text | Yes |\n| PDF | Yes (requires image URLs in painting annotations) |\n| Manifest-level line annotations | Yes |\n| Figures / tables / illustrations | **Never** |\n\n### WebVTT (audio / video captions)\n\nWebVTT is time-coded, not spatially positioned. Words carry `#t=` time fragments instead of\n`#xywh=` bounding boxes. Full-text search works, but spatial overlays and PDFs do not apply.\n\n| Artefact | Always? |\n|---|---|\n| Text index (word positions + search) | Yes |\n| AutoComplete index | Yes |\n| Plain text | Yes |\n| PDF | **Never** (no page images) |\n| Manifest-level line annotations (temporal) | Yes |\n| Figures / tables / illustrations | **Never** |\n\n### W3C Annotations (`AnnotationPage`)\n\nWhen a canvas has an external `AnnotationPage` link in its `annotations` array, the Builder API\nfetches that page and extracts words from `TextualBody` items. Bounding boxes are taken from the\nannotation `target` fragment (`#xywh=` or `#t=`).\n\nThis path is used for Manifests that already carry their own transcription annotations — for\nexample, a Manifest produced by a crowd-sourced transcription tool. The words from those\nannotations are re-indexed so that IIIF Content Search and all the derived endpoints work against\nthem, even if the original source did not provide a search service.\n\n| Artefact | Notes |\n|---|---|\n| Text index (word positions + search) | Yes |\n| AutoComplete index | Yes |\n| Plain text | Yes |\n| PDF | Only if annotations have `#xywh=` targets and image services are present |\n| Manifest-level line annotations | Yes |\n| Figures / tables / illustrations | **Never** |\n\n---\n\n## Augmentations made to `/text-augmented/v3/{id}`\n\nThe `/text-augmented/v3/{id}` endpoint loads the Manifest stored by the Builder API and injects\nsearch services, annotation links, and rendering links on-the-fly, then returns the result.\n\nWhat is stored — and therefore what the caller receives — depends on the job source:\n\n- **`sourceUri` jobs**: the original Manifest is fetched once, stored unmodified, and served as\n  the base. The caller gets back the publisher's full Manifest — labels, thumbnails, metadata,\n  rights, existing services — with the new augmentations added. Nothing in the original is\n  changed or removed.\n- **`sourceData` jobs**: no source Manifest exists. The Builder API synthesises a minimal skeleton\n  at build time containing only one canvas per page (with canvas id, dimensions, and optional\n  painting annotation). The caller gets this skeleton plus the augmentations. Labels, thumbnails,\n  and other descriptive metadata are absent.\n\nEach augmentation below has a condition — if the condition is not met (e.g. no words were indexed,\nor no canvases have painting annotations), that augmentation is silently omitted.\n\n### `service` array — search services\n\nAdded when: the text index contains at least one word.\n\n```json\n[\n  {\n    \"id\": \"https://search.example.org/search/v2/my-collection/my-book\",\n    \"type\": \"SearchService2\",\n    \"service\": [{ \"id\": \"…/autocomplete/v2/…\", \"type\": \"AutoCompleteService2\" }]\n  },\n  {\n    \"id\": \"https://search.example.org/search/v1/my-collection/my-book\",\n    \"type\": \"SearchService1\",\n    \"service\": [{ \"id\": \"…/autocomplete/v1/…\", \"type\": \"AutoCompleteService1\" }]\n  }\n]\n```\n\nSearchService2 (IIIF Search 2) is listed first; SearchService1 (IIIF Search 1) follows for\nbackward-compatible viewers. Any pre-existing `service` entries in the Manifest are preserved.\n\n### `rendering` array — plain text and PDF\n\n**Plain text link** — added when: any words were indexed.\n\n**PDF link** — added when: at least one image-based (non-temporal) canvas has words indexed\nand the stored Manifest has painting annotations with image URLs the PDF renderer can fetch.\nFor `sourceUri` jobs these come from the source Manifest's own painting annotations; for\n`sourceData` jobs they are the `imageUri` values echoed into the synthesised Manifest.\nTemporal-only sources (audio/video with WebVTT) never produce a PDF.\n\n### Per-canvas `annotations` — line and word annotation pages\n\nAdded to each canvas's `annotations` array when: that canvas has at least one word indexed.\nTwo references are added per canvas — one for line-level and one for word-level annotations.\n\n```json\n[\n  { \"id\": \"…/annotations/lines/v1/0/my-collection/my-book\", \"type\": \"AnnotationPage\",\n    \"label\": { \"en\": [\"Line-level transcription\"] } },\n  { \"id\": \"…/annotations/words/v1/0/my-collection/my-book\", \"type\": \"AnnotationPage\",\n    \"label\": { \"en\": [\"Word-level transcription\"] } }\n]\n```\n\nThese annotation pages are generated dynamically from the in-memory index on each request.\nCanvas index `{n}` is zero-based and precedes the job ID in the URL (a routing constraint).\n\n### Manifest-level `annotations` — full-document line annotations\n\nAdded to the manifest's top-level `annotations` array when: any words were indexed.\n\n```json\n{ \"id\": \"…/annotations/manifest/v1/my-collection/my-book\", \"type\": \"AnnotationPage\",\n  \"profile\": \"https://dlcs.io/profiles/all-text\",\n  \"label\": { \"en\": [\"Text of all canvases\"] } }\n```\n\nThis is a single stored `AnnotationPage` covering every canvas, at line granularity. It is\nbuilt once at index time and served directly from storage — unlike the per-canvas pages, which\nare generated dynamically. It is intended for bulk harvesting: callers can fetch this one file\nto obtain all line annotations for the document without iterating individual canvas endpoints.\n\n### Manifest-level `annotations` — figures, tables and illustrations\n\nAdded to the manifest's top-level `annotations` array when: the source contained METS-ALTO\n`\u003cComposedBlock\u003e` elements with non-zero dimensions.\n\n```json\n{ \"id\": \"…/identified/figures/my-collection/my-book\", \"type\": \"AnnotationPage\",\n  \"label\": { \"en\": [\"Figures, tables and illustrations\"] } }\n```\n\nThis is never produced for hOCR, WebVTT, or W3C Annotation sources — only METS-ALTO.\n\n---\n\n## Summary: augmentations by source format\n\n| Augmentation | METS-ALTO | hOCR | WebVTT | W3C Annotations |\n|---|:---:|:---:|:---:|:---:|\n| Search + Autocomplete services | ✓ | ✓ | ✓ | ✓ |\n| Plain text rendering link | ✓ | ✓ | ✓ | ✓ |\n| PDF rendering link | ✓ | ✓ | — | ✓ (spatial only) |\n| Per-canvas line + word annotations | ✓ | ✓ | ✓ (temporal) | ✓ |\n| Manifest-level line annotations | ✓ | ✓ | ✓ (temporal) | ✓ |\n| Figures / tables / illustrations | ✓ (if present) | — | — | — |\n\n---\n\n## Quick start\n\n### 1 — Build a text index\n\n```http\nPOST /textbuilder\nContent-Type: application/json\n\n{\n  \"id\": \"my-collection/my-book\",\n  \"sourceUri\": \"https://example.org/iiif/my-book/manifest\"\n}\n```\n\nAdd a `\"services\"` integer field to restrict which endpoints are built (omit for all). See\n[Service flags](docs/builder-api.md#service-flags) for the flag values.\n\nThe response is `202 Accepted` with a `Location` header. Poll until `status` is `Completed`:\n\n```http\nGET /textbuilder/my-collection/my-book\n```\n\n### 2 — Search\n\n```http\nGET /search/v2/my-collection/my-book?q=annual+report\n```\n\n### 3 — Load the decorated Manifest in a viewer\n\n```http\nGET /text-augmented/v3/my-collection/my-book\n```\n\nPaste this URL into Universal Viewer, Clover, or any IIIF v3 viewer that supports IIIF Content\nSearch. The Manifest returned already has `SearchService2` and `SearchService1` injected.\n\n---\n\n## Local development setup\n\n### Prerequisites\n\n- [.NET 10 SDK](https://dotnet.microsoft.com/download/dotnet/10.0)\n- PostgreSQL 14 or later (used by the Builder API for job state and the Hangfire queue)\n\n### Ports (Development profile)\n\n| Application | URL |\n|---|---|\n| Builder API | http://localhost:5283 |\n| Search API | http://localhost:5294 |\n| Demo UI | http://localhost:5100 |\n\nThe Builder API and Search API share a storage directory (`C:/textservices-data` by default on\nWindows). Both must point at the same path.\n\n### 1 — Configure the Builder API\n\nCreate `src/TextServices.Builder.Api/appsettings.Development.json` (not committed — add your own\ncredentials):\n\n```json\n{\n  \"ConnectionStrings\": {\n    \"BuilderDb\": \"Host=localhost;Database=textservices_builder;Username=postgres;Password=your-password\"\n  },\n  \"TextServices\": {\n    \"SearchApiBaseUrl\": \"http://localhost:5294\",\n    \"AllowFileImageProxy\": true\n  }\n}\n```\n\n`AllowFileImageProxy: true` lets the Demo UI serve locally stored images through the Search API's\n`/proxy/image` endpoint — only enable this in local development.\n\n### 2 — Run the EF Core migrations\n\nThe Builder API manages its own database schema. Run the migrations once (and again after any\nfuture schema changes):\n\n```bash\ncd src/TextServices.Builder.Api\ndotnet ef database update\n```\n\nIf `dotnet ef` is not installed: `dotnet tool install -g dotnet-ef`\n\n### 3 — Configure the Search API (optional)\n\nThe Search API has no database and works out of the box for local development. If you need to\noverride defaults, create `src/TextServices.Search.Api/appsettings.Development.json`:\n\n```json\n{\n  \"TextServices\": {\n    \"BaseUrl\": \"http://localhost:5294\",\n    \"AllowFileImageProxy\": true\n  }\n}\n```\n\n### 4 — Start the applications\n\nOpen three terminals and run each application:\n\n```bash\n# Terminal 1 — Builder API\ncd src/TextServices.Builder.Api\ndotnet run\n```\n\n```bash\n# Terminal 2 — Search API\ncd src/TextServices.Search.Api\ndotnet run\n```\n\n```bash\n# Terminal 3 — Demo UI\ncd src/TextServices.Demo\ndotnet run\n```\n\nThen open http://localhost:5100 in a browser.\n\n### 5 — Run the tests\n\n```bash\ncd src\ndotnet test TextServices.Tests/TextServices.Tests.csproj\n```\n\nThe unit/integration tests run entirely in-process and do not require a running database or\neither API to be up.\n\n### 6 — Code formatting\n\nThis project enforces `dotnet format` on every commit via [`pre-commit`](https://pre-commit.com/).\nInstall it once (requires Python):\n\n```bash\npip install pre-commit   # or: brew install pre-commit / scoop install pre-commit\n```\n\nThen wire up the hook once per clone:\n\n```bash\npre-commit install\n```\n\nAfter that, `dotnet format` runs automatically on `git commit` and will block the commit if any\nC# files need reformatting. To run it manually across all files:\n\n```bash\npre-commit run --all-files\n```\n\nTo auto-fix formatting issues without committing:\n\n```bash\ndotnet format src/TextServices.sln\n```\n\n### Storage directory\n\nThe default storage root is `C:/textservices-data`. The Builder API creates subdirectories\nautomatically as jobs complete. On Linux/macOS, change `Storage:RootPath` in\n`appsettings.Development.json` to a writable path (e.g. `/tmp/textservices-data`).\n\nThe Search API must be configured with the same path via `StorageRootPath`.\n\n### Builder API configuration reference\n\nAll settings live under the `TextServices` key.\n\n| Setting | Default | Description |\n|---|---|---|\n| `SearchApiBaseUrl` | `\"\"` | Public base URL of the Search API. Used to populate `searchV1`/`searchV2` fields in job responses and to construct `/proxy/image` URLs in synthesised Manifests. Leave empty if the Search API is not yet deployed. |\n| `MaxConcurrentPageFetches` | `8` | Maximum number of text files (ALTO, VTT, AnnotationPage) fetched in parallel within a single job. Keep low (4–8) for third-party HTTP hosts; increase to 16–32 for internal sources; 64–128 is reasonable for S3. |\n| `ReportBatchProgress` | `true` | When `true`, `PagesCompleted` is flushed to the database every 10 pages so `GET /textbuilder/{id}` reflects live progress. Set to `false` to reduce database writes on large manifests. |\n| `AllowFileImageProxy` | `false` | Allow the Search API's `/proxy/image` endpoint to serve `file://` image URIs. Only enable in trusted local-dev environments. |\n| `Storage:RootPath` | `textservices-data` | Root directory for stored artefacts (filesystem store). Must be readable by the Search API. |\n| `Storage:S3:BucketName` | `\"\"` | S3 bucket for stored artefacts. When set, S3 is used instead of the filesystem store. |\n| `Storage:S3:KeyPrefix` | `\"\"` | Optional prefix for all S3 object keys (e.g. `\"textservices/\"`). |\n| `CorsAllowedOrigins` | `[]` | Allowed CORS origins for the Builder API. Empty array disables CORS. |\n\nSee [Builder API reference](docs/builder-api.md) for the full configuration reference including all notification options.\n\n---\n\n## Docker\n\nThree Dockerfiles are provided at the repo root — one per application — all using a two-stage\nbuild (SDK for compile, ASP.NET runtime for the final image). The build context is always the\n**repo root**.\n\n### docker-compose (recommended for local dev)\n\n`docker-compose.yml` brings up the full stack with a single command:\n\n```bash\ndocker compose up --build\n```\n\n| Service | Host port | Description |\n|---|---|---|\n| `postgres` | 5452 | PostgreSQL 14 (job state + Hangfire queue) |\n| `builder` | 5283 | Builder API |\n| `search` | 5294 | Search API |\n| `demo` | 5100 | Demo UI |\n\nThe Builder and Search APIs share a named Docker volume (`txt_textservices_data`) for text\nartefacts. The Builder API applies EF Core migrations automatically on startup\n(`RunMigrations=true`).\n\nBoth APIs run with `ASPNETCORE_ENVIRONMENT=Development`, which enables the OpenAPI docs\n(`/openapi/v1.json`) and the Hangfire dashboard (`http://localhost:5283/hangfire`).\n\nAfter startup, open http://localhost:5100.\n\n### Building images individually\n\n```bash\ndocker build -f Dockerfile.Builder -t textservices-builder .\ndocker build -f Dockerfile.Search  -t textservices-search .\ndocker build -f Dockerfile.Demo    -t textservices-demo .\n```\n\nEach image exposes port `8080`. Configuration is supplied via environment variables using the\nstandard .NET double-underscore separator for nested keys (e.g.\n`TextServices__Storage__RootPath=/data`).\n\n---\n\n## Documentation\n\n- [Builder API reference](docs/builder-api.md)\n- [Search API reference](docs/search-api.md)\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdlcs%2Ftext-services","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fdlcs%2Ftext-services","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdlcs%2Ftext-services/lists"}