{"id":51130765,"url":"https://github.com/anders94/gutenberg-reader","last_synced_at":"2026-06-25T12:01:27.383Z","repository":{"id":359604312,"uuid":"1233478866","full_name":"anders94/gutenberg-reader","owner":"anders94","description":"Convert any Project Gutenberg book into structured JSON ready for text-to-speech (TTS) audiobook generation.","archived":false,"fork":false,"pushed_at":"2026-05-22T15:24:36.000Z","size":119,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2026-05-22T20:22:30.269Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/anders94.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2026-05-09T02:21:21.000Z","updated_at":"2026-05-22T15:24:40.000Z","dependencies_parsed_at":null,"dependency_job_id":null,"html_url":"https://github.com/anders94/gutenberg-reader","commit_stats":null,"previous_names":["anders94/gutenberg-reader"],"tags_count":null,"template":false,"template_full_name":null,"purl":"pkg:github/anders94/gutenberg-reader","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/anders94%2Fgutenberg-reader","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/anders94%2Fgutenberg-reader/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/anders94%2Fgutenberg-reader/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/anders94%2Fgutenberg-reader/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/anders94","download_url":"https://codeload.github.com/anders94/gutenberg-reader/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/anders94%2Fgutenberg-reader/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":34773843,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-26T15:22:16.424Z","status":"online","status_checked_at":"2026-06-25T02:00:05.521Z","response_time":101,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2026-06-25T12:01:26.647Z","updated_at":"2026-06-25T12:01:27.368Z","avatar_url":"https://github.com/anders94.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# gutenberg-reader\n\nConvert any [Project Gutenberg](https://www.gutenberg.org/) book into structured JSON ready for text-to-speech (TTS) audiobook generation with [tts-audiobook](https://github.com/anders94/tts-audiobook). Each sentence is labelled as **narration** or **dialogue**, with speaker attribution and pronunciation hints, so TTS engines can apply per-character voices automatically.\n\nProcessing runs entirely locally using [Ollama](https://ollama.com/) — no cloud API keys required.\n\n---\n\n## Features\n\n- Downloads and processes any Gutenberg book by numeric ID\n- Splits text into narration and dialogue segments with speaker attribution\n- Identifies characters and their aliases across the full book\n- Deterministic anchor-propagation pass corrects misattributed back-and-forth dialogue before the LLM review\n- Resumable pipeline: interrupted runs pick up where they left off\n- Atomic cache writes — safe to kill at any point\n\n---\n\n## Requirements\n\n- Python 3.11+\n- [Ollama](https://ollama.com/) running locally (default: `http://localhost:11434`)\n- A capable instruction-following model — see [Model recommendations](#model-recommendations)\n\n---\n\n## Installation\n\n```bash\npip install gutenberg-reader\n```\n\nOr install from source:\n\n```bash\ngit clone https://github.com/your-username/gutenberg-reader\ncd gutenberg-reader\npip install -e .\n```\n\n---\n\n## Quick start\n\n```bash\n# Process Pride and Prejudice (book 1342)\ngutenberg-reader 1342\n\n# Process Dr. Jekyll and Mr. Hyde with verbose output\ngutenberg-reader 43 --verbose\n\n# Use a specific model\ngutenberg-reader 1342 --model gemma3:27b --verbose\n```\n\nOutput is written to `cache/\u003cbook_id\u003e/07-final/\u003cbook_id\u003e.json`.\n\n---\n\n## CLI reference\n\n```\ngutenberg-reader BOOK_ID [OPTIONS]\n```\n\n`BOOK_ID` is the numeric Project Gutenberg book identifier. Find it in the book's URL:\n`https://www.gutenberg.org/ebooks/1342` → ID is `1342`.\n\n### Options\n\n| Option | Default | Description |\n|--------|---------|-------------|\n| `--model MODEL` | `qwen2.5:14b` | Ollama model for segmentation and character discovery |\n| `--validator MODEL` | *(same as --model)* | Separate model for the Stage 06 critic pass |\n| `--ollama-url URL` | `http://localhost:11434` | Ollama API base URL |\n| `--cache-dir DIR` | `./cache` | Directory for all cached stage outputs |\n| `--output FILE` | *(auto)* | Override output JSON path |\n| `--chunk-size N` | `400` | Words per processing chunk (lower = more API calls, less context pressure) |\n| `--overlap N` | `150` | Overlap words between chunks |\n| `--no-critic` | off | Skip Stage 06 LLM critic pass (faster; deterministic anchor pass still runs) |\n| `--force-stage N` | — | Re-run from stage N (1–7) forward, discarding cached results from that stage on |\n| `--chapters N[,N,…]` | — | Process only specific chapter numbers (e.g. `1,2,5`) |\n| `--max-retries N` | `3` | Max LLM retries per chunk on integrity failure |\n| `-v / --verbose` | off | Show per-stage progress and correction details |\n\n### Examples\n\n```bash\n# Fast run: skip LLM critic, use deterministic anchor pass only\ngutenberg-reader 1342 --no-critic --verbose\n\n# Use a larger model for segmentation, lighter model for critic\ngutenberg-reader 1342 --model gemma3:27b --validator gemma3:12b\n\n# Re-run only the assembly stage (Stage 07) to regenerate final JSON\ngutenberg-reader 1342 --force-stage 7\n\n# Re-run segmentation + everything after, for chapters 1–3 only\ngutenberg-reader 1342 --force-stage 5 --chapters 1,2,3\n\n# Point at a remote Ollama instance\ngutenberg-reader 1342 --ollama-url http://192.168.1.10:11434 --model llama3.3:70b\n\n# Write output to a custom path\ngutenberg-reader 1342 --output ~/audiobooks/pride_and_prejudice.json\n```\n\n---\n\n## Model recommendations\n\nAny Ollama model that follows structured JSON instructions will work. Tested configurations:\n\n| Model | Quality | Speed | Notes |\n|-------|---------|-------|-------|\n| `gemma3:27b` | ★★★★★ | Slow | Best attribution accuracy |\n| `qwen2.5:14b` | ★★★★☆ | Medium | Good default choice |\n| `gemma3:12b` | ★★★★☆ | Fast | Good balance |\n| `gemma4:latest` | ★★★★☆ | Fast | Tested reference model |\n| `llama3.3:70b` | ★★★★★ | Very slow | For maximum quality |\n\nPull a model before first use:\n\n```bash\nollama pull qwen2.5:14b\n```\n\nSmaller models (7B and below) tend to produce more JSON formatting errors and speaker misattributions. The pipeline has automatic repair and retry logic, but larger models need fewer corrections.\n\n---\n\n## Output format\n\nThe final JSON has this structure:\n\n```json\n{\n  \"metadata\": {\n    \"title\": \"The Strange Case of Dr. Jekyll and Mr. Hyde\",\n    \"author\": \"Robert Louis Stevenson\",\n    \"language\": \"English\",\n    \"gutenberg_id\": \"43\",\n    \"release_date\": \"June 27, 2008\",\n    \"credits\": \"David Widger\"\n  },\n  \"chapters\": [\n    {\n      \"chapter\": {\n        \"number\": 1,\n        \"title\": \"STORY OF THE DOOR\",\n        \"start_line\": 42,\n        \"end_line\": 310,\n        \"word_count\": 2150\n      },\n      \"processed\": {\n        \"chapter_number\": 1,\n        \"chapter_title\": \"STORY OF THE DOOR\",\n        \"segments\": [\n          {\n            \"type\": \"narration\",\n            \"text\": \"Mr. Utterson the lawyer was a man of a rugged countenance that was never lighted by a smile;\",\n            \"speaker\": null,\n            \"pronunciation_hints\": [],\n            \"notes\": null\n          },\n          {\n            \"type\": \"dialogue\",\n            \"text\": \"\\u201cI incline to Cain\\u2019s heresy,\\u201d\",\n            \"speaker\": \"Mr. Utterson\",\n            \"pronunciation_hints\": [],\n            \"notes\": null\n          },\n          {\n            \"type\": \"narration\",\n            \"text\": \"he used to say quaintly:\",\n            \"speaker\": null,\n            \"pronunciation_hints\": [],\n            \"notes\": null\n          },\n          {\n            \"type\": \"dialogue\",\n            \"text\": \"\\u201cI let my brother go to the devil in his own way.\\u201d\",\n            \"speaker\": \"Mr. Utterson\",\n            \"pronunciation_hints\": [],\n            \"notes\": null\n          }\n        ],\n        \"word_count\": 2150\n      },\n      \"validation\": null\n    }\n  ],\n  \"characters\": [\n    {\n      \"name\": \"Mr. Utterson\",\n      \"aliases\": [],\n      \"pronunciation_hints\": [],\n      \"first_appearance_chapter\": 1\n    },\n    {\n      \"name\": \"Dr. Jekyll\",\n      \"aliases\": [\"Henry Jekyll\"],\n      \"pronunciation_hints\": [],\n      \"first_appearance_chapter\": 1\n    }\n  ],\n  \"statistics\": {\n    \"total_chapters\": 10,\n    \"total_words\": 25615,\n    \"total_segments\": 1315,\n    \"total_characters\": 8,\n    \"processing_time_seconds\": 777.3,\n    \"validation_performed\": true,\n    \"pipeline_version\": \"1.0.0\"\n  }\n}\n```\n\n### Segment types\n\n| `type` | `speaker` | Description |\n|--------|-----------|-------------|\n| `\"narration\"` | `null` | All non-spoken text: description, action, attribution tags |\n| `\"dialogue\"` | `\"Character Name\"` | Text inside quotation marks, spoken aloud by a character |\n\nAttribution phrases like *\"said Mr. Bennet\"* or *\"replied his wife\"* are always `\"narration\"` segments, never merged into the surrounding dialogue. This lets TTS engines handle attribution in a natural voice separate from the speaking characters.\n\nWhen the speaker cannot be determined, `speaker` is `\"Unknown\"`.\n\n---\n\n## Pipeline stages\n\nThe pipeline runs in 7 stages. Each stage writes to `cache/\u003cbook_id\u003e/0N-\u003cname\u003e/` and is skipped on subsequent runs if its output already exists.\n\n| Stage | Name | Description |\n|-------|------|-------------|\n| 01 | Download | Fetches `pg\u003cid\u003e.txt` from Gutenberg with retry/backoff |\n| 02 | Discovery | Strips boilerplate, detects chapter boundaries, extracts metadata |\n| 03 | Chapter split | Extracts each chapter into a plain-text file |\n| 04 | Characters | LLM identifies all named characters and aliases |\n| 05 | Segmentation | LLM splits each chapter into narration/dialogue segments with speaker attribution |\n| 06 | Critic | Deterministic anchor-propagation pass + optional LLM quality review |\n| 07 | Assembly | Merges all chapters into the final JSON |\n\n### Resume and force-rerun\n\nEvery stage checks for its cached output before running. If you interrupt the pipeline (Ctrl+C), restarting the same command resumes from the interrupted chapter automatically.\n\nTo reprocess from a specific stage:\n\n```bash\n# Regenerate final JSON only\ngutenberg-reader 1342 --force-stage 7\n\n# Re-run critic pass and reassemble\ngutenberg-reader 1342 --force-stage 6\n\n# Start completely from scratch\ngutenberg-reader 1342 --force-stage 1\n```\n\n### Cache layout\n\n```\ncache/\n└── 1342/\n    ├── 01-raw/          book.txt\n    ├── 02-discovery/    discovery.json\n    ├── 03-chapters/     chapter-01.txt … chapter-61.txt\n    ├── 04-characters/   characters.json\n    ├── 05-segments/     chapter-01.json … chapter-61.json\n    ├── 06-critic/       chapter-01.json … chapter-61.json\n    └── 07-final/        1342.json\n```\n\n---\n\n## How attribution works\n\nBack-and-forth dialogue that lacks explicit attribution tags (e.g., *\"said Mr. Bennet\"*) is handled in two passes:\n\n1. **Anchor propagation (Stage 06, deterministic):** The pipeline locates every narration segment containing a speech verb adjacent to dialogue. It groups consecutive dialogue into *conversation chains*, then uses confirmed speakers as anchors and propagates via strict alternation for 2-character scenes. Characters present in a scene are detected from vocative name use inside dialogue (e.g., *\"My dear Mr. Bennet,\"* confirms Mr. Bennet is present).\n\n2. **LLM critic (Stage 06, optional):** A second LLM pass reviews the anchor-corrected segments for remaining attribution issues, name inconsistencies, and coverage gaps. Disable with `--no-critic` for faster processing.\n\n---\n\n## Limitations\n\n- Requires a locally running Ollama instance\n- Processing a full novel takes 15–90 minutes depending on model and hardware\n- Very long chapters (\u003e5,000 words) are chunked; chunk boundaries can occasionally split a dialogue exchange across two LLM calls\n- Books in languages other than English work if the model supports the language, but the speech-verb detection regex is English-only\n- Some books use non-standard chapter structures (Roman numerals, titled chapters, etc.) — the pipeline falls back to LLM-based chapter discovery when regex detection finds fewer than 2 chapters\n\n---\n\n## License\n\nMIT — see [LICENSE](LICENSE).\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fanders94%2Fgutenberg-reader","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fanders94%2Fgutenberg-reader","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fanders94%2Fgutenberg-reader/lists"}