{"id":15502870,"url":"https://github.com/craigtrim/fast-sentence-segment","last_synced_at":"2026-01-29T06:31:46.132Z","repository":{"id":99705771,"uuid":"526991564","full_name":"craigtrim/fast-sentence-segment","owner":"craigtrim","description":"Fast and Efficient Sentence Segmentation","archived":false,"fork":false,"pushed_at":"2025-12-29T17:54:45.000Z","size":12643,"stargazers_count":2,"open_issues_count":3,"forks_count":0,"subscribers_count":1,"default_branch":"master","last_synced_at":"2025-12-31T17:36:45.503Z","etag":null,"topics":["natural-language-processing","nlp","python","segmentation","sentence-segmentation","spacy","text-processing","text-segmentation"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/craigtrim.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null}},"created_at":"2022-08-20T17:14:40.000Z","updated_at":"2025-12-29T17:09:03.000Z","dependencies_parsed_at":"2023-11-16T03:11:34.002Z","dependency_job_id":"f748c9e3-c989-4751-85e5-802f74539eae","html_url":"https://github.com/craigtrim/fast-sentence-segment","commit_stats":{"total_commits":14,"total_committers":1,"mean_commits":14.0,"dds":0.0,"last_synced_commit":"a0cf43d1259277a6ab29b56a1bb47db1f91eadd9"},"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/craigtrim/fast-sentence-segment","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/craigtrim%2Ffast-sentence-segment","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/craigtrim%2Ffast-sentence-segment/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/craigtrim%2Ffast-sentence-segment/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/craigtrim%2Ffast-sentence-segment/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/craigtrim","download_url":"https://codeload.github.com/craigtrim/fast-sentence-segment/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/craigtrim%2Ffast-sentence-segment/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":28866983,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-01-29T05:56:06.453Z","status":"ssl_error","status_checked_at":"2026-01-29T05:55:57.668Z","response_time":59,"last_error":"SSL_connect returned=1 errno=0 peeraddr=140.82.121.6:443 state=error: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["natural-language-processing","nlp","python","segmentation","sentence-segmentation","spacy","text-processing","text-segmentation"],"created_at":"2024-10-02T09:11:25.636Z","updated_at":"2026-01-29T06:31:46.128Z","avatar_url":"https://github.com/craigtrim.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Fast Sentence Segmentation\n\n[![PyPI version](https://img.shields.io/pypi/v/fast-sentence-segment.svg)](https://pypi.org/project/fast-sentence-segment/)\n[![Python versions](https://img.shields.io/pypi/pyversions/fast-sentence-segment.svg)](https://pypi.org/project/fast-sentence-segment/)\n[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT)\n[![spaCy](https://img.shields.io/badge/spaCy-3.8-blue.svg)](https://spacy.io/)\n\nFast and efficient sentence segmentation using spaCy with surgical post-processing fixes. Handles complex edge cases like abbreviations (Dr., Mr., etc.), ellipses, quoted text, and multi-paragraph documents.\n\n## Why This Library?\n\n1. **Keep it local**: LLM API calls cost money and send your data to third parties. Run sentence segmentation entirely on your machine.\n2. **spaCy perfected**: spaCy is a great local model, but it makes mistakes. This library fixes most of spaCy's shortcomings.\n\n## Features\n\n- **Paragraph-aware segmentation**: Returns sentences grouped by paragraph\n- **Abbreviation handling**: Correctly handles \"Dr.\", \"Mr.\", \"etc.\", \"p.m.\", \"a.m.\" without false splits\n- **Ellipsis preservation**: Keeps `...` intact while detecting sentence boundaries\n- **Question/exclamation splitting**: Properly splits on `?` and `!` followed by capital letters\n- **Cached processing**: LRU cache for repeated text processing\n- **Flexible output**: Nested lists (by paragraph) or flattened list of sentences\n- **Bullet point \u0026 numbered list normalization**: Cleans common list formats\n- **CLI tool**: Command-line interface for quick segmentation\n\n## Installation\n\n```bash\npip install fast-sentence-segment\n```\n\nAfter installation, download the spaCy model:\n\n```bash\npython -m spacy download en_core_web_sm\n```\n\n## Quick Start\n\n```python\nfrom fast_sentence_segment import segment_text\n\ntext = \"Do you like Dr. Who? I prefer Dr. Strange! Mr. T is also cool.\"\n\nresults = segment_text(text, flatten=True)\n```\n\n```json\n[\n  \"Do you like Dr. Who?\",\n  \"I prefer Dr. Strange!\",\n  \"Mr. T is also cool.\"\n]\n```\n\nNotice how \"Dr. Who?\" stays together as a single sentence—the library correctly recognizes that a title followed by a single-word name ending in `?` or `!` is a name reference, not a sentence boundary.\n\n## Usage\n\n### Basic Segmentation\n\nThe `segment_text` function returns a list of lists, where each inner list represents a paragraph containing its sentences:\n\n```python\nfrom fast_sentence_segment import segment_text\n\ntext = \"\"\"Gandalf spoke softly. \"All we have to decide is what to do with the time given us.\"\n\nFrodo nodded. The weight of the Ring pressed against his chest.\"\"\"\n\nresults = segment_text(text)\n```\n\n```json\n[\n  [\n    \"Gandalf spoke softly.\",\n    \"\\\"All we have to decide is what to do with the time given us.\\\".\"\n  ],\n  [\n    \"Frodo nodded.\",\n    \"The weight of the Ring pressed against his chest.\"\n  ]\n]\n```\n\n### Flattened Output\n\nIf you don't need paragraph boundaries, use the `flatten` parameter:\n\n```python\ntext = \"At 9 a.m. the hobbits set out. By 3 p.m. they reached Rivendell. Mr. Frodo was exhausted.\"\n\nresults = segment_text(text, flatten=True)\n```\n\n```json\n[\n  \"At 9 a.m. the hobbits set out.\",\n  \"By 3 p.m. they reached Rivendell.\",\n  \"Mr. Frodo was exhausted.\"\n]\n```\n\n### Direct Segmenter Access\n\nFor more control, use the `Segmenter` class directly:\n\n```python\nfrom fast_sentence_segment import Segmenter\n\nsegmenter = Segmenter()\nresults = segmenter.input_text(\"Your text here.\")\n```\n\n### Command Line Interface\n\nSegment text directly from the terminal:\n\n```bash\n# Direct text input\necho \"Have you seen Dr. Who? It's brilliant!\" | segment\n```\n\n```\nHave you seen Dr. Who?\nIt's brilliant!\n```\n\n```bash\n# Numbered output\nsegment -n \"Gandalf paused... You shall not pass! The Balrog roared.\"\n```\n\n```\n1. Gandalf paused...\n2. You shall not pass!\n3. The Balrog roared.\n```\n\n```bash\n# From file\nsegment -f silmarillion.txt\n```\n\n## API Reference\n\n| Function | Parameters | Returns | Description |\n|----------|------------|---------|-------------|\n| `segment_text()` | `input_text: str`, `flatten: bool = False` | `list` | Main entry point for segmentation |\n| `Segmenter.input_text()` | `input_text: str` | `list[list[str]]` | Cached paragraph-aware segmentation |\n\n### CLI Options\n\n| Option | Description |\n|--------|-------------|\n| `text` | Text to segment (positional argument) |\n| `-f, --file` | Read text from file |\n| `-n, --numbered` | Number output lines |\n\n## Why Nested Lists?\n\nThe segmentation process preserves document structure by segmenting into both paragraphs and sentences. Each outer list represents a paragraph, and each inner list contains that paragraph's sentences. This is useful for:\n\n- Document structure analysis\n- Paragraph-level processing\n- Maintaining original text organization\n\nUse `flatten=True` when you only need sentences without paragraph context.\n\n## Requirements\n\n- Python 3.9+\n- spaCy 3.8+\n- en_core_web_sm spaCy model\n\n## How It Works\n\nThis library uses spaCy for initial sentence segmentation, then applies surgical post-processing fixes for cases where spaCy's default behavior is incorrect:\n\n1. **Pre-processing**: Normalize numbered lists, preserve ellipses with placeholders\n2. **spaCy segmentation**: Use spaCy's sentence boundary detection\n3. **Post-processing**: Split on abbreviation boundaries, handle `?`/`!` + capital patterns\n4. **Denormalization**: Restore placeholders to original text\n\n## License\n\nMIT License - see [LICENSE](LICENSE) for details.\n\n## Contributing\n\nContributions are welcome! Please feel free to submit a Pull Request.\n\n1. Fork the repository\n2. Create your feature branch (`git checkout -b feature/amazing-feature`)\n3. Run tests (`make test`)\n4. Commit your changes\n5. Push to the branch\n6. Open a Pull Request\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcraigtrim%2Ffast-sentence-segment","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fcraigtrim%2Ffast-sentence-segment","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcraigtrim%2Ffast-sentence-segment/lists"}