{"id":13413562,"url":"https://github.com/neurosnap/sentences","last_synced_at":"2025-05-16T10:06:25.945Z","repository":{"id":1101944,"uuid":"40334054","full_name":"neurosnap/sentences","owner":"neurosnap","description":"A multilingual command line sentence tokenizer in Golang","archived":false,"fork":false,"pushed_at":"2024-02-28T04:28:03.000Z","size":16061,"stargazers_count":450,"open_issues_count":5,"forks_count":38,"subscribers_count":13,"default_branch":"main","last_synced_at":"2025-05-16T10:06:14.805Z","etag":null,"topics":["cli","sentence-tokenizer","sentences","tokenizer"],"latest_commit_sha":null,"homepage":"https://sentences-231000.appspot.com/","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/neurosnap.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGES.md","contributing":null,"funding":null,"license":"LICENSE.md","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2015-08-07T01:08:20.000Z","updated_at":"2025-05-12T21:36:32.000Z","dependencies_parsed_at":"2024-02-28T05:41:43.231Z","dependency_job_id":null,"html_url":"https://github.com/neurosnap/sentences","commit_stats":{"total_commits":209,"total_committers":10,"mean_commits":20.9,"dds":"0.17703349282296654","last_synced_commit":"7d02644a411757cc6de9341f6e4809774eb2e729"},"previous_names":["neurosnap/go-sentences"],"tags_count":16,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/neurosnap%2Fsentences","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/neurosnap%2Fsentences/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/neurosnap%2Fsentences/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/neurosnap%2Fsentences/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/neurosnap","download_url":"https://codeload.github.com/neurosnap/sentences/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":254509475,"owners_count":22082891,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["cli","sentence-tokenizer","sentences","tokenizer"],"created_at":"2024-07-30T20:01:43.241Z","updated_at":"2025-05-16T10:06:25.926Z","avatar_url":"https://github.com/neurosnap.png","language":"Go","funding_links":[],"categories":["Natural Language Processing","Go","自然語言處理","自然语言处理","Relational Databases","Bot Building","Microsoft Office","\u003cspan id=\"自然语言处理-natural-language-processing\"\u003e自然语言处理 Natural Language Processing\u003c/span\u003e"],"sub_categories":["Uncategorized","Tokenizers","高級控制台界面","分词器","Strings","Tools","[Tools](#tools-1)","Speech Recognition","交流","暂未分类","Advanced Console UIs","暂未分类这些库被放在这里是因为其他类别似乎都不适合。","高级控制台界面","\u003cspan id=\"高级控制台用户界面-advanced-console-uis\"\u003e高级控制台用户界面 Advanced Console UIs\u003c/span\u003e"],"readme":"[![release](https://github.com/neurosnap/sentences/actions/workflows/release.yml/badge.svg)](https://github.com/neurosnap/sentences/actions/workflows/release.yml)\n[![GODOC](https://godoc.org/github.com/nathany/looper?status.svg)](https://godoc.org/github.com/neurosnap/sentences)\n![MIT](https://img.shields.io/packagist/l/doctrine/orm.svg)\n[![Go Report Card](https://goreportcard.com/badge/github.com/neurosnap/sentences)](https://goreportcard.com/report/github.com/neurosnap/sentences)\n\n# Sentences - A command line sentence tokenizer\n\nThis command line utility will convert a blob of text into a list of sentences.\n\n* [Demo](https://sentences-231000.appspot.com/)\n* [Docs](https://godoc.org/github.com/neurosnap/sentences)\n\n## Features\n\n* Supports multiple languages (english, czech, dutch, estonian, finnish,\n  german, greek, italian, norwegian, polish, portuguese, slovene, and turkish)\n* Zero dependencies\n* Extendable\n* Fast\n\n## Install\n\n### arch\n\n[aur](https://aur.archlinux.org/packages/sentences-bin)\n\n### mac\n\n```\nbrew tap neurosnap/sentences\nbrew install sentences\n```\n\n### other\n\nOr you can find the pre-built binaries on [the github\nreleases page](https://github.com/neurosnap/sentences/releases).\n\n### using golang\n\n```\ngo get github.com/neurosnap/sentences\ngo install github.com/neurosnap/sentences/cmd/sentences\n```\n\n## Command\n\n![Command line](sentences.gif?raw=true)\n\n## Get it\n\n```\ngo get github.com/neurosnap/sentences\n```\n\n## Use it\n\n```Go\nimport (\n    \"fmt\"\n    \"os\"\n\n    \"github.com/neurosnap/sentences\"\n)\n\nfunc main() {\n    text := `A perennial also-ran, Stallings won his seat when longtime lawmaker David Holmes\n    died 11 days after the filing deadline. Suddenly, Stallings was a shoo-in, not\n    the long shot. In short order, the Legislature attempted to pass a law allowing\n    former U.S. Rep. Carolyn Cheeks Kilpatrick to file; Stallings challenged the\n    law in court and won. Kilpatrick mounted a write-in campaign, but Stallings won.`\n\n    // download the training data from this repo (./data) and save it somewhere\n    b, _ := os.ReadFile(\"./path/to/english.json\")\n\n    // load the training data\n    training, _ := sentences.LoadTraining(b)\n\n    // create the default sentence tokenizer\n    tokenizer := sentences.NewSentenceTokenizer(training)\n    sentences := tokenizer.Tokenize(text)\n\n    for _, s := range sentences {\n        fmt.Println(s.Text)\n    }\n}\n```\n\n## English\n\nThis package attempts to fix some problems I noticed for english.\n\n```Go\nimport (\n    \"fmt\"\n\n    \"github.com/neurosnap/sentences/english\"\n)\n\nfunc main() {\n    text := \"Hi there. Does this really work?\"\n\n    tokenizer, err := english.NewSentenceTokenizer(nil)\n    if err != nil {\n        panic(err)\n    }\n\n    sentences := tokenizer.Tokenize(text)\n    for _, s := range sentences {\n        fmt.Println(s.Text)\n    }\n}\n```\n\n## Contributing\n\nI need help maintaining this library.  If you are interested in contributing\nto this library then please start by looking at the [golden-rules](https://github.com/neurosnap/sentences/tree/golden-rule) branch which\ntests the [Golden Rules](https://github.com/diasks2/pragmatic_segmenter/blob/master/README.md#the-golden-rules)\nfor english sentence tokenization created by the [Pragmatic Segmenter](https://github.com/diasks2/pragmatic_segmenter)\nlibrary.\n\nCreate an issue for a particular failing test and submit an issue/PR.\n\nI'm happy to help anyone willing to contribute.\n\n## Customize\n\n`sentences` was built around composability, most major components of this package\ncan be extended.\n\nEager to make ad-hoc changes but don't know how to start?\nHave a look at `github.com/neurosnap/sentences/english` for a solid example.\n\n## Notice\n\nI have not tested this tokenizer in any other language besides English.  By default\nthe command line utility loads english. I welcome anyone willing to test the\nother languages to submit updates as needed.\n\nA primary goal for this package is to be multilingual so I'm willing to help in\nany way possible.\n\nThis library is a port of the [nltk's](http://www.nltk.org) punkt tokenizer.\n\n## A Punkt Tokenizer\n\nAn unsupervised multilingual sentence boundary detection library for golang.\nThe way the punkt system accomplishes this goal is through training the tokenizer\nwith text in that given language.  Once the likelihoods of abbreviations, collocations,\nand sentence starters are determined, finding sentence boundaries becomes easier.\n\nThere are many problems that arise when tokenizing text into sentences, the primary\nissue being abbreviations.  The punkt system attempts to determine whether a  word\nis an abbreviation, an end to a sentence, or even both through training the system with text\nin the given language.  The punkt system incorporates both token- and type-based\nanalysis on the text through two different phases of annotation.\n\n[Unsupervised multilingual sentence boundary detection](http://citeseerx.ist.psu.edu/viewdoc/download;jsessionid=BAE5C34E5C3B9DC60DFC4D93B85D8BB1?doi=10.1.1.85.5017\u0026rep=rep1\u0026type=pdf)\n\n## Performance\n\nUsing [Brown Corpus](http://www.hit.uib.no/icame/brown/bcm.html) which is annotated American English\ntext, we compare this package with other libraries across multiple programming languages.\n\n|Library    | Avg Speed (s, 10 runs) | Accuracy (%)\n|:----------|:----------------------:|:-----------:\n| Sentences | 1.96                   | 98.95\n| NLTK      | 5.22                   | 99.21\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fneurosnap%2Fsentences","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fneurosnap%2Fsentences","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fneurosnap%2Fsentences/lists"}