{"id":17181892,"url":"https://github.com/clipperhouse/uax29","last_synced_at":"2026-02-01T19:02:19.135Z","repository":{"id":37007646,"uuid":"256002539","full_name":"clipperhouse/uax29","owner":"clipperhouse","description":"A tokenizer based on Unicode text segmentation (UAX #29), for Go. Split graphemes, words, sentences.","archived":false,"fork":false,"pushed_at":"2026-02-01T02:35:47.000Z","size":1183,"stargazers_count":98,"open_issues_count":1,"forks_count":5,"subscribers_count":3,"default_branch":"master","last_synced_at":"2026-02-01T03:29:30.258Z","etag":null,"topics":["go","golang","nlp","tokenization","tokenizer","uax29","unicode"],"latest_commit_sha":null,"homepage":"","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/clipperhouse.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":"AGENTS.md","dco":null,"cla":null}},"created_at":"2020-04-15T18:24:47.000Z","updated_at":"2026-02-01T02:35:50.000Z","dependencies_parsed_at":"2024-08-26T03:33:40.950Z","dependency_job_id":"bf99e24e-be2a-4478-95de-af4c7b14a8b7","html_url":"https://github.com/clipperhouse/uax29","commit_stats":{"total_commits":239,"total_committers":2,"mean_commits":119.5,"dds":0.004184100418409997,"last_synced_commit":"a2dc2931d37966b1c188ccab91777678e5d154ad"},"previous_names":[],"tags_count":54,"template":false,"template_full_name":null,"purl":"pkg:github/clipperhouse/uax29","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/clipperhouse%2Fuax29","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/clipperhouse%2Fuax29/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/clipperhouse%2Fuax29/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/clipperhouse%2Fuax29/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/clipperhouse","download_url":"https://codeload.github.com/clipperhouse/uax29/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/clipperhouse%2Fuax29/sbom","scorecard":{"id":291804,"data":{"date":"2025-08-11","repo":{"name":"github.com/clipperhouse/uax29","commit":"63e776e07ba8ca913273f57d6bfa60906f7e266a"},"scorecard":{"version":"v5.2.1-40-gf6ed084d","commit":"f6ed084d17c9236477efd66e5b258b9d4cc7b389"},"score":4.3,"checks":[{"name":"Dangerous-Workflow","score":10,"reason":"no dangerous workflow patterns detected","details":null,"documentation":{"short":"Determines if the project's GitHub Action workflows avoid dangerous patterns.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#dangerous-workflow"}},{"name":"Code-Review","score":0,"reason":"Found 0/21 approved changesets -- score normalized to 0","details":null,"documentation":{"short":"Determines if the project requires human code review before pull requests (aka merge requests) are merged.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#code-review"}},{"name":"Maintained","score":3,"reason":"4 commit(s) and 0 issue activity found in the last 90 days -- score normalized to 3","details":null,"documentation":{"short":"Determines if the project is \"actively maintained\".","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#maintained"}},{"name":"Packaging","score":-1,"reason":"packaging workflow not detected","details":["Warn: no GitHub/GitLab publishing workflow detected."],"documentation":{"short":"Determines if the project is published as a package that others can easily download, install, easily update, and uninstall.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#packaging"}},{"name":"Token-Permissions","score":0,"reason":"detected GitHub workflow tokens with excessive permissions","details":["Warn: no topLevel permission defined: .github/workflows/gofuzz.yml:1","Warn: no topLevel permission defined: .github/workflows/gotest.yml:1","Info: no jobLevel write permissions found"],"documentation":{"short":"Determines if the project's workflows follow the principle of least privilege.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#token-permissions"}},{"name":"Binary-Artifacts","score":10,"reason":"no binaries found in the repo","details":null,"documentation":{"short":"Determines if the project has generated executable (binary) artifacts in the source repository.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#binary-artifacts"}},{"name":"Pinned-Dependencies","score":0,"reason":"dependency not pinned by hash detected -- score normalized to 0","details":["Warn: GitHub-owned GitHubAction not pinned by hash: .github/workflows/gofuzz.yml:18: update your workflow using https://app.stepsecurity.io/secureworkflow/clipperhouse/uax29/gofuzz.yml/master?enable=pin","Warn: GitHub-owned GitHubAction not pinned by hash: .github/workflows/gofuzz.yml:20: update your workflow using https://app.stepsecurity.io/secureworkflow/clipperhouse/uax29/gofuzz.yml/master?enable=pin","Warn: GitHub-owned GitHubAction not pinned by hash: .github/workflows/gotest.yml:17: update your workflow using https://app.stepsecurity.io/secureworkflow/clipperhouse/uax29/gotest.yml/master?enable=pin","Warn: GitHub-owned GitHubAction not pinned by hash: .github/workflows/gotest.yml:22: update your workflow using https://app.stepsecurity.io/secureworkflow/clipperhouse/uax29/gotest.yml/master?enable=pin","Warn: GitHub-owned GitHubAction not pinned by hash: .github/workflows/gotest.yml:25: update your workflow using https://app.stepsecurity.io/secureworkflow/clipperhouse/uax29/gotest.yml/master?enable=pin","Info:   0 out of   5 GitHub-owned GitHubAction dependencies pinned"],"documentation":{"short":"Determines if the project has declared and pinned the dependencies of its build process.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#pinned-dependencies"}},{"name":"CII-Best-Practices","score":0,"reason":"no effort to earn an OpenSSF best practices badge detected","details":null,"documentation":{"short":"Determines if the project has an OpenSSF (formerly CII) Best Practices Badge.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#cii-best-practices"}},{"name":"Security-Policy","score":0,"reason":"security policy file not detected","details":["Warn: no security policy file detected","Warn: no security file to analyze","Warn: no security file to analyze","Warn: no security file to analyze"],"documentation":{"short":"Determines if the project has published a security policy.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#security-policy"}},{"name":"Fuzzing","score":10,"reason":"project is fuzzed","details":["Info: GoBuiltInFuzzer integration found: graphemes/fuzz_test.go:18","Info: GoBuiltInFuzzer integration found: graphemes/fuzz_test.go:63","Info: GoBuiltInFuzzer integration found: graphemes/fuzz_test.go:103","Info: GoBuiltInFuzzer integration found: phrases/fuzz_test.go:18","Info: GoBuiltInFuzzer integration found: phrases/fuzz_test.go:58","Info: GoBuiltInFuzzer integration found: phrases/fuzz_test.go:98","Info: GoBuiltInFuzzer integration found: sentences/fuzz_test.go:18","Info: GoBuiltInFuzzer integration found: sentences/fuzz_test.go:63","Info: GoBuiltInFuzzer integration found: sentences/fuzz_test.go:103","Info: GoBuiltInFuzzer integration found: words/fuzz_test.go:18","Info: GoBuiltInFuzzer integration found: words/fuzz_test.go:63","Info: GoBuiltInFuzzer integration found: words/fuzz_test.go:103"],"documentation":{"short":"Determines if the project uses fuzzing.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#fuzzing"}},{"name":"License","score":10,"reason":"license file detected","details":["Info: project has a license file: LICENSE:0","Info: FSF or OSI recognized license: MIT License: LICENSE:0"],"documentation":{"short":"Determines if the project has defined a license.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#license"}},{"name":"Signed-Releases","score":-1,"reason":"no releases found","details":null,"documentation":{"short":"Determines if the project cryptographically signs release artifacts.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#signed-releases"}},{"name":"Branch-Protection","score":0,"reason":"branch protection not enabled on development/release branches","details":["Warn: branch protection not enabled for branch 'master'"],"documentation":{"short":"Determines if the default and release branches are protected with GitHub's branch protection settings.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#branch-protection"}},{"name":"Vulnerabilities","score":10,"reason":"0 existing vulnerabilities detected","details":null,"documentation":{"short":"Determines if the project has open, known unfixed vulnerabilities.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#vulnerabilities"}},{"name":"SAST","score":0,"reason":"SAST tool is not run on all commits -- score normalized to 0","details":["Warn: 0 commits out of 16 are checked with a SAST tool"],"documentation":{"short":"Determines if the project uses static code analysis.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#sast"}}]},"last_synced_at":"2025-08-17T18:26:47.041Z","repository_id":37007646,"created_at":"2025-08-17T18:26:47.041Z","updated_at":"2025-08-17T18:26:47.041Z"},"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":28986365,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-02-01T18:17:03.387Z","status":"ssl_error","status_checked_at":"2026-02-01T18:16:57.287Z","response_time":56,"last_error":"SSL_read: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["go","golang","nlp","tokenization","tokenizer","uax29","unicode"],"created_at":"2024-10-15T00:35:29.396Z","updated_at":"2026-02-01T19:02:19.122Z","avatar_url":"https://github.com/clipperhouse.png","language":"Go","readme":"This package tokenizes (splits) words, sentences and graphemes, based on [Unicode text segmentation](https://unicode.org/reports/tr29/) (UAX #29), for Unicode 17. Details and usage are in the respective packages:\n\n[uax29/graphemes](https://github.com/clipperhouse/uax29/tree/master/graphemes)\n\n[uax29/words](https://github.com/clipperhouse/uax29/tree/master/words)\n\n[uax29/phrases](https://github.com/clipperhouse/uax29/tree/master/phrases)\n\n[uax29/sentences](https://github.com/clipperhouse/uax29/tree/master/sentences)\n\n### Why tokenize?\n\nAny time our code operates on individual words, we are tokenizing. Often, we do it ad hoc, such as splitting on spaces, which gives inconsistent results. The Unicode standard is better: it is multi-lingual, and handles punctuation, special characters, etc.\n\n### Uses\n\nThe uax29 module has 4 tokenizers. In decreasing granularity: sentences → phrases → words → graphemes. Words and graphemes are the most common uses.\n\nYou might use `words` for inverted indexes, full-text search, TF-IDF, BM25, embeddings, etc.\n\nIf you're doing embeddings, the definition of “meaningful unit” will depend on your application. You might choose sentences, phrases, words, or a combination.\n\n### Conformance\n\nWe use the official [Unicode test suites](https://unicode.org/reports/tr41/tr41-36.html#Tests29). Status:\n\n![Go](https://github.com/clipperhouse/uax29/actions/workflows/gotest.yml/badge.svg)\n\n## Quick start\n\n```\ngo get \"github.com/clipperhouse/uax29/v2/words\"\n```\n\n```go\nimport \"github.com/clipperhouse/uax29/v2/words\"\n\ntext := \"Hello, 世界. Nice dog! 👍🐶\"\n\ntokens := words.FromString(text)\n\nfor tokens.Next() {                     // Next() returns true until end of data\n\tfmt.Println(tokens.Value())         // Do something with the current token\n}\n```\n\n### See also\n\n[jargon](https://github.com/clipperhouse/jargon), a text pipelines package for CLI and Go, which consumes this package.\n\n### Prior art\n\n[blevesearch/segment](https://github.com/blevesearch/segment)\n\n[rivo/uniseg](https://github.com/rivo/uniseg)\n\n### Other language implementations\n\n[C#](https://github.com/clipperhouse/uax29.net) (also by me)\n\n[JavaScript](https://github.com/tc39/proposal-intl-segmenter)\n\n[Rust](https://unicode-rs.github.io/unicode-segmentation/unicode_segmentation/trait.UnicodeSegmentation.html)\n\n[Java](https://lucene.apache.org/core/6_5_0/core/org/apache/lucene/analysis/standard/StandardTokenizer.html)\n\n[Python](https://uniseg-python.readthedocs.io/en/latest/)\n","funding_links":[],"categories":[],"sub_categories":[],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fclipperhouse%2Fuax29","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fclipperhouse%2Fuax29","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fclipperhouse%2Fuax29/lists"}