{"id":16689905,"url":"https://github.com/eddieantonio/unicode-default-word-boundary","last_synced_at":"2025-10-14T10:42:23.683Z","repository":{"id":48013039,"uuid":"184778927","full_name":"eddieantonio/unicode-default-word-boundary","owner":"eddieantonio","description":"Split words with Unicode's default word boundary specification","archived":false,"fork":false,"pushed_at":"2024-09-12T10:11:02.000Z","size":677,"stargazers_count":13,"open_issues_count":0,"forks_count":1,"subscribers_count":2,"default_branch":"master","last_synced_at":"2025-10-05T04:21:02.194Z","etag":null,"topics":["internationalization","split","text","unicode","word-boundary","word-break"],"latest_commit_sha":null,"homepage":"","language":"TypeScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/eddieantonio.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2019-05-03T15:26:46.000Z","updated_at":"2025-07-20T16:19:54.000Z","dependencies_parsed_at":"2023-02-01T10:02:04.892Z","dependency_job_id":"239372fa-5bc1-4eba-a996-5901e44062e2","html_url":"https://github.com/eddieantonio/unicode-default-word-boundary","commit_stats":{"total_commits":87,"total_committers":3,"mean_commits":29.0,"dds":"0.12643678160919536","last_synced_commit":"0f22e684fd306f21f72bec50e0c506369f16e3c7"},"previous_names":[],"tags_count":10,"template":false,"template_full_name":null,"purl":"pkg:github/eddieantonio/unicode-default-word-boundary","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/eddieantonio%2Funicode-default-word-boundary","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/eddieantonio%2Funicode-default-word-boundary/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/eddieantonio%2Funicode-default-word-boundary/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/eddieantonio%2Funicode-default-word-boundary/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/eddieantonio","download_url":"https://codeload.github.com/eddieantonio/unicode-default-word-boundary/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/eddieantonio%2Funicode-default-word-boundary/sbom","scorecard":{"id":366098,"data":{"date":"2025-08-11","repo":{"name":"github.com/eddieantonio/unicode-default-word-boundary","commit":"ef3d1dfef70674a058ca8f963761f1e2cad216d7"},"scorecard":{"version":"v5.2.1-40-gf6ed084d","commit":"f6ed084d17c9236477efd66e5b258b9d4cc7b389"},"score":3.4,"checks":[{"name":"Binary-Artifacts","score":10,"reason":"no binaries found in the repo","details":null,"documentation":{"short":"Determines if the project has generated executable (binary) artifacts in the source repository.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#binary-artifacts"}},{"name":"Code-Review","score":0,"reason":"Found 0/25 approved changesets -- score normalized to 0","details":null,"documentation":{"short":"Determines if the project requires human code review before pull requests (aka merge requests) are merged.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#code-review"}},{"name":"Maintained","score":0,"reason":"0 commit(s) and 0 issue activity found in the last 90 days -- score normalized to 0","details":null,"documentation":{"short":"Determines if the project is \"actively maintained\".","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#maintained"}},{"name":"Dangerous-Workflow","score":10,"reason":"no dangerous workflow patterns detected","details":null,"documentation":{"short":"Determines if the project's GitHub Action workflows avoid dangerous patterns.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#dangerous-workflow"}},{"name":"Packaging","score":-1,"reason":"packaging workflow not detected","details":["Warn: no GitHub/GitLab publishing workflow detected."],"documentation":{"short":"Determines if the project is published as a package that others can easily download, install, easily update, and uninstall.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#packaging"}},{"name":"Token-Permissions","score":0,"reason":"detected GitHub workflow tokens with excessive permissions","details":["Warn: no topLevel permission defined: .github/workflows/node.js.yml:1","Info: no jobLevel write permissions found"],"documentation":{"short":"Determines if the project's workflows follow the principle of least privilege.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#token-permissions"}},{"name":"Pinned-Dependencies","score":3,"reason":"dependency not pinned by hash detected -- score normalized to 3","details":["Warn: GitHub-owned GitHubAction not pinned by hash: .github/workflows/node.js.yml:23: update your workflow using https://app.stepsecurity.io/secureworkflow/eddieantonio/unicode-default-word-boundary/node.js.yml/master?enable=pin","Warn: GitHub-owned GitHubAction not pinned by hash: .github/workflows/node.js.yml:25: update your workflow using https://app.stepsecurity.io/secureworkflow/eddieantonio/unicode-default-word-boundary/node.js.yml/master?enable=pin","Info:   0 out of   2 GitHub-owned GitHubAction dependencies pinned","Info:   1 out of   1 npmCommand dependencies pinned"],"documentation":{"short":"Determines if the project has declared and pinned the dependencies of its build process.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#pinned-dependencies"}},{"name":"CII-Best-Practices","score":0,"reason":"no effort to earn an OpenSSF best practices badge detected","details":null,"documentation":{"short":"Determines if the project has an OpenSSF (formerly CII) Best Practices Badge.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#cii-best-practices"}},{"name":"Security-Policy","score":0,"reason":"security policy file not detected","details":["Warn: no security policy file detected","Warn: no security file to analyze","Warn: no security file to analyze","Warn: no security file to analyze"],"documentation":{"short":"Determines if the project has published a security policy.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#security-policy"}},{"name":"Fuzzing","score":0,"reason":"project is not fuzzed","details":["Warn: no fuzzer integrations found"],"documentation":{"short":"Determines if the project uses fuzzing.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#fuzzing"}},{"name":"License","score":10,"reason":"license file detected","details":["Info: project has a license file: LICENSE:0","Info: FSF or OSI recognized license: MIT License: LICENSE:0"],"documentation":{"short":"Determines if the project has defined a license.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#license"}},{"name":"Signed-Releases","score":-1,"reason":"no releases found","details":null,"documentation":{"short":"Determines if the project cryptographically signs release artifacts.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#signed-releases"}},{"name":"Branch-Protection","score":0,"reason":"branch protection not enabled on development/release branches","details":["Warn: branch protection not enabled for branch 'master'"],"documentation":{"short":"Determines if the default and release branches are protected with GitHub's branch protection settings.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#branch-protection"}},{"name":"SAST","score":0,"reason":"SAST tool is not run on all commits -- score normalized to 0","details":["Warn: 0 commits out of 12 are checked with a SAST tool"],"documentation":{"short":"Determines if the project uses static code analysis.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#sast"}},{"name":"Vulnerabilities","score":8,"reason":"2 existing vulnerabilities detected","details":["Warn: Project is vulnerable to: GHSA-968p-4wvh-cqc8","Warn: Project is vulnerable to: GHSA-952p-6rrq-rcjv"],"documentation":{"short":"Determines if the project has open, known unfixed vulnerabilities.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#vulnerabilities"}}]},"last_synced_at":"2025-08-18T11:52:38.178Z","repository_id":48013039,"created_at":"2025-08-18T11:52:38.178Z","updated_at":"2025-08-18T11:52:38.178Z"},"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":279018785,"owners_count":26086452,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-10-14T02:00:06.444Z","response_time":60,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["internationalization","split","text","unicode","word-boundary","word-break"],"created_at":"2024-10-12T15:49:44.631Z","updated_at":"2025-10-14T10:42:23.667Z","avatar_url":"https://github.com/eddieantonio.png","language":"TypeScript","funding_links":[],"categories":[],"sub_categories":[],"readme":"Unicode Default Word Boundary\n=============================\n\n[![Build status](https://github.com/eddieantonio/unicode-default-word-boundary/actions/workflows/node.js.yml/badge.svg)](https://github.com/eddieantonio/unicode-default-word-boundary/actions/workflows/node.js.yml)\n[![npm](https://img.shields.io/npm/v/unicode-default-word-boundary.svg)](https://www.npmjs.com/package/unicode-default-word-boundary)\n\nImplements the [Unicode UAX #29 §4.1 default word boundary\nspecification][defaultwb], for finding **word breaks** in **multilingual\ntext**.\n\nUse this to split words in text! Using UAX #29 is a lot smarter than the\n`\\b` word boundary in JavaScript's regular expressions! Note that\ncharacter classes like `\\b`, `\\w`, `\\d` [only work on ASCII\ncharacters][mdnregexp].\n\n\nUsage\n-----\n\nImport the module and use the `split()` function:\n\n```js\nconst split = require('unicode-default-word-boundary').split;\n\nconsole.log(split(`The quick (“brown”) fox can’t jump 32.3 feet, right?`));\n```\n\nOutput:\n\n    [ 'The', 'quick', '(', '“', 'brown', '”', ')', 'fox', 'can’t', 'jump', '32.3', 'feet', ',', 'right', '?' ]\n\n\nBut that's not all! Try it with non-English text, like Russian:\n\n```javascript\nsplit(`В чащах юга жил бы цитрус? Да, но фальшивый экземпляр!`)\n```\n\n    [ 'В', 'чащах', 'юга', 'жил', 'бы', 'цитрус', '?', 'Да', ',', 'но', 'фальшивый', 'экземпляр', '!' ]\n\n...Hebrew:\n\n```javascript\nsplit(`איך בלש תפס גמד רוצח עז קטנה?`);\n```\n\n    [ 'איך', 'בלש', 'תפס', 'גמד', 'רוצח', 'עז', 'קטנה', '?' ]\n\n...[nêhiyawêwin][]:\n\n```javascript\nsplit(`ᑕᐻ ᒥᔪ ᑭᓯᑲᐤ ᐊᓄᐦᐨ᙮`);\n```\n\n    [ 'ᑕᐻ', 'ᒥᔪ ᑭᓯᑲᐤ', 'ᐊᓄᐦᐨ', '᙮' ]\n\n...and many more!\n\nMore advanced use cases will want to use the `findSpans()` or the\n`findBoundaries()` function.\n\n\nWhat doesn't work\n-----------------\n\nLanguages that do not have obvious word breaks, such as Chinese,\nJapanese, Thai, Lao, and Khmer. You'll need to use statistical or\ndictionary-based approaches to split words in these languages.\n\n\nAPI Documentation\n-----------------\n\nThe following functions make up the primary API:\n\n### `split(text: string): string[]`\n\n`split()` splits the text at word boundaries, returning an array of all\n\"words\" from the text that contain characters other than whitespace.\n\nSee above for examples.\n\n\n### `findSpans(text: string): Iterable\u003cBasicSpan\u003e`\n\n`findSpans()` is a generator that yields successive _basic spans_ from\nthe text. A basic span is a chunk of text that is guaranteed to\nstart at a word boundary and end at the next word boundary. In other\nwords, basic spans are _indivisible_ in that there are no word\nboundaries contained within a basic span.\n\nA basic span has the following properties:\n\n```typescript\ninterface BasicSpan {\n    /** Where the span starts, relative to the input text. */\n    start: number;\n    /** At what index does the **next** span begin. */\n    end: number;\n    /** How many characters are in this span. */\n    length: number;\n    /** The text contained within this span. */\n    text: string;\n}\n```\n\nNote that unlike, `split()`, `findSpans()` **does** yield spans that\ncontain whitespace.\n\n#### Example\n\n`Array.from(findSpans(\"Hello, world🌎!\"))`\n\nWill yield spans with the following properties:\n\n```javascript\n[ { start: 0, end: 5, length: 5, text: 'Hello' },\n  { start: 5, end: 6, length: 1, text: ',' },\n  { start: 6, end: 7, length: 1, text: ' ' },\n  { start: 7, end: 12, length: 5, text: 'world' },\n  { start: 12, end: 14, length: 2, text: '🌎' },\n  { start: 14, end: 15, length: 1, text: '!' } ]\n```\n\n**N.B.**: `findSpans()` may _not_ yield plain JavaScript objects, as\nshown above. The objects that `findSpans()` yield will adhere to the\n`BasicSpan` interface, however what `findSpans()` actually yields may\ndiffer from simple objects.\n\n\n### `findBoundaries(text: string): Generator\u003cnumber, void, void\u003e`\n\n`findBoundaries()` is like `findSpans()` except it yields the _index_ of\neach successive word boundary. Anecdotally, using this function directly\nmay be faster than generating spans objects with `findSpans()`.\n\n\nContributing and Maintaining\n----------------------------\n\nWhen maintaining this package, you might notice something strange.\n`index.ts` depends on `./src/gen/WordBreakProperty.ts`, but this file\ndoes not exist! It is a **generated** file, created by reading Unicode\nproperty data files, [downloaded from Unicode's website][unicodefiles].\nThese data files have been compressed and committed to this repository\nin `libexec/`:\n\n    libexec/\n    libexec/\n    ├── WordBreakProperty-15.1.0.txt.gz\n    ├── compile-word-break.js\n    └── emoji-data-15.1.0.txt.gz\n\n\n**Note that `compile-word-break.js` actually creates\n`./src/gen/WordBreakProperty.ts`!**\n\n\n### How to generate `./src/gen/WordBreakProperty.ts`\n\nWhen you have _just_ cloned the repository, this file will be generated\nwhen you run `npm install`:\n\n    npm install\n\nIf you want to regenerate it afterwards, you can run the build script:\n\n    npm run build\n\n\n### Benchmarking\n\nTo run the benchmarks, you can run the following:\n\n    npm run benchmarks\n\nIf you want to compare the current implementation with a new\nimplementation, what I do is create a new working tree called `opt/`:\n\n    git worktree add -b «NEW-BRANCH-NAME» opt\n\nThen, I make changes in the working tree inside `opt/`, **compile\nand run the tests**, then, in the main working tree, I run the\nbenchmarks:\n\n    cd opt/\n    npm install\n    vim       # do whatever you need to do here\n    npm test  # this also compiles the TypeScript\n    cd ..\n    npm run benchmarks\n\n\nLicense\n-------\n\nTypeScript implementation © 2019 National Research Council Canada,\n© 2024 Eddie Antonio Santos. MIT Licensed.\n\nThe algorithm comes from [UAX #29: Unicode Text Segmentation, an\nintegral part of the Unicode Standard, version 15.1][uax29].\n\n[defaultwb]: https://unicode.org/reports/tr29/#Default_Word_Boundaries\n[mdnregexp]: https://developer.mozilla.org/en-US/docs/Web/JavaScript/Guide/Regular_Expressions/Character_Classes#Types\n[nêhiyawêwin]: https://en.wikipedia.org/wiki/Plains_Cree\n[uax29]: https://unicode.org/reports/tr29/\n[unicodefiles]: https://unicode.org/reports/tr41/tr41-24.html\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Feddieantonio%2Funicode-default-word-boundary","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Feddieantonio%2Funicode-default-word-boundary","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Feddieantonio%2Funicode-default-word-boundary/lists"}