{"id":38085292,"url":"https://github.com/dev-pi2pie/word-counter","last_synced_at":"2026-02-17T16:02:35.463Z","repository":{"id":332845405,"uuid":"1091528528","full_name":"dev-pi2pie/word-counter","owner":"dev-pi2pie","description":"Locale-aware word counter CLI and library powered by `Intl.Segmenter` with automatic script detection and per-locale breakdowns.","archived":false,"fork":false,"pushed_at":"2026-02-15T20:15:24.000Z","size":221,"stargazers_count":0,"open_issues_count":3,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2026-02-15T20:38:02.191Z","etag":null,"topics":["cli","intl-segmenter","language-detection","nodejs","text-processing","typescript","unicode","word-count","word-counter"],"latest_commit_sha":null,"homepage":"","language":"TypeScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/dev-pi2pie.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":"AGENTS.md","dco":null,"cla":null}},"created_at":"2025-11-07T06:21:45.000Z","updated_at":"2026-02-15T20:14:15.000Z","dependencies_parsed_at":null,"dependency_job_id":null,"html_url":"https://github.com/dev-pi2pie/word-counter","commit_stats":null,"previous_names":["dev-pi2pie/word-counter"],"tags_count":18,"template":false,"template_full_name":null,"purl":"pkg:github/dev-pi2pie/word-counter","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dev-pi2pie%2Fword-counter","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dev-pi2pie%2Fword-counter/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dev-pi2pie%2Fword-counter/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dev-pi2pie%2Fword-counter/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/dev-pi2pie","download_url":"https://codeload.github.com/dev-pi2pie/word-counter/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dev-pi2pie%2Fword-counter/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":29549223,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-02-17T14:33:00.708Z","status":"ssl_error","status_checked_at":"2026-02-17T14:32:58.657Z","response_time":100,"last_error":"SSL_connect returned=1 errno=0 peeraddr=140.82.121.5:443 state=error: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["cli","intl-segmenter","language-detection","nodejs","text-processing","typescript","unicode","word-count","word-counter"],"created_at":"2026-01-16T20:52:02.544Z","updated_at":"2026-02-17T16:02:35.457Z","avatar_url":"https://github.com/dev-pi2pie.png","language":"TypeScript","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Word Counter\n\nLocale-aware word counting powered by the Web API [`Intl.Segmenter`](https://developer.mozilla.org/docs/Web/JavaScript/Reference/Global_Objects/Intl/Segmenter). The script automatically detects the primary writing system for each portion of the input, segments the text with matching BCP 47 locale tags, and reports word totals per locale.\n\n## How It Works\n\n- The runtime inspects each character's Unicode script to infer its likely locale tag (e.g., `und-Latn`, `zh-Hani`, `ja`).\n- Adjacent characters that share the same locale tag are grouped into a chunk.\n- Each chunk is counted with `Intl.Segmenter` at `granularity: \"word\"`, caching segmenters to avoid re-instantiation.\n- Per-locale counts are summed into a overall total and printed to stdout.\n\n## Locale vs Language Code\n\n- Output keeps the field name `locale` for compatibility.\n- In this project, locale values are BCP 47 tags and are often language/script focused (for example: `en`, `und-Latn`, `zh-Hani`) rather than region-specific tags (for example: `en-US`, `zh-TW`).\n- Default detection prefers language/script tags to avoid incorrect region assumptions.\n- You can still provide region-specific locale tags through hint flags when needed.\n\n## Installation\n\n### For Development\n\nClone the repository and set up locally:\n\n```bash\ngit clone https://github.com/dev-pi2pie/word-counter.git\ncd word-counter\nbun install\nbun run build\nnpm link\n```\n\nAfter linking, you can use the `word-counter` command globally:\n\n```bash\nword-counter \"Hello 世界 안녕\"\n```\n\nTo use the linked package inside another project:\n\n```bash\nnpm link @dev-pi2pie/word-counter\n```\n\nTo uninstall the global link:\n\n```bash\nnpm unlink --global @dev-pi2pie/word-counter\n```\n\n### From npm Registry (npmjs.com)\n\n```bash\nnpm install -g @dev-pi2pie/word-counter@latest\n```\n\n## Usage\n\nOnce installed (via `npm link` or the npm registry), you can use the CLI directly:\n\n```bash\nword-counter \"Hello 世界 안녕\"\n```\n\nAlternatively, run the built CLI with Node:\n\n```bash\nnode dist/esm/bin.mjs \"Hello 世界 안녕\"\n```\n\nYou can also pipe text:\n\n```bash\necho \"こんにちは world مرحبا\" | word-counter\n```\n\nHint a locale tag for ambiguous Latin text (ASCII-heavy content):\n\n```bash\nword-counter --latin-language en \"Hello world\"\nword-counter --latin-tag en \"Hello world\"\n```\n\nHint a locale tag for Han text fallback:\n\n```bash\nword-counter --han-language zh-Hant \"漢字測試\"\nword-counter --han-tag zh-Hans \"汉字测试\"\n```\n\nCollect non-word segments (emoji, symbols, punctuation):\n\n```bash\nword-counter --non-words \"Hi 👋, world!\"\n```\n\nWhen enabled, `total` includes words + non-words (emoji, symbols, punctuation).\n\nOr read from a file:\n\n```bash\nword-counter --path ./fixtures/sample.txt\n```\n\n`--path` accepts any readable text-like file, including empty or whitespace-only files.\nSuch files are treated as valid inputs and contribute zero words by default.\n\n### Batch Counting\n\nProcess multiple files by repeating `--path`:\n\n```bash\nword-counter --path ./docs/a.md --path ./docs/b.txt\n```\n\nPass a directory path to scan files recursively (default):\n\n```bash\nword-counter --path ./examples/test-case-multi-files-support\n```\n\nShow per-file results plus merged summary:\n\n```bash\nword-counter --path ./examples/test-case-multi-files-support --per-file\n```\n\nRestrict directory scanning extensions:\n\n```bash\nword-counter --path ./examples/test-case-multi-files-support --include-ext .md,.mdx\nword-counter --path ./examples/test-case-multi-files-support --include-ext .md,.txt --exclude-ext .txt\n```\n\nSkip diagnostics are debug-gated. By default, skipped-file details are hidden.\nUse `--debug` to print skipped-file diagnostics to `stderr`:\n\n```bash\nword-counter --path ./examples/test-case-multi-files-support --debug\n```\n\n## Library Usage\n\nThe package exports can be used after installing from the npm registry or linking locally with `npm link`.\n\n### ESM\n\n```js\nimport wordCounter, {\n  countCharsForLocale,\n  countWordsForLocale,\n  countSections,\n  parseMarkdown,\n  segmentTextByLocale,\n  showSingularOrPluralWord,\n} from \"@dev-pi2pie/word-counter\";\n\nwordCounter(\"Hello world\", { latinLanguageHint: \"en\" });\nwordCounter(\"Hello world\", { latinTagHint: \"en\" });\nwordCounter(\"漢字測試\", { hanTagHint: \"zh-Hant\" });\nwordCounter(\"Hi 👋, world!\", { nonWords: true });\nwordCounter(\"Hi 👋, world!\", { mode: \"char\", nonWords: true });\nwordCounter(\"Hi\\tthere\\n\", { nonWords: true, includeWhitespace: true });\ncountCharsForLocale(\"👋\", \"en\");\n```\n\nNote: `includeWhitespace` only affects results when `nonWords: true` is enabled.\n\nSample output (with `nonWords: true` and `includeWhitespace: true`):\n\n```json\n{\n  \"total\": 4,\n  \"counts\": { \"words\": 2, \"nonWords\": 2, \"total\": 4 },\n  \"breakdown\": {\n    \"mode\": \"chunk\",\n    \"items\": [\n      {\n        // ...\n        \"words\": 2,\n        \"nonWords\": {\n          \"emoji\": [],\n          \"symbols\": [],\n          \"punctuation\": [],\n          \"counts\": { \"emoji\": 0, \"symbols\": 0, \"punctuation\": 0, \"whitespace\": 2 },\n          \"whitespace\": { \"spaces\": 0, \"tabs\": 1, \"newlines\": 1, \"other\": 0 }\n        }\n      }\n    ]\n  }\n}\n```\n\n### CJS\n\n```js\nconst wordCounter = require(\"@dev-pi2pie/word-counter\");\nconst {\n  countCharsForLocale,\n  countWordsForLocale,\n  countSections,\n  parseMarkdown,\n  segmentTextByLocale,\n  showSingularOrPluralWord,\n} = wordCounter;\n\nwordCounter(\"Hello world\", { latinLanguageHint: \"en\" });\nwordCounter(\"Hello world\", { latinTagHint: \"en\" });\nwordCounter(\"漢字測試\", { hanTagHint: \"zh-Hant\" });\nwordCounter(\"Hi 👋, world!\", { nonWords: true });\nwordCounter(\"Hi 👋, world!\", { mode: \"char\", nonWords: true });\nwordCounter(\"Hi\\tthere\\n\", { nonWords: true, includeWhitespace: true });\ncountCharsForLocale(\"👋\", \"en\");\n```\n\nNote: `includeWhitespace` only affects results when `nonWords: true` is enabled.\n\nSample output (with `nonWords: true` and `includeWhitespace: true`):\n\n```json\n{\n  \"total\": 4,\n  \"counts\": { \"words\": 2, \"nonWords\": 2, \"total\": 4 },\n  \"breakdown\": {\n    \"mode\": \"chunk\",\n    \"items\": [\n      {\n        // ...\n        \"words\": 2,\n        \"nonWords\": {\n          \"emoji\": [],\n          \"symbols\": [],\n          \"punctuation\": [],\n          \"counts\": { \"emoji\": 0, \"symbols\": 0, \"punctuation\": 0, \"whitespace\": 2 },\n          \"whitespace\": { \"spaces\": 0, \"tabs\": 1, \"newlines\": 1, \"other\": 0 }\n        }\n      }\n    ]\n  }\n}\n```\n\n### Export Summary\n\n#### Core API\n\n| Export                | Kind     | Notes                                              |\n| --------------------- | -------- | -------------------------------------------------- |\n| `default`             | function | `wordCounter(text, options?) -\u003e WordCounterResult` |\n| `wordCounter`         | function | Alias of the default export.                       |\n| `countCharsForLocale` | function | Low-level helper for per-locale char counts.       |\n| `countWordsForLocale` | function | Low-level helper for per-locale counts.            |\n| `segmentTextByLocale` | function | Low-level helper for locale-tag segmentation.      |\n\n#### Markdown Helpers\n\n| Export          | Kind     | Notes                                         |\n| --------------- | -------- | --------------------------------------------- |\n| `parseMarkdown` | function | Parses Markdown and detects frontmatter.      |\n| `countSections` | function | Counts words by frontmatter/content sections. |\n\n#### Utility Helpers\n\n| Export                     | Kind     | Notes                          |\n| -------------------------- | -------- | ------------------------------ |\n| `showSingularOrPluralWord` | function | Formats singular/plural words. |\n\n#### Types\n\n| Export                 | Kind | Notes                                             |\n| ---------------------- | ---- | ------------------------------------------------- |\n| `WordCounterOptions`   | type | Options for the `wordCounter` function.           |\n| `WordCounterResult`    | type | Returned by `wordCounter`.                        |\n| `WordCounterBreakdown` | type | Breakdown payload in `WordCounterResult`.         |\n| `WordCounterMode`      | type | `\"chunk\" \\| \"segments\" \\| \"collector\" \\| \"char\"`. |\n| `NonWordCollection`    | type | Non-word segments + counts payload.               |\n\n### Display Modes\n\nChoose a breakdown style with `--mode` (or `-m`):\n\n- `chunk` (default) – list each contiguous locale block in order of appearance.\n- `segments` – show the actual wordlike segments used for counting.\n- `collector` – aggregate counts per locale regardless of text position.\n- `char` – count grapheme clusters (user-perceived characters) per locale.\n\nAliases are normalized for CLI + API:\n\n- `chunk`, `chunks`\n- `segments`, `segment`, `seg`\n- `collector`, `collect`, `colle`\n- `char`, `chars`, `character`, `characters`\n\nExamples:\n\n```bash\n# chunk mode (default)\nword-counter \"飛鳥 bird 貓 cat; how do you do?\"\n\n# show captured segments\nword-counter --mode segments \"飛鳥 bird 貓 cat; how do you do?\"\n\n# aggregate per locale\nword-counter -m collector \"飛鳥 bird 貓 cat; how do you do?\"\n\n# grapheme-aware character count\nword-counter -m char \"Hi 👋, world!\"\n```\n\n### Section Modes (Frontmatter)\n\nUse `--section` to control which parts of a markdown document are counted:\n\n- `all` (default) – count the whole file (fast path, no section split).\n- `split` – count frontmatter and content separately.\n- `frontmatter` – count frontmatter only.\n- `content` – count content only.\n- `per-key` – count frontmatter per key (frontmatter only).\n- `split-per-key` – per-key frontmatter counts plus a content total.\n\nSupported frontmatter formats:\n\n- YAML fenced with `---`\n- TOML fenced with `+++`\n- JSON fenced with `;;;` or a top-of-file JSON object (`{ ... }`)\n\nExamples:\n\n```bash\nword-counter --section split -p examples/yaml-basic.md\nword-counter --section per-key -p examples/yaml-basic.md\nword-counter --section split-per-key -p examples/yaml-basic.md\n```\n\nJSON output includes a `source` field (`frontmatter` or `content`) to avoid key collisions:\n\n```bash\nword-counter --section split-per-key --format json -p examples/yaml-content-key.md\n```\n\nExample (trimmed):\n\n```json\n{\n  \"section\": \"split-per-key\",\n  \"frontmatterType\": \"yaml\",\n  \"total\": 7,\n  \"items\": [\n    { \"name\": \"content\", \"source\": \"frontmatter\", \"result\": { \"total\": 3 } },\n    { \"name\": \"content\", \"source\": \"content\", \"result\": { \"total\": 4 } }\n  ]\n}\n```\n\n### Output Formats\n\nSelect how results are printed with `--format`:\n\n- `standard` (default) – total plus per-locale breakdown.\n- `raw` – only the total count (single number).\n- `json` – machine-readable output; add `--pretty` for indentation.\n\nExamples:\n\n```bash\nword-counter --format raw \"Hello world\"\nword-counter --format json --pretty \"Hello world\"\n```\n\n### Non-Word Collection\n\nUse `--non-words` (or `nonWords: true` in the API) to collect emoji, symbols, and punctuation as separate categories. When enabled, the `total` includes both words and non-words.\n\n```bash\nword-counter --non-words \"Hi 👋, world!\"\n```\n\nExample: `total = words + emoji + symbols + punctuation` when enabled.\nStandard output labels this as `Total count` to reflect the combined total; `--format raw` still prints a single number.\n\nInclude whitespace-like characters in the non-words bucket (API: `includeWhitespace: true`):\n\n```bash\nword-counter --include-whitespace \"Hi\\tthere\\n\"\nword-counter --misc \"Hi\\tthere\\n\"\n```\n\nIn the CLI, `--include-whitespace` implies with `--non-words` (same behavior as `--misc`). `--non-words` alone does not include whitespace. When enabled, whitespace counts appear under `nonWords.whitespace`, and `total = words + nonWords` (emoji + symbols + punctuation + whitespace). JSON output also includes top-level `counts` when `nonWords` is enabled. See `docs/schemas/whitespace-categories.md` for how whitespace is categorized.\n\nExample JSON (trimmed):\n\n```json\n{\n  \"total\": 5,\n  \"counts\": { \"words\": 2, \"nonWords\": 3, \"total\": 5 },\n  \"breakdown\": {\n    \"mode\": \"chunk\",\n    \"items\": [\n      {\n        \"locale\": \"und-Latn\",\n        \"words\": 2,\n        \"nonWords\": {\n          \"counts\": { \"emoji\": 0, \"symbols\": 0, \"punctuation\": 0, \"whitespace\": 3 },\n          \"whitespace\": { \"spaces\": 1, \"tabs\": 1, \"newlines\": 1, \"other\": 0 }\n        }\n      }\n    ]\n  }\n}\n```\n\n\u003e [!Note]\n\u003e Text-default symbols (e.g. ©) count as `symbols` unless explicitly emoji-presented (e.g. ©️ with VS16).\n\n## Locale Tag Detection Notes (Migration)\n\n- Ambiguous Latin text uses `und-Latn` unless a Latin hint is provided.\n- Han-script fallback uses `zh-Hani` by default because regex script checks cannot natively distinguish `zh-Hans` vs `zh-Hant`.\n- Use `--mode chunk`/`--mode segments` or `--format json` to see the exact locale tag assigned to each chunk.\n- Regex/script-only detection cannot reliably identify English vs. other Latin-script languages; 100% certainty requires explicit metadata (document language tags, user-provided locale, headers) or a language-ID model.\n- Use `--latin-language \u003ctag\u003e` or `--latin-tag \u003ctag\u003e` for ambiguous Latin text.\n- Use `--han-language \u003ctag\u003e` or `--han-tag \u003ctag\u003e` for Han-script fallback.\n- `--latin-locale` remains supported as a legacy alias for now and is planned for future deprecation.\n\n## Breaking Changes Notes\n\n- Planned deprecations and migration notes are tracked in `docs/breaking-changes-notes.md`.\n\n## Testing\n\nRun the build before tests so the CJS interop test can load the emitted\n`dist/cjs/index.cjs` bundle:\n\n```bash\nbun run build\nbun test\n```\n\n## Sample Inputs\n\nTry the following mixed-language phrases to see how detection behaves:\n\n- `\"Hello world 你好世界\"`\n- `\"Bonjour le monde こんにちは 세계\"`\n- `\"¡Hola! مرحبا Hello\"`\n\nEach run prints the total word count plus a per-locale breakdown, helping you understand how multilingual text is segmented.\n\n## License\n\nThis project is licensed under the MIT License — see the [LICENSE](https://github.com/dev-pi2pie/word-counter/blob/main/LICENSE) file for details.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdev-pi2pie%2Fword-counter","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fdev-pi2pie%2Fword-counter","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdev-pi2pie%2Fword-counter/lists"}