{"id":19501089,"url":"https://github.com/gitbookio/tokenize-text","last_synced_at":"2025-04-25T23:30:52.231Z","repository":{"id":57377754,"uuid":"43807864","full_name":"GitbookIO/tokenize-text","owner":"GitbookIO","description":"Javascript text tokenizer that is easy to use and compose.","archived":false,"fork":false,"pushed_at":"2015-10-15T11:02:18.000Z","size":208,"stargazers_count":32,"open_issues_count":2,"forks_count":2,"subscribers_count":2,"default_branch":"master","last_synced_at":"2025-04-20T17:01:48.624Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":"JavaScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/GitbookIO.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2015-10-07T10:05:40.000Z","updated_at":"2024-12-01T15:37:18.000Z","dependencies_parsed_at":"2022-09-19T03:30:56.643Z","dependency_job_id":null,"html_url":"https://github.com/GitbookIO/tokenize-text","commit_stats":null,"previous_names":[],"tags_count":7,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/GitbookIO%2Ftokenize-text","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/GitbookIO%2Ftokenize-text/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/GitbookIO%2Ftokenize-text/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/GitbookIO%2Ftokenize-text/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/GitbookIO","download_url":"https://codeload.github.com/GitbookIO/tokenize-text/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":250912660,"owners_count":21506865,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-10T22:11:09.358Z","updated_at":"2025-04-25T23:30:51.942Z","avatar_url":"https://github.com/GitbookIO.png","language":"JavaScript","funding_links":[],"categories":[],"sub_categories":[],"readme":"# tokenize-text\n\n[![Build Status](https://travis-ci.org/GitbookIO/tokenize-text.png?branch=master)](https://travis-ci.org/GitbookIO/tokenize-text)\n[![NPM version](https://badge.fury.io/js/tokenize-text.svg)](http://badge.fury.io/js/tokenize-text)\n\nJavascript text tokenizer that is easy to use and compose.\n\n### Installation\n\n```\n$ npm install tokenize-text\n```\n\n### Usage\n\n```js\nvar Tokenizer = require('tokenize-text');\nvar tokenize = new Tokenizer();\n```\n\n#### tokenize.split(fn)\n\nThis is the main method of this module, all other methods are using it.\n\n`fn` will be called with 4 arguments:\n\n- `text`: text value of the token (`text == currentToken.value`)\n- `currentToken`: current token object\n- `prevToken`: precedent token (or null)\n- `nextToken`: next token (or null)\n\n`fn` should return a string, an array of string, a token or an array of tokens.\n\n`tokenize.split(fn)` returns a tokenizer function that accept a list of tokens or a string argument (it will be convert as one token).\n\nThe tokenizer function returns an array of tokens with the following properties:\n\n- `value`: text content of the token\n- `index`: absolute position in the original text\n- `offset`: length of the token (equivalent to `value.length`)\n\n```js\n// Simple tokenizer that split into 2 sections\nvar splitIn2 = tokenize.split(function(text, currentToken, prevToken, nextToken) {\n    return [\n        text.slice(0, text.length / 2),\n        text.slice(text.length / 2)\n    ]\n});\n\nvar tokens = splitIn2('hello');\n\n/*\n[\n    { value: 'he', index: 0, offset: 2 },\n    { value: 'llo', index: 2, offset: 3 }\n]\n*/\n```\n\n#### tokenize.re(re)\n\nTokenize using a regular expression:\n\n```js\nvar extractUppercase = tokenize.re(/[A-Z]/);\nvar tokens = extractUppercase('aBcD');\n\n/*\n[\n    { value: 'B', index: 1, offset: 1 },\n    { value: 'D', index: 3, offset: 1 }\n]\n*/\n```\n\n#### tokenize.characters()\n\nTokenize and split as characters, `tokenize.characters()` is equivalent to `tokenize.re(/[^\\s]/)`.\n\n```js\nvar tokens = tokenize.characters()('abc');\n\n/*\n[\n    { value: 'a', index: 0, offset: 1 },\n    { value: 'b', index: 1, offset: 1 },\n    { value: 'c', index: 2, offset: 1 }\n]\n*/\n```\n\n#### tokenize.sections()\n\nSplit in sections, sections are split by `\\n . , ; ! ?`.\n\n```js\nvar tokens = tokenize.sections()('this is sentence 1. this is sentence 2');\n\n/*\n[\n    {\n        value: 'this is sentence 1',\n        index: 0,\n        offset: 18\n    },\n    {\n        value: ' this is sentence 2',\n        index: 19,\n        offset: 19\n    }\n]\n*/\n```\n\n#### tokenize.words()\n\nSplit in words:\n\n```js\nvar tokens = tokenize.words()('hello, how are you?');\n\n/*\n[\n    { value: 'hello', index: 0, offset: 5 },\n    { value: 'how', index: 7, offset: 3 },\n    { value: 'are', index: 11, offset: 3 },\n    { value: 'you', index: 15, offset: 3 }\n]\n*/\n```\n\n#### tokenize.filter(fn)\n\nFilter the list of tokens by calling `fn(token)`:\n\n```js\n// Filter the words to extract the ones that start with an uppercase\nvar extractNames = tokenize.filter(function(word, current, prev) {\n    return (prev \u0026\u0026 /[A-Z]/.test(word[0]));\n});\n\n// Split texts in words\nvar words = tokenize.words()('My name is Samy.');\n\n// Apply the filter\nvar tokens = extractNames(words);\n\n/*\n[\n    { value: 'Samy', index: 11, offset: 4 }\n]\n*/\n```\n\n#### tokenize.flow(fn1, fn2, [...])\n\nCreates a tokenizer that returns the result of invoking the provided tokenizers for each input token.\n\n```js\nvar extractNames = tokenize.flow(\n    // Split text as words\n    tokenize.words(),\n\n    // Filter the words to extract the ones that start with an uppercase\n    tokenize.filter(function(word, current, prev) {\n        return (prev \u0026\u0026 /[A-Z]/.test(word[0]));\n    })\n);\n\nvar tokens = extractNames('My name is Samy.');\n```\n\nTo execute all tokenizer in series, you can use `tokenize.serie(fn1, fn2, [...])` instead.\n\n### Examples\n\n#### Extract repeated words in sentences\n\nExample to extract all repeated words in sentences:\n\n```js\nvar repeatedWords = tokenize.flow(\n    // Tokenize as sections\n    tokenize.sections(),\n\n    // For each sentence\n    tokenize.flow(\n        // Tokenize as words\n        tokenize.words(),\n\n        // Filter words to extract only repeated ones\n        tokenize.filter(function(word, token, prev) {\n            return (\n                prev \u0026\u0026\n                token.value.toLowerCase() === prev.value.toLowerCase()\n            );\n        })\n    )\n);\n\n\nvar tokens = repeatedWords('This is great great. Great is an an awesome words');\n\n/*\n[\n    { value: 'great', index: 14, offset: 5 },\n    { value: 'an', index: 33, offset: 2 }\n]\n*/\n```\n\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fgitbookio%2Ftokenize-text","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fgitbookio%2Ftokenize-text","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fgitbookio%2Ftokenize-text/lists"}