{"id":16698074,"url":"https://github.com/arlac77/chunk-tokenizer","last_synced_at":"2025-08-21T10:15:33.093Z","repository":{"id":23631723,"uuid":"99480842","full_name":"arlac77/chunk-tokenizer","owner":"arlac77","description":"Transform stream that emits tokens","archived":false,"fork":false,"pushed_at":"2025-04-24T12:35:33.000Z","size":1802,"stargazers_count":0,"open_issues_count":2,"forks_count":0,"subscribers_count":2,"default_branch":"master","last_synced_at":"2025-04-24T13:36:22.587Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":"JavaScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"0bsd","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/arlac77.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2017-08-06T11:01:24.000Z","updated_at":"2025-04-23T18:26:03.000Z","dependencies_parsed_at":"2023-09-23T04:47:06.709Z","dependency_job_id":"4f785a03-7fcd-44c6-8afc-7eb09ce36c96","html_url":"https://github.com/arlac77/chunk-tokenizer","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/arlac77%2Fchunk-tokenizer","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/arlac77%2Fchunk-tokenizer/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/arlac77%2Fchunk-tokenizer/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/arlac77%2Fchunk-tokenizer/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/arlac77","download_url":"https://codeload.github.com/arlac77/chunk-tokenizer/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":251755890,"owners_count":21638786,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-10-12T17:50:39.480Z","updated_at":"2025-04-30T17:49:33.025Z","avatar_url":"https://github.com/arlac77.png","language":"JavaScript","funding_links":[],"categories":[],"sub_categories":[],"readme":"[![npm](https://img.shields.io/npm/v/chunk-tokenizer.svg)](https://www.npmjs.com/package/chunk-tokenizer)\n[![License](https://img.shields.io/badge/License-0BSD-blue.svg)](https://spdx.org/licenses/0BSD.html)\n[![bundlejs](https://deno.bundlejs.com/?q=chunk-tokenizer\\\u0026badge=detailed)](https://bundlejs.com/?q=chunk-tokenizer)\n[![downloads](http://img.shields.io/npm/dm/chunk-tokenizer.svg?style=flat-square)](https://npmjs.org/package/chunk-tokenizer)\n[![GitHub Issues](https://img.shields.io/github/issues/arlac77/chunk-tokenizer.svg?style=flat-square)](https://github.com/arlac77/chunk-tokenizer/issues)\n[![Build Status](https://img.shields.io/endpoint.svg?url=https%3A%2F%2Factions-badge.atrox.dev%2Farlac77%2Fchunk-tokenizer%2Fbadge\\\u0026style=flat)](https://actions-badge.atrox.dev/arlac77/chunk-tokenizer/goto)\n[![Styled with prettier](https://img.shields.io/badge/styled_with-prettier-ff69b4.svg)](https://github.com/prettier/prettier)\n[![Commitizen friendly](https://img.shields.io/badge/commitizen-friendly-brightgreen.svg)](http://commitizen.github.io/cz-cli/)\n[![Known Vulnerabilities](https://snyk.io/test/github/arlac77/chunk-tokenizer/badge.svg)](https://snyk.io/test/github/arlac77/chunk-tokenizer)\n[![Coverage Status](https://coveralls.io/repos/arlac77/chunk-tokenizer/badge.svg)](https://coveralls.io/github/arlac77/chunk-tokenizer)\n\n# chunk-tokenizer\n\nTransform stream that emits tokens\n\n# API\n\n\u003c!-- Generated by documentation.js. Update this documentation by updating the source code. --\u003e\n\n### Table of Contents\n\n*   [EOFToken](#eoftoken)\n*   [KeywordToken](#keywordtoken)\n*   [makeKeywordTokens](#makekeywordtokens)\n    *   [Parameters](#parameters)\n*   [LineCommentIgnoreToken](#linecommentignoretoken)\n*   [makeLineCommentToken](#makelinecommenttoken)\n    *   [Parameters](#parameters-1)\n*   [makeOperatorTokens](#makeoperatortokens)\n    *   [Parameters](#parameters-2)\n*   [StringChunk](#stringchunk)\n    *   [Parameters](#parameters-3)\n    *   [Properties](#properties)\n    *   [append](#append)\n        *   [Parameters](#parameters-4)\n    *   [appendLast](#appendlast)\n        *   [Parameters](#parameters-5)\n    *   [extractFromMarkedPosition](#extractfrommarkedposition)\n    *   [markPosition](#markposition)\n        *   [Parameters](#parameters-6)\n    *   [peek](#peek)\n    *   [advance](#advance)\n    *   [advanceBy](#advanceby)\n        *   [Parameters](#parameters-7)\n*   [parse](#parse)\n    *   [Parameters](#parameters-8)\n*   [TokenMatcher](#tokenmatcher)\n    *   [Parameters](#parameters-9)\n    *   [Properties](#properties-1)\n*   [Token](#token)\n    *   [possibleFirstChars](#possiblefirstchars)\n    *   [register](#register)\n        *   [Parameters](#parameters-10)\n*   [characterSetFromString](#charactersetfromstring)\n    *   [Parameters](#parameters-11)\n*   [WhitespaceIgnoreToken](#whitespaceignoretoken)\n\n## EOFToken\n\n**Extends Token**\n\nToken representing EOF\n\n## KeywordToken\n\n**Extends Token**\n\n## makeKeywordTokens\n\nCreates a new token class for each token definition.\n\n### Parameters\n\n*   `tokenDefinitions` **[Object](https://developer.mozilla.org/docs/Web/JavaScript/Reference/Global_Objects/Object)** keys are the operator names\n*   `baseToken` **[KeywordToken](#keywordtoken)**  (optional, default `KeywordToken`)\n\nReturns **[Array](https://developer.mozilla.org/docs/Web/JavaScript/Reference/Global_Objects/Array)\u003c[KeywordToken](#keywordtoken)\u003e** newly created KeywordToken classes\n\n## LineCommentIgnoreToken\n\n**Extends Token**\n\nToken to skip until end of line\n\n## makeLineCommentToken\n\n### Parameters\n\n*   `prefix` **[string](https://developer.mozilla.org/docs/Web/JavaScript/Reference/Global_Objects/String)**\u0026#x20;\n*   `baseToken` **Class**  (optional, default `LineCommentIgnoreToken`)\n\n## makeOperatorTokens\n\nCreates a new token class for each token definition.\n\n### Parameters\n\n*   `tokenDefinitions`  {Object} keys are the operator names\n*   `baseToken`  {OperatorToken} (optional, default `OperatorToken`)\n\nReturns **[Array](https://developer.mozilla.org/docs/Web/JavaScript/Reference/Global_Objects/Array)\\\u003cOpearatorToken\u003e** newly created OperatorToken classes\n\n## StringChunk\n\n### Parameters\n\n*   `buffer`   (optional, default `\"\"`)\n*   `isLast`   (optional, default `false`)\n\n### Properties\n\n*   `position` **[Number](https://developer.mozilla.org/docs/Web/JavaScript/Reference/Global_Objects/Number)** current peek position in the buffer\n*   `buffer` **[string](https://developer.mozilla.org/docs/Web/JavaScript/Reference/Global_Objects/String)**\u0026#x20;\n*   `currentLine` **[Number](https://developer.mozilla.org/docs/Web/JavaScript/Reference/Global_Objects/Number)**\u0026#x20;\n\n### append\n\nappend content of buffer\nand reset the position(s)\n\n#### Parameters\n\n*   `buffer` **[string](https://developer.mozilla.org/docs/Web/JavaScript/Reference/Global_Objects/String)**\u0026#x20;\n\n### appendLast\n\nIndicate that this will be the last chunk\nappend content of buffer\nand reset the position(s)\n\n#### Parameters\n\n*   `buffer` **[string](https://developer.mozilla.org/docs/Web/JavaScript/Reference/Global_Objects/String)**\u0026#x20;\n\n### extractFromMarkedPosition\n\n### markPosition\n\nmark position and prepserve state\n\n#### Parameters\n\n*   `state` **[object](https://developer.mozilla.org/docs/Web/JavaScript/Reference/Global_Objects/Object)**\u0026#x20;\n\nReturns **any** former preseved or newly set state\n\n### peek\n\nReturns **[Number](https://developer.mozilla.org/docs/Web/JavaScript/Reference/Global_Objects/Number)** char at the current position\n\n### advance\n\nAdvance current position by one (after delivring the current char)\n\nReturns **[Number](https://developer.mozilla.org/docs/Web/JavaScript/Reference/Global_Objects/Number)** char at the current position\n\n### advanceBy\n\nAdvance current position by numberOfChars\n\n#### Parameters\n\n*   `numberOfChars` **[Number](https://developer.mozilla.org/docs/Web/JavaScript/Reference/Global_Objects/Number)**\u0026#x20;\n\n## parse\n\n0 -\u003e skip leading \"\n1 -\u003e copy chars\n2 -\u003e escape\n\n### Parameters\n\n*   `chunk` \u0026#x20;\n\n## TokenMatcher\n\nHolds a Set of tokens and identifies them based on the longest matching character string\n\n### Parameters\n\n*   `tokens` **[Array](https://developer.mozilla.org/docs/Web/JavaScript/Reference/Global_Objects/Array)\u003c[Token](#token)\u003e**\u0026#x20;\n\n### Properties\n\n*   `tokens` **[Array](https://developer.mozilla.org/docs/Web/JavaScript/Reference/Global_Objects/Array)\u003c[Token](#token)\u003e**\u0026#x20;\n*   `registeredTokens` **[Map](https://developer.mozilla.org/docs/Web/JavaScript/Reference/Global_Objects/Map)\u003c[string](https://developer.mozilla.org/docs/Web/JavaScript/Reference/Global_Objects/String), [Token](#token)\u003e**\u0026#x20;\n*   `maxTokenLengthForFirstChar` **[Map](https://developer.mozilla.org/docs/Web/JavaScript/Reference/Global_Objects/Map)\\\u003cChar, [number](https://developer.mozilla.org/docs/Web/JavaScript/Reference/Global_Objects/Number)\u003e**\u0026#x20;\n\n## Token\n\nAbstract base token\n\n### possibleFirstChars\n\nPossible first chars\n\nReturns **any** Set(\u003cNumber\u003e) all possible first chars for the token\n\n### register\n\nregister the token in the TokenMatcher\n\n#### Parameters\n\n*   `tokenMatcher` **[TokenMatcher](#tokenmatcher)**\u0026#x20;\n\n## characterSetFromString\n\n### Parameters\n\n*   `str` **[string](https://developer.mozilla.org/docs/Web/JavaScript/Reference/Global_Objects/String)**\u0026#x20;\n\nReturns **[Set](https://developer.mozilla.org/docs/Web/JavaScript/Reference/Global_Objects/Set)\u003c[string](https://developer.mozilla.org/docs/Web/JavaScript/Reference/Global_Objects/String)\u003e** all characters from the string\n\n## WhitespaceIgnoreToken\n\n**Extends Token**\n\nToken to consume all consecutive whitespace\n\n# install\n\nWith [npm](http://npmjs.org) do:\n\n```shell\nnpm install chunk-tokenizer\n```\n\n# license\n\nBSD-2-Clause\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Farlac77%2Fchunk-tokenizer","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Farlac77%2Fchunk-tokenizer","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Farlac77%2Fchunk-tokenizer/lists"}