{"id":18062842,"url":"https://github.com/stagas/tokenizer-next","last_synced_at":"2025-04-05T13:13:45.904Z","repository":{"id":52890885,"uuid":"423585098","full_name":"stagas/tokenizer-next","owner":"stagas","description":"iterator based tokenizer for writing parsers","archived":false,"fork":false,"pushed_at":"2022-08-04T05:48:41.000Z","size":963,"stargazers_count":2,"open_issues_count":0,"forks_count":0,"subscribers_count":2,"default_branch":"main","last_synced_at":"2025-03-29T08:35:29.427Z","etag":null,"topics":["iterable","iterator","lexer","parser","parsing","regexp","regexp-match","tokenizer"],"latest_commit_sha":null,"homepage":"","language":"TypeScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/stagas.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2021-11-01T19:07:39.000Z","updated_at":"2023-02-17T19:48:37.000Z","dependencies_parsed_at":"2022-09-19T03:30:41.527Z","dependency_job_id":null,"html_url":"https://github.com/stagas/tokenizer-next","commit_stats":null,"previous_names":[],"tags_count":5,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/stagas%2Ftokenizer-next","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/stagas%2Ftokenizer-next/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/stagas%2Ftokenizer-next/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/stagas%2Ftokenizer-next/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/stagas","download_url":"https://codeload.github.com/stagas/tokenizer-next/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":247339158,"owners_count":20923014,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["iterable","iterator","lexer","parser","parsing","regexp","regexp-match","tokenizer"],"created_at":"2024-10-31T05:08:40.662Z","updated_at":"2025-04-05T13:13:45.882Z","avatar_url":"https://github.com/stagas.png","language":"TypeScript","funding_links":[],"categories":[],"sub_categories":[],"readme":"\u003ch1\u003e\ntokenizer-next \u003ca href=\"https://npmjs.org/package/tokenizer-next\"\u003e\u003cimg src=\"https://img.shields.io/badge/npm-v3.0.1-F00.svg?colorA=000\"/\u003e\u003c/a\u003e \u003ca href=\"src\"\u003e\u003cimg src=\"https://img.shields.io/badge/loc-27-FFF.svg?colorA=000\"/\u003e\u003c/a\u003e \u003ca href=\"https://cdn.jsdelivr.net/npm/tokenizer-next@3.0.1/dist/tokenizer-next.min.js\"\u003e\u003cimg src=\"https://img.shields.io/badge/brotli-478b-333.svg?colorA=000\"/\u003e\u003c/a\u003e \u003ca href=\"LICENSE\"\u003e\u003cimg src=\"https://img.shields.io/badge/license-MIT-F0B.svg?colorA=000\"/\u003e\u003c/a\u003e\n\u003c/h1\u003e\n\n\u003cp\u003e\u003c/p\u003e\n\niterator based tokenizer for writing parsers\n\n\u003ch4\u003e\n\u003ctable\u003e\u003ctr\u003e\u003ctd title=\"Triple click to select and copy paste\"\u003e\n\u003ccode\u003enpm i tokenizer-next \u003c/code\u003e\n\u003c/td\u003e\u003ctd title=\"Triple click to select and copy paste\"\u003e\n\u003ccode\u003epnpm add tokenizer-next \u003c/code\u003e\n\u003c/td\u003e\u003ctd title=\"Triple click to select and copy paste\"\u003e\n\u003ccode\u003eyarn add tokenizer-next\u003c/code\u003e\n\u003c/td\u003e\u003c/tr\u003e\u003c/table\u003e\n\u003c/h4\u003e\n\n## API\n\n\u003cp\u003e  \u003cdetails id=\"createTokenizer$1\" title=\"Function\" \u003e\u003csummary\u003e\u003cspan\u003e\u003ca href=\"#createTokenizer$1\"\u003e#\u003c/a\u003e\u003c/span\u003e  \u003ccode\u003e\u003cstrong\u003ecreateTokenizer\u003c/strong\u003e\u003c/code\u003e\u003cem\u003e(regexps)\u003c/em\u003e     \u0026ndash; Create a {@link TokenizerFactory} for the given RegExps.\u003c/summary\u003e  \u003ca href=\"src/index.ts#L19\"\u003esrc/index.ts#L19\u003c/a\u003e  \u003cul\u003e    \u003cp\u003e  \u003cp\u003e\n\nTo capture, RegExps must use a [named group](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Guide/Regular_Expressions/Groups_and_Ranges#using_named_groups).\n\n```ts\nconst tokenize = createTokenizer(\n  /(?\u003cident\u003e[a-z]+)/, // named groups determine token `group`\n  /(?\u003cnumber\u003e[0-9]+)/\n)\n```\n\n\u003c/p\u003e\n  \u003cdetails id=\"regexps$3\" title=\"Parameter\" \u003e\u003csummary\u003e\u003cspan\u003e\u003ca href=\"#regexps$3\"\u003e#\u003c/a\u003e\u003c/span\u003e  \u003ccode\u003e\u003cstrong\u003eregexps\u003c/strong\u003e\u003c/code\u003e     \u0026ndash; RegExps to match.\n\u003c/summary\u003e    \u003cul\u003e\u003cp\u003e\u003cspan\u003eRegExp\u003c/span\u003e  []\u003c/p\u003e        \u003c/ul\u003e\u003c/details\u003e  \u003cp\u003e\u003cstrong\u003ecreateTokenizer\u003c/strong\u003e\u003cem\u003e(regexps)\u003c/em\u003e  \u0026nbsp;=\u0026gt;  \u003cul\u003e\u003ca href=\"#TokenizerFactory$4\"\u003eTokenizerFactory\u003c/a\u003e\u003c/ul\u003e\u003c/p\u003e\u003c/p\u003e    \u003c/ul\u003e\u003c/details\u003e\u003cdetails id=\"Token$14\" title=\"Class\" \u003e\u003csummary\u003e\u003cspan\u003e\u003ca href=\"#Token$14\"\u003e#\u003c/a\u003e\u003c/span\u003e  \u003ccode\u003e\u003cstrong\u003eToken\u003c/strong\u003e\u003c/code\u003e     \u0026ndash; Token interface\u003c/summary\u003e  \u003ca href=\"src/match-to-token/dist/types/token.d.ts#L5\"\u003esrc/match-to-token/dist/types/token.d.ts#L5\u003c/a\u003e  \u003cul\u003e        \u003cp\u003e  \u003cdetails id=\"constructor$20\" title=\"Constructor\" \u003e\u003csummary\u003e\u003cspan\u003e\u003ca href=\"#constructor$20\"\u003e#\u003c/a\u003e\u003c/span\u003e  \u003ccode\u003e\u003cstrong\u003econstructor\u003c/strong\u003e\u003c/code\u003e\u003cem\u003e(value, group, source)\u003c/em\u003e    \u003c/summary\u003e  \u003ca href=\"src/match-to-token/dist/types/token.d.ts#L23\"\u003esrc/match-to-token/dist/types/token.d.ts#L23\u003c/a\u003e  \u003cul\u003e    \u003cp\u003e  \u003cdetails id=\"new Token$21\" title=\"ConstructorSignature\" \u003e\u003csummary\u003e\u003cspan\u003e\u003ca href=\"#new Token$21\"\u003e#\u003c/a\u003e\u003c/span\u003e  \u003ccode\u003e\u003cstrong\u003enew Token\u003c/strong\u003e\u003c/code\u003e\u003cem\u003e()\u003c/em\u003e    \u003c/summary\u003e    \u003cul\u003e\u003cp\u003e\u003ca href=\"#Token$14\"\u003eToken\u003c/a\u003e\u003c/p\u003e      \u003cp\u003e  \u003cdetails id=\"value$22\" title=\"Parameter\" \u003e\u003csummary\u003e\u003cspan\u003e\u003ca href=\"#value$22\"\u003e#\u003c/a\u003e\u003c/span\u003e  \u003ccode\u003e\u003cstrong\u003evalue\u003c/strong\u003e\u003c/code\u003e    \u003c/summary\u003e    \u003cul\u003e\u003cp\u003estring\u003c/p\u003e        \u003c/ul\u003e\u003c/details\u003e\u003cdetails id=\"group$23\" title=\"Parameter\" \u003e\u003csummary\u003e\u003cspan\u003e\u003ca href=\"#group$23\"\u003e#\u003c/a\u003e\u003c/span\u003e  \u003ccode\u003e\u003cstrong\u003egroup\u003c/strong\u003e\u003c/code\u003e    \u003c/summary\u003e    \u003cul\u003e\u003cp\u003estring\u003c/p\u003e        \u003c/ul\u003e\u003c/details\u003e\u003cdetails id=\"source$24\" title=\"Parameter\" \u003e\u003csummary\u003e\u003cspan\u003e\u003ca href=\"#source$24\"\u003e#\u003c/a\u003e\u003c/span\u003e  \u003ccode\u003e\u003cstrong\u003esource\u003c/strong\u003e\u003c/code\u003e    \u003c/summary\u003e    \u003cul\u003e\u003cp\u003e\u003ca href=\"#RegExpMatchArrayLike$39\"\u003eRegExpMatchArrayLike\u003c/a\u003e\u003c/p\u003e        \u003c/ul\u003e\u003c/details\u003e\u003c/p\u003e  \u003c/ul\u003e\u003c/details\u003e\u003c/p\u003e    \u003c/ul\u003e\u003c/details\u003e\u003cdetails id=\"group$25\" title=\"Property\" \u003e\u003csummary\u003e\u003cspan\u003e\u003ca href=\"#group$25\"\u003e#\u003c/a\u003e\u003c/span\u003e  \u003ccode\u003e\u003cstrong\u003egroup\u003c/strong\u003e\u003c/code\u003e     \u0026ndash; The group it matched.\u003c/summary\u003e  \u003ca href=\"src/match-to-token/dist/types/token.d.ts#L10\"\u003esrc/match-to-token/dist/types/token.d.ts#L10\u003c/a\u003e  \u003cul\u003e\u003cp\u003estring\u003c/p\u003e        \u003c/ul\u003e\u003c/details\u003e\u003cdetails id=\"source$30\" title=\"Property\" \u003e\u003csummary\u003e\u003cspan\u003e\u003ca href=\"#source$30\"\u003e#\u003c/a\u003e\u003c/span\u003e  \u003ccode\u003e\u003cstrong\u003esource\u003c/strong\u003e\u003c/code\u003e     \u0026ndash; The input string.\u003c/summary\u003e  \u003ca href=\"src/match-to-token/dist/types/token.d.ts#L22\"\u003esrc/match-to-token/dist/types/token.d.ts#L22\u003c/a\u003e  \u003cul\u003e\u003cp\u003e\u003ca href=\"#RegExpMatchArrayLike$39\"\u003eRegExpMatchArrayLike\u003c/a\u003e\u003c/p\u003e        \u003c/ul\u003e\u003c/details\u003e\u003cdetails id=\"index$28\" title=\"Accessor\" \u003e\u003csummary\u003e\u003cspan\u003e\u003ca href=\"#index$28\"\u003e#\u003c/a\u003e\u003c/span\u003e  \u003ccode\u003e\u003cstrong\u003eindex\u003c/strong\u003e\u003c/code\u003e    \u003c/summary\u003e  \u003ca href=\"src/match-to-token/dist/types/token.d.ts#L18\"\u003esrc/match-to-token/dist/types/token.d.ts#L18\u003c/a\u003e  \u003cul\u003e        \u003c/ul\u003e\u003c/details\u003e\u003cdetails id=\"value$26\" title=\"Accessor\" \u003e\u003csummary\u003e\u003cspan\u003e\u003ca href=\"#value$26\"\u003e#\u003c/a\u003e\u003c/span\u003e  \u003ccode\u003e\u003cstrong\u003evalue\u003c/strong\u003e\u003c/code\u003e    \u003c/summary\u003e  \u003ca href=\"src/match-to-token/dist/types/token.d.ts#L14\"\u003esrc/match-to-token/dist/types/token.d.ts#L14\u003c/a\u003e  \u003cul\u003e        \u003c/ul\u003e\u003c/details\u003e\u003cdetails id=\"as$35\" title=\"Method\" \u003e\u003csummary\u003e\u003cspan\u003e\u003ca href=\"#as$35\"\u003e#\u003c/a\u003e\u003c/span\u003e  \u003ccode\u003e\u003cstrong\u003eas\u003c/strong\u003e\u003c/code\u003e\u003cem\u003e(value, group)\u003c/em\u003e    \u003c/summary\u003e  \u003ca href=\"src/match-to-token/dist/types/token.d.ts#L25\"\u003esrc/match-to-token/dist/types/token.d.ts#L25\u003c/a\u003e  \u003cul\u003e    \u003cp\u003e    \u003cdetails id=\"value$37\" title=\"Parameter\" \u003e\u003csummary\u003e\u003cspan\u003e\u003ca href=\"#value$37\"\u003e#\u003c/a\u003e\u003c/span\u003e  \u003ccode\u003e\u003cstrong\u003evalue\u003c/strong\u003e\u003c/code\u003e    \u003c/summary\u003e    \u003cul\u003e\u003cp\u003estring\u003c/p\u003e        \u003c/ul\u003e\u003c/details\u003e\u003cdetails id=\"group$38\" title=\"Parameter\" \u003e\u003csummary\u003e\u003cspan\u003e\u003ca href=\"#group$38\"\u003e#\u003c/a\u003e\u003c/span\u003e  \u003ccode\u003e\u003cstrong\u003egroup\u003c/strong\u003e\u003c/code\u003e    \u003c/summary\u003e    \u003cul\u003e\u003cp\u003estring\u003c/p\u003e        \u003c/ul\u003e\u003c/details\u003e  \u003cp\u003e\u003cstrong\u003eas\u003c/strong\u003e\u003cem\u003e(value, group)\u003c/em\u003e  \u0026nbsp;=\u0026gt;  \u003cul\u003e\u003ca href=\"#Token$14\"\u003eToken\u003c/a\u003e\u003c/ul\u003e\u003c/p\u003e\u003c/p\u003e    \u003c/ul\u003e\u003c/details\u003e\u003cdetails id=\"is$31\" title=\"Method\" \u003e\u003csummary\u003e\u003cspan\u003e\u003ca href=\"#is$31\"\u003e#\u003c/a\u003e\u003c/span\u003e  \u003ccode\u003e\u003cstrong\u003eis\u003c/strong\u003e\u003c/code\u003e\u003cem\u003e(group, value)\u003c/em\u003e    \u003c/summary\u003e  \u003ca href=\"src/match-to-token/dist/types/token.d.ts#L24\"\u003esrc/match-to-token/dist/types/token.d.ts#L24\u003c/a\u003e  \u003cul\u003e    \u003cp\u003e    \u003cdetails id=\"group$33\" title=\"Parameter\" \u003e\u003csummary\u003e\u003cspan\u003e\u003ca href=\"#group$33\"\u003e#\u003c/a\u003e\u003c/span\u003e  \u003ccode\u003e\u003cstrong\u003egroup\u003c/strong\u003e\u003c/code\u003e    \u003c/summary\u003e    \u003cul\u003e\u003cp\u003estring\u003c/p\u003e        \u003c/ul\u003e\u003c/details\u003e\u003cdetails id=\"value$34\" title=\"Parameter\" \u003e\u003csummary\u003e\u003cspan\u003e\u003ca href=\"#value$34\"\u003e#\u003c/a\u003e\u003c/span\u003e  \u003ccode\u003e\u003cstrong\u003evalue\u003c/strong\u003e\u003c/code\u003e    \u003c/summary\u003e    \u003cul\u003e\u003cp\u003estring\u003c/p\u003e        \u003c/ul\u003e\u003c/details\u003e  \u003cp\u003e\u003cstrong\u003eis\u003c/strong\u003e\u003cem\u003e(group, value)\u003c/em\u003e  \u0026nbsp;=\u0026gt;  \u003cul\u003eboolean\u003c/ul\u003e\u003c/p\u003e\u003c/p\u003e    \u003c/ul\u003e\u003c/details\u003e\u003cdetails id=\"create$15\" title=\"Method\" \u003e\u003csummary\u003e\u003cspan\u003e\u003ca href=\"#create$15\"\u003e#\u003c/a\u003e\u003c/span\u003e  \u003ccode\u003e\u003cstrong\u003ecreate\u003c/strong\u003e\u003c/code\u003e\u003cem\u003e(value, group, source)\u003c/em\u003e    \u003c/summary\u003e  \u003ca href=\"src/match-to-token/dist/types/token.d.ts#L6\"\u003esrc/match-to-token/dist/types/token.d.ts#L6\u003c/a\u003e  \u003cul\u003e    \u003cp\u003e    \u003cdetails id=\"value$17\" title=\"Parameter\" \u003e\u003csummary\u003e\u003cspan\u003e\u003ca href=\"#value$17\"\u003e#\u003c/a\u003e\u003c/span\u003e  \u003ccode\u003e\u003cstrong\u003evalue\u003c/strong\u003e\u003c/code\u003e    \u003c/summary\u003e    \u003cul\u003e\u003cp\u003estring\u003c/p\u003e        \u003c/ul\u003e\u003c/details\u003e\u003cdetails id=\"group$18\" title=\"Parameter\" \u003e\u003csummary\u003e\u003cspan\u003e\u003ca href=\"#group$18\"\u003e#\u003c/a\u003e\u003c/span\u003e  \u003ccode\u003e\u003cstrong\u003egroup\u003c/strong\u003e\u003c/code\u003e    \u003c/summary\u003e    \u003cul\u003e\u003cp\u003estring\u003c/p\u003e        \u003c/ul\u003e\u003c/details\u003e\u003cdetails id=\"source$19\" title=\"Parameter\" \u003e\u003csummary\u003e\u003cspan\u003e\u003ca href=\"#source$19\"\u003e#\u003c/a\u003e\u003c/span\u003e  \u003ccode\u003e\u003cstrong\u003esource\u003c/strong\u003e\u003c/code\u003e    \u003c/summary\u003e    \u003cul\u003e\u003cp\u003e\u003ca href=\"#RegExpMatchArrayLike$39\"\u003eRegExpMatchArrayLike\u003c/a\u003e\u003c/p\u003e        \u003c/ul\u003e\u003c/details\u003e  \u003cp\u003e\u003cstrong\u003ecreate\u003c/strong\u003e\u003cem\u003e(value, group, source)\u003c/em\u003e  \u0026nbsp;=\u0026gt;  \u003cul\u003e\u003ca href=\"#Token$14\"\u003eToken\u003c/a\u003e\u003c/ul\u003e\u003c/p\u003e\u003c/p\u003e    \u003c/ul\u003e\u003c/details\u003e\u003c/p\u003e\u003c/ul\u003e\u003c/details\u003e  \u003cdetails id=\"RegExpMatchArrayLike$39\" title=\"Interface\" \u003e\u003csummary\u003e\u003cspan\u003e\u003ca href=\"#RegExpMatchArrayLike$39\"\u003e#\u003c/a\u003e\u003c/span\u003e  \u003ccode\u003e\u003cstrong\u003eRegExpMatchArrayLike\u003c/strong\u003e\u003c/code\u003e    \u003c/summary\u003e  \u003ca href=\"src/match-to-token/dist/types/index.d.ts#L2\"\u003esrc/match-to-token/dist/types/index.d.ts#L2\u003c/a\u003e  \u003cul\u003e        \u003cp\u003e  \u003cdetails id=\"index$40\" title=\"Property\" \u003e\u003csummary\u003e\u003cspan\u003e\u003ca href=\"#index$40\"\u003e#\u003c/a\u003e\u003c/span\u003e  \u003ccode\u003e\u003cstrong\u003eindex\u003c/strong\u003e\u003c/code\u003e    \u003c/summary\u003e  \u003ca href=\"src/match-to-token/dist/types/index.d.ts#L3\"\u003esrc/match-to-token/dist/types/index.d.ts#L3\u003c/a\u003e  \u003cul\u003e\u003cp\u003enumber\u003c/p\u003e        \u003c/ul\u003e\u003c/details\u003e\u003cdetails id=\"input$41\" title=\"Property\" \u003e\u003csummary\u003e\u003cspan\u003e\u003ca href=\"#input$41\"\u003e#\u003c/a\u003e\u003c/span\u003e  \u003ccode\u003e\u003cstrong\u003einput\u003c/strong\u003e\u003c/code\u003e    \u003c/summary\u003e  \u003ca href=\"src/match-to-token/dist/types/index.d.ts#L4\"\u003esrc/match-to-token/dist/types/index.d.ts#L4\u003c/a\u003e  \u003cul\u003e\u003cp\u003estring\u003c/p\u003e        \u003c/ul\u003e\u003c/details\u003e\u003c/p\u003e\u003c/ul\u003e\u003c/details\u003e  \u003cdetails id=\"Token$13\" title=\"TypeAlias\" \u003e\u003csummary\u003e\u003cspan\u003e\u003ca href=\"#Token$13\"\u003e#\u003c/a\u003e\u003c/span\u003e  \u003ccode\u003e\u003cstrong\u003eToken\u003c/strong\u003e\u003c/code\u003e    \u003c/summary\u003e  \u003ca href=\"src/match-to-token/dist/types/index.d.ts#L6\"\u003esrc/match-to-token/dist/types/index.d.ts#L6\u003c/a\u003e  \u003cul\u003e\u003cp\u003e\u003ca href=\"#MatchToken$14\"\u003eMatchToken\u003c/a\u003e \u0026amp; string\u003c/p\u003e        \u003c/ul\u003e\u003c/details\u003e\u003cdetails id=\"TokenizerCallableIterable$8\" title=\"TypeAlias\" \u003e\u003csummary\u003e\u003cspan\u003e\u003ca href=\"#TokenizerCallableIterable$8\"\u003e#\u003c/a\u003e\u003c/span\u003e  \u003ccode\u003e\u003cstrong\u003eTokenizerCallableIterable\u003c/strong\u003e\u003c/code\u003e     \u0026ndash; Can be called to return next \u0026lt;a href=\u0026quot;https://github.com/stagas/match-to-token#token\u0026quot;\u0026gt;Token\u0026lt;/a\u0026gt; or can be used as an\n\u003ca href=\"https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Iteration_protocols#the_iterable_protocol\"\u003eIterable\u003c/a\u003e\non \u003cstrong\u003efor-of\u003c/strong\u003e and \u003cstrong\u003espread\u003c/strong\u003e operations.\u003c/summary\u003e  \u003ca href=\"src/index.ts#L74\"\u003esrc/index.ts#L74\u003c/a\u003e  \u003cul\u003e\u003cp\u003e\u003cdetails id=\"__type$9\" title=\"Function\" \u003e\u003csummary\u003e\u003cspan\u003e\u003ca href=\"#__type$9\"\u003e#\u003c/a\u003e\u003c/span\u003e  \u003cem\u003e()\u003c/em\u003e    \u003c/summary\u003e    \u003cul\u003e    \u003cp\u003e      \u003cp\u003e\u003cstrong\u003e\u003c/strong\u003e\u003cem\u003e()\u003c/em\u003e  \u0026nbsp;=\u0026gt;  \u003cul\u003e\u003ca href=\"#Token$13\"\u003eToken\u003c/a\u003e\u003c/ul\u003e\u003c/p\u003e\u003c/p\u003e    \u003c/ul\u003e\u003c/details\u003e \u0026amp; \u003cspan\u003eIterable\u003c/span\u003e\u0026lt;\u003ca href=\"#Token$13\"\u003eToken\u003c/a\u003e\u0026gt;\u003c/p\u003e        \u003c/ul\u003e\u003c/details\u003e\u003cdetails id=\"TokenizerFactory$4\" title=\"TypeAlias\" \u003e\u003csummary\u003e\u003cspan\u003e\u003ca href=\"#TokenizerFactory$4\"\u003e#\u003c/a\u003e\u003c/span\u003e  \u003ccode\u003e\u003cstrong\u003eTokenizerFactory\u003c/strong\u003e\u003c/code\u003e    \u003c/summary\u003e  \u003ca href=\"src/index.ts#L67\"\u003esrc/index.ts#L67\u003c/a\u003e  \u003cul\u003e\u003cp\u003e\u003cdetails id=\"__type$5\" title=\"Function\" \u003e\u003csummary\u003e\u003cspan\u003e\u003ca href=\"#__type$5\"\u003e#\u003c/a\u003e\u003c/span\u003e  \u003cem\u003e(input)\u003c/em\u003e     \u0026ndash; Create a {@link TokenizerCallableIterable} for given input string.\u003c/summary\u003e    \u003cul\u003e    \u003cp\u003e  \u003cp\u003e\n\n```ts\n// using next()\nconst next = tokenize('hello 123')\nconsole.log(next()) // =\u003e {group: 'ident', value: 'hello', index: 0}\nconsole.log(next()) // =\u003e {group: 'number', value: '123', index: 6}\nconsole.log(next()) // =\u003e undefined\n\n// using for of\nfor (const token of tokenize('hello 123')) {\n  console.log(token)\n  // =\u003e {group: 'ident', value: 'hello', index: 0}\n  // =\u003e {group: 'number', value: '123', index: 6}\n}\n\n// using spread\nconst tokens = [...tokenize('hello 123')]\nconsole.log(tokens)\n// =\u003e [\n//   {group: 'ident', value: 'hello', index: 0},\n//   {group: 'number', value: '123', index: 6}\n// ]\n```\n\n\u003c/p\u003e\n  \u003cdetails id=\"input$7\" title=\"Parameter\" \u003e\u003csummary\u003e\u003cspan\u003e\u003ca href=\"#input$7\"\u003e#\u003c/a\u003e\u003c/span\u003e  \u003ccode\u003e\u003cstrong\u003einput\u003c/strong\u003e\u003c/code\u003e     \u0026ndash; The string to tokenize.\n\u003c/summary\u003e    \u003cul\u003e\u003cp\u003estring\u003c/p\u003e        \u003c/ul\u003e\u003c/details\u003e  \u003cp\u003e\u003cstrong\u003e\u003c/strong\u003e\u003cem\u003e(input)\u003c/em\u003e  \u0026nbsp;=\u0026gt;  \u003cul\u003e\u003ca href=\"#TokenizerCallableIterable$8\"\u003eTokenizerCallableIterable\u003c/a\u003e\u003c/ul\u003e\u003c/p\u003e\u003c/p\u003e    \u003c/ul\u003e\u003c/details\u003e\u003c/p\u003e        \u003c/ul\u003e\u003c/details\u003e\u003cdetails id=\"createTokenizer$1\" title=\"Function\" \u003e\u003csummary\u003e\u003cspan\u003e\u003ca href=\"#createTokenizer$1\"\u003e#\u003c/a\u003e\u003c/span\u003e  \u003ccode\u003e\u003cstrong\u003ecreateTokenizer\u003c/strong\u003e\u003c/code\u003e\u003cem\u003e(regexps)\u003c/em\u003e     \u0026ndash; Create a {@link TokenizerFactory} for the given RegExps.\u003c/summary\u003e  \u003ca href=\"src/index.ts#L19\"\u003esrc/index.ts#L19\u003c/a\u003e  \u003cul\u003e    \u003cp\u003e  \u003cp\u003e\n\nTo capture, RegExps must use a [named group](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Guide/Regular_Expressions/Groups_and_Ranges#using_named_groups).\n\n```ts\nconst tokenize = createTokenizer(\n  /(?\u003cident\u003e[a-z]+)/, // named groups determine token `group`\n  /(?\u003cnumber\u003e[0-9]+)/\n)\n```\n\n\u003c/p\u003e\n  \u003cdetails id=\"regexps$3\" title=\"Parameter\" \u003e\u003csummary\u003e\u003cspan\u003e\u003ca href=\"#regexps$3\"\u003e#\u003c/a\u003e\u003c/span\u003e  \u003ccode\u003e\u003cstrong\u003eregexps\u003c/strong\u003e\u003c/code\u003e     \u0026ndash; RegExps to match.\n\u003c/summary\u003e    \u003cul\u003e\u003cp\u003e\u003cspan\u003eRegExp\u003c/span\u003e  []\u003c/p\u003e        \u003c/ul\u003e\u003c/details\u003e  \u003cp\u003e\u003cstrong\u003ecreateTokenizer\u003c/strong\u003e\u003cem\u003e(regexps)\u003c/em\u003e  \u0026nbsp;=\u0026gt;  \u003cul\u003e\u003ca href=\"#TokenizerFactory$4\"\u003eTokenizerFactory\u003c/a\u003e\u003c/ul\u003e\u003c/p\u003e\u003c/p\u003e    \u003c/ul\u003e\u003c/details\u003e\u003c/p\u003e\n\n## Credits\n\n- [match-to-token](https://npmjs.org/package/match-to-token) by [stagas](https://github.com/stagas) \u0026ndash; transform a RegExp named group match to a more useful object\n\n## Contributing\n\n[Fork](https://github.com/stagas/tokenizer-next/fork) or [edit](https://github.dev/stagas/tokenizer-next) and submit a PR.\n\nAll contributions are welcome!\n\n## License\n\n\u003ca href=\"LICENSE\"\u003eMIT\u003c/a\u003e \u0026copy; 2022 [stagas](https://github.com/stagas)\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fstagas%2Ftokenizer-next","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fstagas%2Ftokenizer-next","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fstagas%2Ftokenizer-next/lists"}