{"id":13492749,"url":"https://github.com/lydell/js-tokens","last_synced_at":"2025-05-13T20:12:35.283Z","repository":{"id":14829707,"uuid":"17552490","full_name":"lydell/js-tokens","owner":"lydell","description":"Tiny JavaScript tokenizer.","archived":false,"fork":false,"pushed_at":"2025-04-04T17:26:19.000Z","size":917,"stargazers_count":518,"open_issues_count":0,"forks_count":33,"subscribers_count":6,"default_branch":"main","last_synced_at":"2025-05-07T07:40:08.950Z","etag":null,"topics":["ecmascript","javascript","regex","tokenizer"],"latest_commit_sha":null,"homepage":"","language":"JavaScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/lydell.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2014-03-08T22:28:09.000Z","updated_at":"2025-04-30T13:31:56.000Z","dependencies_parsed_at":"2024-05-13T08:26:19.649Z","dependency_job_id":"f452a4a0-e376-459f-948e-bb44574ebaee","html_url":"https://github.com/lydell/js-tokens","commit_stats":{"total_commits":145,"total_committers":4,"mean_commits":36.25,"dds":0.1448275862068965,"last_synced_commit":"359b8f10c80de290d8be8c888b9dea05f17b0e50"},"previous_names":[],"tags_count":24,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lydell%2Fjs-tokens","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lydell%2Fjs-tokens/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lydell%2Fjs-tokens/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lydell%2Fjs-tokens/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/lydell","download_url":"https://codeload.github.com/lydell/js-tokens/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":253891783,"owners_count":21979856,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["ecmascript","javascript","regex","tokenizer"],"created_at":"2024-07-31T19:01:08.861Z","updated_at":"2025-05-13T20:12:35.156Z","avatar_url":"https://github.com/lydell.png","language":"JavaScript","funding_links":[],"categories":["JavaScript","🔧 Utilities \u0026 Miscellaneous"],"sub_categories":[],"readme":"# js-tokens\n\nThe tiny, regex powered, lenient, _almost_ spec-compliant JavaScript tokenizer that never fails.\n\n```js\nconst jsTokens = require(\"js-tokens\");\n\nconst jsString = 'JSON.stringify({k:3.14**2}, null /*replacer*/, \"\\\\t\")';\n\nArray.from(jsTokens(jsString), (token) =\u003e token.value).join(\"|\");\n// JSON|.|stringify|(|{|k|:|3.14|**|2|}|,| |null| |/*replacer*/|,| |\"\\t\"|)\n```\n\n## Installation\n\n`npm install js-tokens`\n\n```js\nimport jsTokens from \"js-tokens\";\n// or:\nconst jsTokens = require(\"js-tokens\");\n```\n\n## Usage\n\n```js\njsTokens(string, options?)\n```\n\n| Option | Type      | Default | Description         |\n| :----- | :-------- | :------ | :------------------ |\n| jsx    | `boolean` | `false` | Enable JSX support. |\n\nThis package exports a generator function, `jsTokens`, that turns a string of JavaScript code into token objects.\n\nFor the empty string, the function yields nothing (which can be turned into an empty list). For any other input, the function always yields _something,_ even for invalid JavaScript, and never throws. Concatenating the token values reproduces the input.\n\nThe package is very close to being fully spec compliant (it passes all [but 3](#known-errors) of [test262-parser-tests]), but has taken a couple of shortcuts. See the following sections for limitations of some tokens.\n\n```js\n// Loop over tokens:\nfor (const token of jsTokens(\"hello, !world\")) {\n  console.log(token);\n}\n\n// Get all tokens as an array:\nconst tokens = Array.from(jsTokens(\"hello, !world\"));\n```\n\n## Tokens\n\n_Spec: [ECMAScript Language: Lexical Grammar] + [Additional Syntax]_\n\n```ts\nexport default function jsTokens(input: string): Iterable\u003cToken\u003e;\n\ntype Token =\n  | { type: \"StringLiteral\"; value: string; closed: boolean }\n  | { type: \"NoSubstitutionTemplate\"; value: string; closed: boolean }\n  | { type: \"TemplateHead\"; value: string }\n  | { type: \"TemplateMiddle\"; value: string }\n  | { type: \"TemplateTail\"; value: string; closed: boolean }\n  | { type: \"RegularExpressionLiteral\"; value: string; closed: boolean }\n  | { type: \"MultiLineComment\"; value: string; closed: boolean }\n  | { type: \"SingleLineComment\"; value: string }\n  | { type: \"HashbangComment\"; value: string }\n  | { type: \"IdentifierName\"; value: string }\n  | { type: \"PrivateIdentifier\"; value: string }\n  | { type: \"NumericLiteral\"; value: string }\n  | { type: \"Punctuator\"; value: string }\n  | { type: \"WhiteSpace\"; value: string }\n  | { type: \"LineTerminatorSequence\"; value: string }\n  | { type: \"Invalid\"; value: string };\n```\n\n### StringLiteral\n\n_Spec: [StringLiteral]_\n\nIf the ending `\"` or `'` is missing, the token has `closed: false`. JavaScript strings cannot contain (unescaped) newlines, so unclosed strings simply end at the end of the line.\n\nEscape sequences are supported, but may be invalid. For example, `\"\\u\"` is matched as a StringLiteral even though it contains an invalid escape.\n\nExamples:\n\n\u003c!-- prettier-ignore --\u003e\n```js\n\"string\"\n'string'\n\"\"\n''\n\"\\\"\"\n'\\''\n\"valid: \\u00a0, invalid: \\u\"\n'valid: \\u00a0, invalid: \\u'\n\"multi-\\\nline\"\n'multi-\\\nline'\n\" unclosed\n' unclosed\n```\n\n### NoSubstitutionTemplate / TemplateHead / TemplateMiddle / TemplateTail\n\n_Spec: [NoSubstitutionTemplate] / [TemplateHead] / [TemplateMiddle] / [TemplateTail]_\n\nA template without interpolations is matched as is. For, example:\n\n- `` `abc` ``: NoSubstitutionTemplate\n- `` `abc ``: NoSubstitutionTemplate with `closed: false`\n\nA template _with_ interpolations is matched as many tokens. For example, `` `head${1}middle${2}tail` `` is matched as follows (apart from the two NumericLiterals):\n\n- `` `head${ ``: TemplateHead\n- `}middle${`: TemplateMiddle\n- `` }tail` ``: TemplateTail\n\nTemplateMiddle is optional, and TemplateTail can be unclosed. For example, `` `head${1}tail `` (note the missing ending `` ` ``):\n\n- `` `head${ ``: TemplateHead\n- `}tail`: TemplateTail with `closed: false`\n\nTemplates can contain unescaped newlines, so unclosed templates go on to the end of input.\n\nJust like for StringLiteral, templates can also contain invalid escapes. `` `\\u` `` is matched as a NoSubstitutionTemplate even though it contains an invalid escape. Also note that in _tagged_ templates, invalid escapes are _not_ syntax errors: `` x`\\u` `` is syntactically valid JavaScript.\n\n### RegularExpressionLiteral\n\n_Spec: [RegularExpressionLiteral]_\n\nRegex literals may contain invalid regex syntax. They are still matched as regex literals. The specification even says:\n\n\u003e The productions below [...] are used by the input element scanner to find the end of the regular expression literal. The source text [...] are subsequently parsed again using the more stringent ECMAScript Regular Expression grammar.\n\nIf the ending `/` is missing, the token has `closed: false`. JavaScript regex literals cannot contain newlines (not even escaped ones), so unclosed regex literals simply end at the end of the line.\n\nAccording to the specification, the flags of regular expressions are [IdentifierPart]s (unknown and repeated regex flags become errors at a later stage).\n\nDifferentiating between regex and division in JavaScript is really tricky. js-tokens looks at the previous token to tell them apart. As long as the previous tokens are valid, it should do the right thing. For invalid code, js-tokens might be confused and start matching division as regex or vice versa.\n\nExamples:\n\n\u003c!-- prettier-ignore --\u003e\n```js\n/a/\n/a/gimsuy\n/a/Inva1id\n/+/\n/[/]\\//\n/[\\p{Decimal_Number}--[0-9]]/v\n```\n\n### MultiLineComment\n\n_Spec: [MultiLineComment]_\n\nIf the ending `*/` is missing, the token has `closed: false`. Unclosed multi-line comments go on to the end of the input.\n\nExamples:\n\n\u003c!-- prettier-ignore --\u003e\n```js\n/* comment */\n/* console.log(\n    \"commented\", out + code);\n    */\n/**/\n/* unclosed\n```\n\n### SingleLineComment\n\n_Spec: [SingleLineComment]_\n\nExamples:\n\n\u003c!-- prettier-ignore --\u003e\n```js\n// comment\n// console.log(\"commented\", out + code);\n//\n```\n\n### HashbangComment\n\n_Spec: [HashbangComment]_\n\nNote that a HashbangComment can only occur at the very start of the string that is being tokenized. Anywhere else you will likely get an Invalid token `#` followed by a Punctuator token `!`.\n\nExamples:\n\n\u003c!-- prettier-ignore --\u003e\n```js\n#!/usr/bin/env node\n#! console.log(\"commented\", out + code);\n#!\n```\n\n### IdentifierName\n\n_Spec: [IdentifierName]_\n\nKeywords, reserved words, `null`, `true`, `false`, variable names and property names.\n\nExamples:\n\n\u003c!-- prettier-ignore --\u003e\n```js\nif\nfor\nvar\ninstanceof\npackage\nnull\ntrue\nfalse\nInfinity\nundefined\nNaN\n$variab1e_name\nπ\n℮\nಠ_ಠ\n\\u006C\\u006F\\u006C\\u0077\\u0061\\u0074\n```\n\n### PrivateIdentifier\n\n_Spec: [PrivateIdentifier]_\n\nAny `IdentifierName` preceded by a `#`.\n\nExamples:\n\n\u003c!-- prettier-ignore --\u003e\n```js\n#if\n#for\n#var\n#instanceof\n#package\n#null\n#true\n#false\n#Infinity\n#undefined\n#NaN\n#$variab1e_name\n#π\n#℮\n#ಠ_ಠ\n#\\u006C\\u006F\\u006C\\u0077\\u0061\\u0074\n```\n\n### NumericLiteral\n\n_Spec: [NumericLiteral]_\n\nExamples:\n\n\u003c!-- prettier-ignore --\u003e\n```js\n0\n1.5\n1\n1_000\n12e9\n0.123e-32\n0xDead_beef\n0b110\n12n\n07\n09.5\n```\n\n### Punctuator\n\n_Spec: [Punctuator] + [DivPunctuator] + [RightBracePunctuator]_\n\nAll possible values:\n\n\u003c!-- prettier-ignore --\u003e\n```js\n--  ++\n.   ?.\n\u003c   \u003c=   \u003e   \u003e=\n!=  !==  ==  ===\n   +   -   %   \u0026   |   ^   /   *   **   \u003c\u003c   \u003e\u003e   \u003e\u003e\u003e   \u0026\u0026   ||   ??\n=  +=  -=  %=  \u0026=  |=  ^=  /=  *=  **=  \u003c\u003c=  \u003e\u003e=  \u003e\u003e\u003e=  \u0026\u0026=  ||=  ??=\n(  )  [  ]  {  }\n!  ?  :  ;  ,  ~  ...  =\u003e\n```\n\n### WhiteSpace\n\n_Spec: [WhiteSpace]_\n\nUnlike the specification, multiple whitespace characters in a row are matched as _one_ token, not one token per character.\n\n### LineTerminatorSequence\n\n_Spec: [LineTerminatorSequence]_\n\nCR, LF and CRLF, plus `\\u2028` and `\\u2029`.\n\n### Invalid\n\n_Spec: n/a_\n\nSingle code points not matched in another token.\n\nExamples:\n\n\u003c!-- prettier-ignore --\u003e\n```js\n#\n@\n💩\n```\n\n## JSX Tokens\n\n_Spec: [JSX Specification]_\n\n```ts\nexport default function jsTokens(\n  input: string,\n  options: { jsx: true },\n): Iterable\u003cToken | JSXToken\u003e;\n\nexport declare type JSXToken =\n  | { type: \"JSXString\"; value: string; closed: boolean }\n  | { type: \"JSXText\"; value: string }\n  | { type: \"JSXIdentifier\"; value: string }\n  | { type: \"JSXPunctuator\"; value: string }\n  | { type: \"JSXInvalid\"; value: string };\n```\n\n- The tokenizer switches between outputting runs of `Token` and runs of `JSXToken`.\n- Runs of `JSXToken` can also contain WhiteSpace, LineTerminatorSequence, MultiLineComment and SingleLineComment.\n\n### JSXString\n\n_Spec: `\"` JSXDoubleStringCharacters `\"` + `'` JSXSingleStringCharacters `'`_\n\nIf the ending `\"` or `'` is missing, the token has `closed: false`. JSX strings can contain unescaped newlines, so unclosed JSX strings go on to the end of input.\n\nNote that JSX don’t support escape sequences as part of the token grammar. A `\"` or `'` always closes the string, even with a backslash before.\n\nExamples:\n\n\u003c!-- prettier-ignore --\u003e\n```\n\"string\"\n'string'\n\"\"\n''\n\"\\\"\n'\\'\n\"multi-\nline\"\n'multi-\nline'\n\" unclosed\n' unclosed\n```\n\n### JSXText\n\n_Spec: JSXText_\n\nAnything but `\u003c`, `\u003e`, `{` and `}`.\n\n### JSXIdentifier\n\n_Spec: JSXIdentifier_\n\nExamples:\n\n\u003c!-- prettier-ignore --\u003e\n```js\ndiv\nclass\nxml\nx-element\nx------\n$htm1_element\nಠ_ಠ\n```\n\n### JSXPunctuator\n\n_Spec: n/a_\n\nAll possible values:\n\n```js\n\u003c\n\u003e\n/\n.\n:\n=\n{\n}\n```\n\n### JSXInvalid\n\n_Spec: n/a_\n\nSingle code points not matched in another token.\n\nExamples in JSX tags:\n\n```js\n1\n`\n+\n,\n#\n@\n💩\n```\n\nAll possible values in JSX children:\n\n```js\n\u003e\n}\n```\n\n## Compatibility\n\n### ECMAScript\n\nThe intention is to always support the latest ECMAScript version whose feature set has been finalized.\n\nCurrently, ECMAScript 2024 is supported.\n\n#### Annex B and C (strict mode)\n\nSection [B: Additional ECMAScript Features for Web Browsers][annexb] of the spec is optional if the ECMAScript host is not a web browser, and specifies some additional syntax. Section [C: The Strict Mode of ECMAScript][annexc] disallows certain syntax in Strict Mode.\n\n- Numeric literals: js-tokens supports legacy octal and octal like numeric literals, regardless of Strict Mode.\n- String literals: js-tokens supports legacy octal escapes, since it allows any invalid escapes.\n- HTML-like comments: **Not supported.** js-tokens prefers treating `5\u003c!--x` as `5 \u003c !(--x)` rather than as `5 //x`.\n- Regular expression patterns: js-tokens doesn’t care what’s between the starting `/` and ending `/`, so this is supported.\n\n### TypeScript\n\nSupporting TypeScript is not an explicit goal, but js-tokens and Babel both tokenize this [TypeScript fixture] and this [TSX fixture] the same way, with one edge case:\n\n\u003c!-- prettier-ignore --\u003e\n```ts\ntype A = Array\u003cArray\u003cstring\u003e\u003e\ntype B = Array\u003cArray\u003cArray\u003cstring\u003e\u003e\u003e\n```\n\nBoth lines above should end with a couple of `\u003e` tokens, but js-tokens instead matches the `\u003e\u003e` and `\u003e\u003e\u003e` operators.\n\n### JSX\n\nJSX is supported: `jsTokens(\"\u003cp\u003eHello, world!\u003c/p\u003e\", { jsx: true })`.\n\n### JavaScript runtimes\n\njs-tokens should work in any JavaScript runtime that supports [Unicode property escapes].\n\n### Known errors\n\nHere are a couple of tricky cases:\n\n\u003c!-- prettier-ignore --\u003e\n```js\n// Case 1:\nswitch (x) {\n  case x: {}/a/g;\n  case x: {}\u003cdiv\u003ex\u003c/div\u003e/g;\n}\n\n// Case 2:\nlabel: {}/a/g;\nlabel: {}\u003cdiv\u003ex\u003c/div\u003e/g;\n\n// Case 3:\n(function f() {}/a/g);\n(function f() {}\u003cdiv\u003ex\u003c/div\u003e/g);\n```\n\nThis is what they mean:\n\n```js\n// Case 1:\nswitch (x) {\n  case x:\n    {\n    }\n    /a/g;\n  case x:\n    {\n    }\n    \u003cdiv\u003ex\u003c/div\u003e / g;\n}\n\n// Case 2:\nlabel: {\n}\n/a/g;\nlabel: {\n}\n\u003cdiv\u003ex\u003c/div\u003e / g;\n\n// Case 3:\n(function f() {}) / a / g;\n(function f() {}) \u003c div \u003e x \u003c /div\u003e/g;\n```\n\nBut js-tokens thinks they mean:\n\n```js\n// Case 1:\nswitch (x) {\n  case x:\n    ({}) / a / g;\n  case x:\n    ({}) \u003c div \u003e x \u003c /div\u003e/g;\n}\n\n// Case 2:\nlabel: ({}) / a / g;\nlabel: ({}) \u003c div \u003e x \u003c /div\u003e/g;\n\n// Case 3:\nfunction f() {}\n/a/g;\nfunction f() {}\n\u003cdiv\u003ex\u003c/div\u003e / g;\n```\n\nIn other words, js-tokens:\n\n- Mis-identifies regex as division and JSX as comparison in case 1 and 2.\n- Mis-identifies division as regex and comparison as JSX in case 3.\n\nThis happens because js-tokens looks at the previous token when deciding between regex and division or JSX and comparison. In these cases, the previous token is `}`, which either means “end of block” (→ regex/JSX) or “end of object literal” (→ division/comparison). How does js-tokens determine if the `}` belongs to a block or an object literal? By looking at the token before the matching `{`.\n\nIn case 1 and 2, that’s a `:`. A `:` _usually_ means that we have an object literal or ternary:\n\n\u003c!-- prettier-ignore --\u003e\n```js\nlet some = weird ? { value: {}/a/g } : {}/a/g;\n```\n\nBut `:` is also used for `case` and labeled statements.\n\nOne idea is to look for `case` before the `:` as an exception to the rule, but it’s not so easy:\n\n\u003c!-- prettier-ignore --\u003e\n```js\nswitch (x) {\n  case weird ? true : {}/a/g: {}/a/g\n}\n```\n\nThe first `{}/a/g` is a division, while the second `{}/a/g` is an empty block followed by a regex. Both are preceded by a colon with a `case` on the same line, and it does not seem like you can distinguish between the two without implementing a parser.\n\nLabeled statements are similarly difficult, since they are so similar to object literals:\n\n\u003c!-- prettier-ignore --\u003e\n```js\n{\n  label: {}/a/g\n}\n\n({\n  key: {}/a/g\n})\n```\n\nFinally, case 3 (`(function f() {}/a/g);`) is also difficult, because a `)` before a `{` means that the `{` is part of a _block,_ and blocks are _usually_ statements:\n\n```js\nif (x) {\n}\n/a/g;\n\nfunction f() {}\n/a/g;\n```\n\nBut _function expressions_ are of course not statements. It’s difficult to tell an function _expression_ from a function _statement_ without parsing.\n\nLuckily, none of these edge cases are likely to occur in real code.\n\n### Known failures\n\njs-tokens advertises that it “never fails”. Tell you what, it _can_ fail on extreme inputs. The regex engine of the runtime can eventually give up. js-tokens has worked around it to some extent by changing its regexes to be easier on the regex engine. To solve completely, js-tokens would have to stop using regex, but then it wouldn’t be _tiny_ anymore which is the whole point. Luckily, only extreme inputs can fail, hopefully ones you’ll never encounter.\n\nFor example, if you try to parse the string literal `\"\\n\\n\\n\"` but with 10 million `\\n` instead of just 3, the regex engine gives up with `RangeError: Maximum call stack size exceeded` (or similar). Try it out:\n\n```js\nArray.from(require(\"js-tokens\")(`\"${\"\\\\n\".repeat(1e7)}\"`));\n```\n\n(Yes, that is the _regex engine_ of the runtime giving up. js-tokens has no recursive functions.)\n\nHowever, if you repeat `a` instead of `\\n` 10 million times (`\"aaaaaa…\"`), it works:\n\n```js\nArray.from(require(\"js-tokens\")(`\"${\"a\".repeat(1e7)}\"`));\n```\n\nThat’s good, because it’s much more common to have lots of non-escapes in a row in a big string literal, than having mostly escapes. (Obfuscated code might have _only_ escapes though.)\n\n#### Safari warning\n\nI’ve seen Safari _give up_ instead of throwing an error.\n\nIn Safari, Chrome, Firefox and Node.js the following code successfully results in a match:\n\n```js\n/(#)(?:a|b)+/.exec(\"#\" + \"a\".repeat(1e5));\n```\n\nBut for the following code (with `1e7` instead of `1e5`), the runtimes differ:\n\n```js\n/(#)(?:a|b)+/.exec(\"#\" + \"a\".repeat(1e7));\n```\n\n- Chrome, Firefox and Node.js all throw `RangeError: Maximum call stack size exceeded` (or similar).\n- Safari returns `null` (at the time of writing), silently giving up on matching the regex. It’s kind of lying that the regex did not match, while in reality it would given enough computing resources.\n\nThis means that in Safari, js-tokens might not fail but instead give you unexpected tokens.\n\n## Performance\n\nWith [@babel/parser] for comparison. Node.js 21.6.1 on a MacBook Pro M1 (Sonoma).\n\n| Lines of code |     Size | js-tokens@8.0.3 | @babel/parser@7.23.9 |\n| ------------: | -------: | --------------: | -------------------: |\n|          ~100 | ~4.0 KiB |           ~2 ms |               ~10 ms |\n|        ~1 000 |  ~39 KiB |           ~5 ms |               ~27 ms |\n|       ~10 000 | ~353 KiB |          ~44 ms |              ~108 ms |\n|      ~100 000 | ~5.1 MiB |         ~333 ms |               ~2.0 s |\n|    ~2 400 000 | ~138 MiB |            ~7 s |        ~4 m 9 s (\\*) |\n\n(\\*) Required increasing the Node.js the memory limit (I set it to 8 GiB).\n\nSee [benchmark.js] if you want to run benchmarks yourself.\n\n[@babel/parser]: https://babeljs.io/docs/en/babel-parser\n[additional syntax]: https://tc39.es/ecma262/#sec-additional-syntax\n[annexb]: https://tc39.es/ecma262/#sec-additional-ecmascript-features-for-web-browsers\n[annexc]: https://tc39.es/ecma262/#sec-strict-mode-of-ecmascript\n[benchmark.js]: benchmark.js\n[divpunctuator]: https://tc39.es/ecma262/#prod-DivPunctuator\n[ecmascript language: lexical grammar]: https://tc39.es/ecma262/#sec-ecmascript-language-lexical-grammar\n[example.test.js]: test/example.test.js\n[hashbangcomment]: https://tc39.es/ecma262/#prod-HashbangComment\n[identifiername]: https://tc39.es/ecma262/#prod-IdentifierName\n[identifierpart]: https://tc39.es/ecma262/#prod-IdentifierPart\n[jsx specification]: https://facebook.github.io/jsx/\n[lineterminatorsequence]: https://tc39.es/ecma262/#prod-LineTerminatorSequence\n[multilinecomment]: https://tc39.es/ecma262/#prod-MultiLineComment\n[nosubstitutiontemplate]: https://tc39.es/ecma262/#prod-NoSubstitutionTemplate\n[numericliteral]: https://tc39.es/ecma262/#prod-NumericLiteral\n[privateidentifier]: https://tc39.es/ecma262/#prod-PrivateIdentifier\n[punctuator]: https://tc39.es/ecma262/#prod-Punctuator\n[regularexpressionliteral]: https://tc39.es/ecma262/#prod-RegularExpressionLiteral\n[rightbracepunctuator]: https://tc39.es/ecma262/#prod-RightBracePunctuator\n[singlelinecomment]: https://tc39.es/ecma262/#prod-SingleLineComment\n[stackoverflow-slash]: https://stackoverflow.com/a/27120110/2010616\n[string.prototype.matchall]: https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/String/matchAll\n[stringliteral]: https://tc39.es/ecma262/#prod-StringLiteral\n[templatehead]: https://tc39.es/ecma262/#prod-TemplateHead\n[templatemiddle]: https://tc39.es/ecma262/#prod-TemplateMiddle\n[templatetail]: https://tc39.es/ecma262/#prod-TemplateTail\n[test262-parser-tests]: https://github.com/tc39/test262-parser-tests\n[tsx fixture]: test/fixtures/valid/jsx2.tsx\n[typescript fixture]: test/fixtures/valid/typescript.module.ts\n[unicode property escapes]: https://developer.mozilla.org/en-US/docs/Web/JavaScript/Guide/Regular_Expressions/Unicode_Property_Escapes\n[whitespace]: https://tc39.es/ecma262/#prod-WhiteSpace\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Flydell%2Fjs-tokens","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Flydell%2Fjs-tokens","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Flydell%2Fjs-tokens/lists"}