{"id":15626077,"url":"https://github.com/pvdz/tenko","last_synced_at":"2025-04-13T07:51:28.115Z","repository":{"id":47693456,"uuid":"220313604","full_name":"pvdz/tenko","owner":"pvdz","description":"An 100% spec compliant ES2021 JavaScript parser written in JS","archived":false,"fork":false,"pushed_at":"2021-08-17T19:24:14.000Z","size":31906,"stargazers_count":511,"open_issues_count":16,"forks_count":14,"subscribers_count":8,"default_branch":"master","last_synced_at":"2025-04-04T06:08:37.445Z","etag":null,"topics":["ast","ecmascript","javascript","parser"],"latest_commit_sha":null,"homepage":"https://pvdz.github.io/tenko/repl","language":"JavaScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/pvdz.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2019-11-07T19:29:17.000Z","updated_at":"2025-03-29T04:09:17.000Z","dependencies_parsed_at":"2022-08-22T15:32:34.257Z","dependency_job_id":null,"html_url":"https://github.com/pvdz/tenko","commit_stats":null,"previous_names":[],"tags_count":18,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/pvdz%2Ftenko","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/pvdz%2Ftenko/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/pvdz%2Ftenko/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/pvdz%2Ftenko/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/pvdz","download_url":"https://codeload.github.com/pvdz/tenko/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248681494,"owners_count":21144700,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["ast","ecmascript","javascript","parser"],"created_at":"2024-10-03T10:10:07.098Z","updated_at":"2025-04-13T07:51:23.100Z","avatar_url":"https://github.com/pvdz.png","language":"JavaScript","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Tenko\n\nA \"pixel perfect\" 100% spec compliant JavaScript parser written in JavaScript, parsing ES6/ES2015 - ES2021.\n\nREPL: https://pvdz.github.io/tenko/repl\n\n- Supports:\n  - Anything stage 4 up to ES2021\n  - Regex syntax (deep)\n  - Parsing modes:\n    - Sloppy / non-strict\n    - Web compat / AnnexB\n    - Strict\n    - Module\n- AST\n  - Is optional, enabled by default\n  - Estree (default)\n  - (Optional chaining AST works but AST spec seems still in flux)\n  - Acorn\n  - Babel (anything stage 4, except comments)\n  - Supports location data (matching Acorn/Babel for reference)\n- Tests\n  - 33k input syntax tests\n  - Passes test262 suite (at least as per March 2020), without exception\n\n# Name\n\nThe name is short for \"The Parser Formerly Known As ZeParser3\".\n\nIt's also an anagram for \"Token\", perfectly fitting this project.\n\nIn Japanese it's a [divine beast](https://en.wikipedia.org/wiki/Tenko_(fox)) (\"heavenly fox\" or \"celestial fox\"), playing into my nicknames.\n\n# REPL\n\nYou can find the REPL in [`repl/index.html`](./repl/index.html), github link: https://pvdz.github.io/tenko/repl\n\n_The REPL runs on dev master branch and needs a very new browser due to es module syntax._\n\n## Usage\n\n```js\nimport {Tenko, GOAL_MODULE, COLLECT_TOKENS_ALL} from 'src/index.mjs';\nconst {\n  ast,                 // estree compatible AST\n  tokens,              // array of numbers (see Lexer)\n  tokenCountSolid,     // number of non-whitespace tokens\n  tokenCountAny,       // number of tokens of any kind\n} = Tenko(\n  inputCode,           // string\n  {\n    // Parse with script or module goal (module allows import/export)\n    goalMode = GOAL_MODULE, // GOAL_MODULE | GOAL_SCRIPT | \"module\" | \"script\"\n    // Do you want to collect generated tokens at all?\n    collectTokens = COLLECT_TOKENS_ALL, // COLLECT_TOKENS_ALL | COLLECT_TOKENS_SOLID | COLLECT_TOKENS_NONE | COLLECT_TOKENS_TYPES | \"all\" | \"solid\" | \"none\" | \"types\"\n    // Apply Annex B rules? (Only works in sloppy mode)\n    webCompat = true,\n    // Start parsing as if in strict mode? (Works with script goal)\n    strictMode = false,\n    // Output a Babel compatible AST? Note: comment nodes are not properly mirrored\n    babelCompat = false,\n    // Add a loc (with `{start: {line, column}, stop: {line, column}}`) to each token?\n    babelTokenCompat = false,\n    // Pass on a reference that will be used as the AST root\n    astRoot = null,\n    // Should it normalize \\r and \\r\\n to \\n in the .raw of template nodes?\n    // Estree spec but makes it hard to serialize template nodes losslessly\n    templateNewlineNormalization = true,\n    // Pass on a reference to store the tokens\n    tokenStorage = [],\n    // Callback to receive the lexer instance once its created\n    getLexer = null, // getLexer(lexer)\n    // You use this to parse `eval` code\n    allowGlobalReturn = false,\n    // Target a very specific ecmascript version (like, reject async). Number; 6 - 12, or 2015 - 2021, or Infinity.\n    targetEsVersion = lastVersion, // (Last supported version is currently ES2021)\n    // Leave built up scope information in the ASTs (good luck)\n    exposeScopes = false,\n    // Assign each node a unique incremental id\n    astUids = false,\n    // Do you want to print a code frame with error messages? (Part of the input around the point of error)\n    errorCodeFrame = true,\n    // For the code frame, do you want to always show the entire input, regardless of size? Or just a small context\n    truncCodeFrame = true,\n    // You can override the logging functions to catch or squash all output\n    $log = console.log,\n    $warn = console.warn,\n    $error = console.error,\n    // Value ot use for the `source` field of each `loc` object\n    sourceField = '',\n    // Generate a `range: {start: number, end: number}` property on all loc objects (does not require `locationTracking`)\n    ranges = false,\n    // Generate a `range: [start: number, end: number]` property on all nodes. `input.slice(range[0], range[1])` should get you the text for a node.\n    nodeRange = false,\n    // Do not populate loc properties on AST nodes (property will be undefined). Since v\u003cunpublished\u003e\n    locationTracking = true,\n  }\n);\n```\n\n## Development\n\nThere is a single entry point in the root project called `t` which calls `tests/t.sh` which calls out to various development related scripts.\n\n### ES modules\n\nNote that the files use `import` and `export` declarations and `import()`, which requires node 10+ or a cutting edge browser.\n\nAt the time of writing node requires the experimental `--experimental-modules` flag.\n\nIt's a burden in some ways and nice in others. A prod build would not have any modules.\n\n### Test cases\n\nAll test cases are in \"special\" plain-text .md files. See [`tests/testcases/README.md`](./tests/testcases/README.md) for details on formatting those.\n\n### Entry point\n\nSome interesting usages of `./t`:\n\n```\n# Show help\n./t --help\n\n# Run and update (inline) all tests. \n# Use git diff to see changes. Will bail fast on unexpected or assertion errors.\n# This tests four modes (sloppy, strict, module, and sloppy-web-compat)\n# This also tests the printer on the first successful parse\n./t u\n# Run all tests step-by-step (same as above) and ask what to do for any changes\n./t m\n\n# Same as `./t u` but compare it against Babel or Acorn. Recorded changes should be discarded afterwards.\n# Use this to test against AST differences. If there are any they will be printed explicitly.\n# Acorn:\n./t a\n# Babel:\n./t b\n\n# Test a particular input from cli\n./t i \"some.input()\"\n# Test a particular test file\n./t f \"tests/testcases/regexes/foo.md\"\n# Use entire contents of given file as input\n./t F \"test262/test/annexB/built-ins/foo.js\"\n\n# Generate prod builds\n# Generate a build. Strips ASSERT*, inline many constants\n./t z\n# Same as above but explicitly set `acornCompat` and `babelCompat` to `false`.\n./t z --no-compat\n# Generate pretty builds for debugging without asserts:\n./t --pretty\n# Minified build with Terser (will lower performance due to inlining)\n./t --min\n\n# Run test262 tests (requires cloning https://github.com/tc39/test262 into tenko/ignore/test262)\n./t t\n\n# Fuzz the parser\n./t fuzz\n\n# Regenerate all autogen test files. Regenerates files still need to be updated (`./t u`).\n# All files, regardless:\n./t g\n# Only create new files:\n./t G\n\n# Find out which tests execute a particular code branch in the parser\n# Add `HIT()` to any part of the code in src\n# Reports (only) all inputs that trigger a `HIT()` call in Tenko\n./t s\n```\n\nSome tooling that requires additional setup;\n\n```\n# Benchmarks (requires benchmark files in projroot/ignore/perf);\n# Simply spawn new node process and run test:\n./t p\n# Run benchmarks repeatedly and report results\n./t p6\n# Configure machine to be as stable as possible (DANGEROUS, read the script before using it, requires root). All \n# changes should be reset after reboot. Then run the benchmarks in the shielded cpus at RT prio (also requires root).\n./t stable\n./t p6 --stabled\n# Same as above but without running `./t stable` previously, and tries to undo certain (but not all) things afterwards\n./t p6 --stable\n\n# Investigate v8 perf regressions with deoptigate:\n./t deoptigate\n\n# Profile the parser in Chrome devtools (open the tab through `about://inspect`)\n./t devtools\n\n# Run a visual heatmap profiler for counts based investigation (private)\n./t hf\n```\n\nThere are many flags. Some are specific to an action, others are generic. Some examples:\n\n```\n--sloppy             Run in non-strict mode (but non-web compat!)\n--strict             Run with script goal but consider the code strict\n--module             Run with module goal (enabling strict mode by default)\n--web                Run with script goal, non-strict, and enable web compat (AnnexB rules)\n--annexb             Force enable AnnexB rules, regardless of mode\n\n6                    Run as close to the rules as of ES6  / ES2015 as possible\n7                    Run as close to the rules as of ES7  / ES2016 as possible\n8                    Run as close to the rules as of ES8  / ES2017 as possible\n9                    Run as close to the rules as of ES9  / ES2018 as possible\n10                   Run as close to the rules as of ES10 / ES2019 as possible\n11                   Run as close to the rules as of ES11 / ES2020 as possible\n12                   Run as close to the rules as of ES11 / ES2021 as possible\n2015\n2016\n2017\n2018\n2019\n2020\n2021\n\n--min                Given a broken input, brute force minify the input while maintaining the same error message\n--acorn              Output a Acorn compatible AST\n--babel              Output a Babel compatible AST\n--test-acorn         Compare the `--acorn` output to the actual output of Acorn on same input (./t a)\n--test-babel         Compare the `--babel` output to the actual output of Babel on same input (./t b)\n--test-node          Compile input in a `Function()` and report whether that throws when Tenko throws, for fuzzer\n--build              Use a prod build (from standard output location), instead of dev sources, for all actions that support it\n--nb                 Do not build (many actions will kick of a build before doing their thing, this prevents that)\n```\n\nAnd many more. For details, `./t --help` should give you an up to date list of all actions and options.\n\n# Building\n\nWhile the parser runs perfectly fine in dev mode it will be a bit slow. A build:\n\n- will remove non-assert dev artifacts\n- can remove inline asserts (lines that start with `ASSERT`)\n- can remove all the AST generation from the build (lines that start with `AST`)\n- inlines many constants used by the parser as enums or bitwise fields\n\nTo generate a build run this in the project root, flags can be combined:\n\n```\n./t z                      # Regular build with everything\n./t z --no-ast             # Strip most AST related code from the build (~50% faster, but obviously no AST)\n./t z --no-compat          # Strip acorn/babel compatibility\n./t z --min                # Run Terser on result (will decrease performance due to inlining)\n./t z --pretty             # Run Prettier on build result\n```\n\nNote that this (initially) uses my own printer to print the AST.\n\nThe build script writes and ESM and CJS file to `./build`\n\n## Validation without AST\n\nThe no-AST build can validate JS almost as perfect as the regular build except for certain validation cases where it requires the AST:\n\n- Binary op after arrow with block body (`()=\u003e{}*x` is illegal)\n- Regular expression on new line after arrow with block body (`()=\u003e{} \\n /foo/g`, prohibited by ASI rules and can't be a division)\n- Update operator anything that's writable but not a valid var or member expression (`++[]`)\n- Delete with an ident that is wrapped in parenthesis (`delete (foo)` is illegal), trivial cases (`delete foo;`) should be fine\n\n# Testing\n\nEach test is individually encapsulated in an `.md` file in `tests/testcases/**`. This file will contain the input code and the output as expected for sloppy mode, strict mode (script goal), module goal, and web compat mode (only works in sloppy mode, script goal).\n\nIf a run passes then the AST and types of tokens are printed in the output. This AST is also printed with `src/tools/printer` and its output is checked to produce the same AST.\n \nIf a run does not pass the error message and a pointer to where the error occurred are stored in the file.\n\nThe files can be auto-updated with `./t u` or `./t m`. This makes it easy to update something in the parser and use git to confirm whether anything changed, and if so what.\n\nThere are also `autogen.md` files, which generate a bunch of combinatorial tests (`./t g` or `./t G`), similar to the other tests.\n\nTo create a new test simply add a new file, start it with `@`, a description, a line with only `###` and the rest will be considered the test case verbatim. When you run the test runner this file will automatically be converted to a proper test case.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fpvdz%2Ftenko","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fpvdz%2Ftenko","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fpvdz%2Ftenko/lists"}