{"id":13492000,"url":"https://github.com/guybedford/es-module-lexer","last_synced_at":"2025-05-12T13:17:52.921Z","repository":{"id":37444411,"uuid":"201167785","full_name":"guybedford/es-module-lexer","owner":"guybedford","description":"Low-overhead lexer dedicated to ES module parsing for fast analysis","archived":false,"fork":false,"pushed_at":"2025-04-22T22:21:49.000Z","size":2203,"stargazers_count":974,"open_issues_count":11,"forks_count":52,"subscribers_count":7,"default_branch":"main","last_synced_at":"2025-05-08T15:04:56.073Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"JavaScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/guybedford.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2019-08-08T03:05:41.000Z","updated_at":"2025-05-07T06:55:40.000Z","dependencies_parsed_at":"2023-11-10T02:53:47.091Z","dependency_job_id":"d481b7db-112d-4eed-b8f7-a9f3f812e6b9","html_url":"https://github.com/guybedford/es-module-lexer","commit_stats":{"total_commits":209,"total_committers":17,"mean_commits":"12.294117647058824","dds":"0.13875598086124397","last_synced_commit":"36ce8f0162c435b722f16c2a41804ce4a76470d5"},"previous_names":[],"tags_count":66,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/guybedford%2Fes-module-lexer","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/guybedford%2Fes-module-lexer/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/guybedford%2Fes-module-lexer/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/guybedford%2Fes-module-lexer/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/guybedford","download_url":"https://codeload.github.com/guybedford/es-module-lexer/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":253282908,"owners_count":21883528,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-07-31T19:01:02.255Z","updated_at":"2025-05-12T13:17:52.893Z","avatar_url":"https://github.com/guybedford.png","language":"JavaScript","readme":"# ES Module Lexer\n\n[![Build Status][actions-image]][actions-url]\n\nA JS module syntax lexer used in [es-module-shims](https://github.com/guybedford/es-module-shims).\n\nOutputs the list of exports and locations of import specifiers, including dynamic import and import meta handling.\n\nSupports new syntax features including import attributes and source phase imports.\n\nA very small single JS file (4KiB gzipped) that includes inlined Web Assembly for very fast source analysis of ECMAScript module syntax only.\n\nFor an example of the performance, Angular 1 (720KiB) is fully parsed in 5ms, in comparison to the fastest JS parser, Acorn which takes over 100ms.\n\n_Comprehensively handles the JS language grammar while remaining small and fast. - ~10ms per MB of JS cold and ~5ms per MB of JS warm, [see benchmarks](#benchmarks) for more info._\n\n\u003e [Built with](https://github.com/guybedford/es-module-lexer/blob/main/chompfile.toml) [Chomp](https://chompbuild.com/)\n\n### Usage\n\n```\nnpm install es-module-lexer\n```\n\nSee [src/lexer.ts](src/lexer.ts) for the type definitions.\n\nFor use in CommonJS:\n\n```js\nconst { init, parse } = require('es-module-lexer');\n\n(async () =\u003e {\n  // either await init, or call parse asynchronously\n  // this is necessary for the Web Assembly boot\n  await init;\n\n  const source = 'export var p = 5';\n  const [imports, exports] = parse(source);\n  \n  // Returns \"p\"\n  source.slice(exports[0].s, exports[0].e);\n  // Returns \"p\"\n  source.slice(exports[0].ls, exports[0].le);\n})();\n```\n\nAn ES module version is also available:\n\n```js\nimport { init, parse } from 'es-module-lexer';\n\n(async () =\u003e {\n  await init;\n\n  const source = `\n    import { name } from 'mod\\\\u1011';\n    import json from './json.json' assert { type: 'json' }\n    export var p = 5;\n    export function q () {\n\n    };\n    export { x as 'external name' } from 'external';\n\n    // Comments provided to demonstrate edge cases\n    import /*comment!*/ (  'asdf', { assert: { type: 'json' }});\n    import /*comment!*/.meta.asdf;\n\n    // Source phase imports:\n    import source mod from './mod.wasm';\n    import.source('./mod.wasm');\n  `;\n\n  const [imports, exports] = parse(source, 'optional-sourcename');\n\n  // Returns \"modထ\"\n  imports[0].n\n  // Returns \"mod\\u1011\"\n  source.slice(imports[0].s, imports[0].e);\n  // \"s\" = start\n  // \"e\" = end\n\n  // Returns \"import { name } from 'mod'\"\n  source.slice(imports[0].ss, imports[0].se);\n  // \"ss\" = statement start\n  // \"se\" = statement end\n\n  // Returns \"{ type: 'json' }\"\n  source.slice(imports[1].a, imports[1].se);\n  // \"a\" = assert, -1 for no assertion\n\n  // Returns \"external\"\n  source.slice(imports[2].s, imports[2].e);\n\n  // Returns \"p\"\n  source.slice(exports[0].s, exports[0].e);\n  // Returns \"p\"\n  source.slice(exports[0].ls, exports[0].le);\n  // Returns \"q\"\n  source.slice(exports[1].s, exports[1].e);\n  // Returns \"q\"\n  source.slice(exports[1].ls, exports[1].le);\n  // Returns \"'external name'\"\n  source.slice(exports[2].s, exports[2].e);\n  // Returns -1\n  exports[2].ls;\n  // Returns -1\n  exports[2].le;\n\n  // Import type is provided by `t` value\n  // (1 for static, 2, for dynamic)\n  // Returns true\n  imports[2].t == 2;\n\n  // Returns \"asdf\" (only for string literal dynamic imports)\n  imports[2].n\n  // Returns \"import /*comment!*/ (  'asdf', { assert: { type: 'json' } })\"\n  source.slice(imports[3].ss, imports[3].se);\n  // Returns \"'asdf'\"\n  source.slice(imports[3].s, imports[3].e);\n  // Returns \"(  'asdf', { assert: { type: 'json' } })\"\n  source.slice(imports[3].d, imports[3].se);\n  // Returns \"{ assert: { type: 'json' } }\"\n  source.slice(imports[3].a, imports[3].se - 1);\n\n  // For non-string dynamic import expressions:\n  // - n will be undefined\n  // - a is currently -1 even if there is an assertion\n  // - e is currently the character before the closing )\n\n  // For nested dynamic imports, the se value of the outer import is -1 as end tracking does not\n  // currently support nested dynamic immports\n\n  // import.meta is indicated by imports[3].d === -2\n  // Returns true\n  imports[4].d === -2;\n  // Returns \"import /*comment!*/.meta\"\n  source.slice(imports[4].s, imports[4].e);\n  // ss and se are the same for import meta\n\n  // Returns \"'./mod.wasm'\"\n  source.slice(imports[5].s, imports[5].e);\n\n  // Import type 4 and 5 for static and dynamic source phase\n  imports[5].t === 4;\n  imports[6].t === 5;\n})();\n```\n\n### CSP asm.js Build\n\nThe default version of the library uses Wasm and (safe) eval usage for performance and a minimal footprint.\n\nNeither of these represent security escalation possibilities since there are no execution string injection vectors, but that can still violate existing CSP policies for applications.\n\nFor a version that works with CSP eval disabled, use the `es-module-lexer/js` build:\n\n```js\nimport { parse } from 'es-module-lexer/js';\n```\n\nInstead of Web Assembly, this uses an asm.js build which is almost as fast as the Wasm version ([see benchmarks below](#benchmarks)).\n\n### Escape Sequences\n\nTo handle escape sequences in specifier strings, the `.n` field of imported specifiers will be provided where possible.\n\nFor dynamic import expressions, this field will be empty if not a valid JS string.\n\n### Facade Detection\n\nFacade modules that only use import / export syntax can be detected via the third return value:\n\n```js\nconst [,, facade] = parse(`\n  export * from 'external';\n  import * as ns from 'external2';\n  export { a as b } from 'external3';\n  export { ns };\n`);\nfacade === true;\n```\n\n### ESM Detection\n\nModules that uses ESM syntaxes can be detected via the fourth return value:\n\n```js\nconst [,,, hasModuleSyntax] = parse(`\n  export {}\n`);\nhasModuleSyntax === true;\n```\n\nDynamic imports are ignored since they can be used in Non-ESM files.\n\n```js\nconst [,,, hasModuleSyntax] = parse(`\n  import('./foo.js')\n`);\nhasModuleSyntax === false;\n```\n\n### Environment Support\n\nNode.js 10+, and [all browsers with Web Assembly support](https://caniuse.com/#feat=wasm).\n\n### Grammar Support\n\n* Token state parses all line comments, block comments, strings, template strings, blocks, parens and punctuators.\n* Division operator / regex token ambiguity is handled via backtracking checks against punctuator prefixes, including closing brace or paren backtracking.\n* Always correctly parses valid JS source, but may parse invalid JS source without errors.\n\n### Limitations\n\nThe lexing approach is designed to deal with the full language grammar including RegEx / division operator ambiguity through backtracking and paren / brace tracking.\n\nThe only limitation to the reduced parser is that the \"exports\" list may not correctly gather all export identifiers in the following edge cases:\n\n```js\n// Only \"a\" is detected as an export, \"q\" isn't\nexport var a = 'asdf', q = z;\n\n// \"b\" is not detected as an export\nexport var { a: b } = asdf;\n```\n\nThe above cases are handled gracefully in that the lexer will keep going fine, it will just not properly detect the export names above.\n\n### Benchmarks\n\nBenchmarks can be run with `npm run bench`.\n\nCurrent results for a high spec machine:\n\n#### Wasm Build\n\n```\nModule load time\n\u003e 5ms\nCold Run, All Samples\ntest/samples/*.js (3123 KiB)\n\u003e 18ms\n\nWarm Runs (average of 25 runs)\ntest/samples/angular.js (739 KiB)\n\u003e 3ms\ntest/samples/angular.min.js (188 KiB)\n\u003e 1ms\ntest/samples/d3.js (508 KiB)\n\u003e 3ms\ntest/samples/d3.min.js (274 KiB)\n\u003e 2ms\ntest/samples/magic-string.js (35 KiB)\n\u003e 0ms\ntest/samples/magic-string.min.js (20 KiB)\n\u003e 0ms\ntest/samples/rollup.js (929 KiB)\n\u003e 4.32ms\ntest/samples/rollup.min.js (429 KiB)\n\u003e 2.16ms\n\nWarm Runs, All Samples (average of 25 runs)\ntest/samples/*.js (3123 KiB)\n\u003e 14.16ms\n```\n\n#### JS Build (asm.js)\n\n```\nModule load time\n\u003e 2ms\nCold Run, All Samples\ntest/samples/*.js (3123 KiB)\n\u003e 34ms\n\nWarm Runs (average of 25 runs)\ntest/samples/angular.js (739 KiB)\n\u003e 3ms\ntest/samples/angular.min.js (188 KiB)\n\u003e 1ms\ntest/samples/d3.js (508 KiB)\n\u003e 3ms\ntest/samples/d3.min.js (274 KiB)\n\u003e 2ms\ntest/samples/magic-string.js (35 KiB)\n\u003e 0ms\ntest/samples/magic-string.min.js (20 KiB)\n\u003e 0ms\ntest/samples/rollup.js (929 KiB)\n\u003e 5ms\ntest/samples/rollup.min.js (429 KiB)\n\u003e 3.04ms\n\nWarm Runs, All Samples (average of 25 runs)\ntest/samples/*.js (3123 KiB)\n\u003e 17.12ms\n```\n\n### Building\n\nThis project uses [Chomp](https://chompbuild.com) for building.\n\nWith Chomp installed, download the WASI SDK 12.0 from https://github.com/WebAssembly/wasi-sdk/releases/tag/wasi-sdk-12.\n\n- [Linux](https://github.com/WebAssembly/wasi-sdk/releases/download/wasi-sdk-12/wasi-sdk-12.0-linux.tar.gz)\n- [Windows (MinGW)](https://github.com/WebAssembly/wasi-sdk/releases/download/wasi-sdk-12/wasi-sdk-12.0-mingw.tar.gz)\n- [macOS](https://github.com/WebAssembly/wasi-sdk/releases/download/wasi-sdk-12/wasi-sdk-12.0-macos.tar.gz)\n\nLocate the WASI-SDK as a sibling folder, or customize the path via the `WASI_PATH` environment variable.\n\nEmscripten emsdk is also assumed to be a sibling folder or via the `EMSDK_PATH` environment variable.\n\nExample setup:\n\n```\ngit clone https://github.com:guybedford/es-module-lexer\ngit clone https://github.com/emscripten-core/emsdk\ncd emsdk\ngit checkout 1.40.1-fastcomp\n./emsdk install 1.40.1-fastcomp\ncd ..\nwget https://github.com/WebAssembly/wasi-sdk/releases/download/wasi-sdk-12/wasi-sdk-12.0-linux.tar.gz\ngunzip wasi-sdk-12.0-linux.tar.gz\ntar -xf wasi-sdk-12.0-linux.tar\nmv wasi-sdk-12.0-linux.tar wasi-sdk-12.0\ncargo install chompbuild\ncd es-module-lexer\nchomp test\n```\n\nFor the `asm.js` build, git clone `emsdk` from  is assumed to be a sibling folder as well.\n\n### License\n\nMIT\n\n[actions-image]: https://github.com/guybedford/es-module-lexer/actions/workflows/build.yml/badge.svg\n[actions-url]: https://github.com/guybedford/es-module-lexer/actions/workflows/build.yml\n","funding_links":[],"categories":["JavaScript","Repository","打包工具","Miscellaneous"],"sub_categories":["AST","非 JavaScript 编译工具"],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fguybedford%2Fes-module-lexer","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fguybedford%2Fes-module-lexer","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fguybedford%2Fes-module-lexer/lists"}