{"id":13995836,"url":"https://github.com/vsce-toolroom/vscode-textmate-languageservice","last_synced_at":"2025-04-13T05:38:36.349Z","repository":{"id":154968851,"uuid":"631369151","full_name":"vsce-toolroom/vscode-textmate-languageservice","owner":"vsce-toolroom","description":"Language APIs and support features from Textmate tokenization in Visual Studio Code.","archived":false,"fork":false,"pushed_at":"2025-03-11T19:27:44.000Z","size":1327,"stargazers_count":17,"open_issues_count":4,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-03-26T22:12:39.633Z","etag":null,"topics":["grammar","language-features","syntax","textmate","tokenization","tokenizer","visual-studio-code","vscode","vscode-extension"],"latest_commit_sha":null,"homepage":"","language":"TypeScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"other","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/vsce-toolroom.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE.md","code_of_conduct":"CODE_OF_CONDUCT.md","threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":"SECURITY.md","support":null,"governance":null,"roadmap":null,"authors":null,"dei":null}},"created_at":"2023-04-22T20:02:52.000Z","updated_at":"2025-02-01T19:00:05.000Z","dependencies_parsed_at":"2023-10-14T15:29:48.889Z","dependency_job_id":"eddc4300-7497-4662-8bcb-19aac1d8d1d5","html_url":"https://github.com/vsce-toolroom/vscode-textmate-languageservice","commit_stats":null,"previous_names":["vsce-toolroom/vscode-textmate-languageservice"],"tags_count":15,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/vsce-toolroom%2Fvscode-textmate-languageservice","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/vsce-toolroom%2Fvscode-textmate-languageservice/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/vsce-toolroom%2Fvscode-textmate-languageservice/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/vsce-toolroom%2Fvscode-textmate-languageservice/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/vsce-toolroom","download_url":"https://codeload.github.com/vsce-toolroom/vscode-textmate-languageservice/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248670505,"owners_count":21142897,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["grammar","language-features","syntax","textmate","tokenization","tokenizer","visual-studio-code","vscode","vscode-extension"],"created_at":"2024-08-09T14:03:36.544Z","updated_at":"2025-04-13T05:38:36.276Z","avatar_url":"https://github.com/vsce-toolroom.png","language":"TypeScript","funding_links":[],"categories":["TypeScript"],"sub_categories":[],"readme":"# `vscode-textmate-languageservice`\n\n\u003cp align=\"center\"\u003e\u003cimg src=\"https://raw.githubusercontent.com/vsce-toolroom/vscode-textmate-languageservice/v4.0.0/assets/logo.png\" width=\"205px\" /\u003e\u003c/p\u003e\n\n\u003e 🎉 **This package has been adopted by the `vsce-toolroom` GitHub collective.**\n\n\u003e *This package is in LTS mode \u0026 the Textmate technology is superseded by the [`tree-sitter` symbolic-expression parser technology][tree-sitter-parser-guide], as used in `vscode-anycode`.*\n\nLanguage service providers \u0026 APIs driven entirely by your Textmate grammar and one configuration file.\n\n\u003cp align=\"center\"\u003e\u003cimg src=\"https://github.com/vsce-toolroom/vscode-textmate-languageservice/raw/v4.0.0/assets/demo-outline.png\" height=\"320\"/\u003e\u003c/p\u003e\n\nTo use the API methods and tokenization / outline services, you only need a Textmate grammar. This can be from your extension or one of VS Code's built-in languages.\n\nIn order to properly generate full-blown language providers from this module, the Textmate grammar must also include the following features:\n\n- meta declaration scopes for block level declarations\n- variable assignment scopes differentiated between `multiple` and `single`\n- granular keyword control tokens with `begin` and `end` scopes\n\n## Installation\n\n```\nnpm install vscode-textmate-languageservice\n```\n\nBrowser support:\n\n- This package supports Webpack and ESBuild.\n- If you use a bundler, you need to set `crypto` as a external (`commonjs crypto` one in webpack).\n  This allows the library to avoid polyfilling the `node:crypto` module.\n\n**Advisory:**\n\n\u003e This package is stable with browser compatibility (`1.1.0`). But I recommend you watch out for `tree-sitter` [native integration][github-vscode-pull-treesitter] into `vscode` ([issue][github-vscode-issue-treesitter]). Maintainable \u0026 with faster retokenization, **it is a Holy Grail** ...\n\n\u003e Whereas this package depends on a [well-written Textmate grammar][macromates-scope-selector-spec] and is a band aid of sorts.\n\n\u003e If there is [native `vscode` support for the language][vscode-known-language-ids], find a Tree-sitter syntax online then suggest it in an [Anycode issue][github-vscode-anycode-issues].\n\u003e Otherwise, please open an issue on the [community-maintained Treesitter syntax highlighter extension][github-epeshkov-syntax-highlighter] and someone might deal with it.\n\n## Setup\n\n- [Language contribution][vscode-language-contributions] and [grammar contribution][vscode-grammar-contributions] defined via `contributes` in the [extension manifest][vscode-extension-manifest] (or `textmate-languageservice-contributes`).\n- Your grammar is bundled in the extension source code and is consumable by `vscode-textmate` (which can load PList XML, JSON or YAML grammars).\n- A configuration file is available in the extension, defaulting to `./textmate-configuration.json`. You can also use `textmate-languageservices` property of `package.json` to map language ID to relative path.\n\nExample language extension manifest - `./package.json`:\n\n```json\n{\n\t\"name\": \"lua\",\n\t\"displayName\": \"Textmate language service for Lua\",\n\t\"description\": \"Lua enhanced support for Visual Studio Code\",\n\t\"version\": \"0.0.1\",\n\t\"publisher\": \"\",\n\t\"license\": \"\",\n\t\"engines\": {\n\t\t\"vscode\": \"^1.51.1\"\n\t},\n\t\"categories\": [\n\t\t\"Programming Languages\"\n\t],\n\t\"contributes\": {\n\t\t\"languages\": [{\n\t\t\t\"id\": \"lua\",\n\t\t\t\"aliases\": [\"Lua\"],\n\t\t\t\"extensions\": [\".lua\", \".moon\", \".luau\"],\n\t\t\t\"configuration\": \"./language-configuration.json\"\n\t\t}],\n\t\t\"grammars\": [{\n\t\t\t\"language\": \"lua\",\n\t\t\t\"scopeName\": \"source.lua\",\n\t\t\t\"path\": \"./syntaxes/Lua.tmLanguage.json\"\n\t\t}]\n\t}\n}\n```\n\n## Configuration\n\nCreate a JSON file named `textmate-configuration.json` in the extension directory. The file accepts comments and trailing commas.\n\nIf you only want to use the document and/or tokenization services, you can skip creating the file!\n\nTextmate configuration fields:\n\n- **`assignment`** - optional (`object`)\u003cbr/\u003e\n  Collection of Textmate scope selectors for variable assignment scopes when including variable symbols:\u003cbr/\u003e\n  **Properties:**\n  - `separator`: Token to separate multiple assignments (`string`)\n  - `single`: Token to match single variable assignment. (`string`)\n  - `multiple`: Token to match multiple variable assignment. (`string`)\n- **`declarations`** - optional (`array`)\u003cbr/\u003e\n  List of Textmate scope selectors for declaration token scopes.\n- **`dedentation`** - optional (`array`)\u003cbr/\u003e\n  List of Textmate tokens for dedented code block declarations (e.g. `ELSE`, `ELSEIF`).\u003cbr/\u003e\n  Tokens still need to be listed in `indentation` with the decrementing value `-1`.\n- **`exclude`** (`string`)\n  VS Code glob pattern for files to exclude from workspace symbol search.\n- **`indentation`** - optional (`object`)\u003cbr/\u003e\n  Indentation level offset for Textmate token types (used to implement folding).\n- **`punctuation`** - optional (`object`)\u003cbr/\u003e\n  Collection of punctuation tokens with a significant effect on syntax providers.\n  **Properties:**\n  - `continuation`: Token scope selector for line continuation (to use in region matching). (`string`)\n- **`markers`** - optional (`object`)\u003cbr/\u003e\n  Stringified regular expression patterns for folding region comments.\n  - `start`: Escaped regular expression for start region marker. (`string`)\n  - `end`: Escaped regular expression for end region marker. (`string`)\n  **Properties:**\n- **`symbols`** - optional (`object`)\u003cbr/\u003e\n  Map of document symbol tokens to their symbol kind ([`vscode.SymbolKind`][vscode-api-symbolkind] value).\n\n### Configuration examples\n\nTemplate for `textmate-configuration.json` file:\n\n```json\n{\n  \"assignment\": {\n    \"single\": \"\",\n    \"multiple\": \"\",\n    \"separator\": \"\"\n  },\n  \"declarations\": [],\n  \"dedentation\": [\n    \"keyword.control.elseif.custom\",\n    \"keyword.control.else.custom\"\n  ],\n  \"exclude\": \"{.modules,.includes}/**\",\n  \"indentation\": {\n    \"punctuation.definition.comment.begin.custom\": 1,\n    \"punctuation.definition.comment.end.custom\": -1,\n    \"keyword.control.begin.custom\": 1,\n    \"keyword.control.end.custom\": -1\n  },\n  \"punctuation\": {\n    \"continuation\": \"punctuation.separator.continuation.line.custom\"\n  },\n  \"markers\": {\n    \"start\": \"^\\\\s*#?region\\\\b\",\n    \"end\": \"^\\\\s*#?end\\\\s?region\\\\b\"\n  },\n  \"symbols\": {\n    \"keyword.control.custom\": 2,\n    \"entity.name.function.custom\": 11\n  }\n}\n```\n\nAn example configuration file that targets Lua:\n\n```json\n{\n  \"assignment\": {\n    \"single\": \"meta.assignment.variable.single.lua\",\n    \"multiple\": \"meta.assignment.variable.group.lua\",\n    \"separator\": \"punctuation.separator.comma.lua\"\n  },\n  \"declarations\": [\n    \"meta.declaration.lua entity.name\",\n    \"meta.assignment.definition.lua entity.name\"\n  ],\n  \"dedentation\": [\n    \"keyword.control.elseif.lua\",\n    \"keyword.control.else.lua\"\n  ],\n  \"exclude\": \"{.luarocks,lua_modules}/**\",\n  \"indentation\": {\n    \"punctuation.definition.comment.begin.lua\": 1,\n    \"punctuation.definition.comment.end.lua\": -1,\n    \"keyword.control.begin.lua\": 1,\n    \"keyword.control.end.lua\": -1\n  },\n  \"markers\": {\n    \"start\": \"^\\\\s*#?region\\\\b\",\n    \"end\": \"^\\\\s*#?end\\\\s?region\\\\b\"\n  },\n  \"symbols\": {\n    \"keyword.control.lua\": 2,\n    \"entity.name.function.lua\": 11\n  }\n}\n```\n\n## Usage\n\n### `TextmateLanguageService`\n\nThe package exports a default class named `TextmateLanguageService`.\n\n- **Parameter:** *`languageId`* - Language ID of grammar contribution in VS Code (`string`).\n- **Parameter:** *`context?`* - Extension context from `activate` entrypoint export (`vscode.ExtensionContext`).\n\nThe library defaults to core behaviour when figuring out which scope name to use - last matching grammar or language wins. If the `context` parameter is supplied, the extension will first search contributions from the extension itself.\n\n### Language extension\n\nExtension code sample - `./src/extension.ts`:\n\n```typescript\nimport TextmateLanguageService from 'vscode-textmate-languageservice';\n\nexport async function activate(context: vscode.ExtensionContext) {\n\tconst selector: vscode.DocumentSelector = 'lua';\n\tconst textmateService = new TextmateLanguageService(selector, context);\n\n\tconst foldingRangeProvider = await textmateService.createFoldingRangeProvider();\n\tconst documentSymbolProvider = await textmateService.createDocumentSymbolProvider();\n\tconst workspaceSymbolProvider = await textmateService.createWorkspaceSymbolProvider();\n\tconst definitionProvider = await textmateService.createDefinitionProvider();\n\n\tcontext.subscriptions.push(vscode.languages.registerDocumentSymbolProvider(selector, documentSymbolProvider));\n\tcontext.subscriptions.push(vscode.languages.registerFoldingRangeProvider(selector, foldingRangeProvider));\n\tcontext.subscriptions.push(vscode.languages.registerWorkspaceSymbolProvider(workspaceSymbolProvider));\n\tcontext.subscriptions.push(vscode.languages.registerDefinitionProvider(selector, peekDefinitionProvider));\n};\n```\n\n### Tokenization\n\nExtension code sample - `./src/extension.ts`:\n\n```typescript\nimport TextmateLanguageService from 'vscode-textmate-languageservice';\n\nexport async function activate(context: vscode.ExtensionContext) {\n    const selector: vscode.DocumentSelector = 'custom';\n    const textmateService = new TextmateLanguageService('custom', context);\n    const textmateTokenService = await textmateService.initTokenService();\n    const textDocument = vscode.window.activeTextEditor!.document;\n    const tokens = textmateTokenService.fetch(textDocument);\n};\n```\n\n**NB:** If you would like to:\n\n- just wire up tokenization or fast document text services to a Textmate grammar,\n- without [(re-)contributing grammar and language configuration to VS Code](#setup),\n- or writing a full [`TextmateLanguageService` provider configuration](#configuration)..\n\nYou can use the custom `\"textmate-languageservice-contributes\"` property in `package.json`:\n\n```json\n{\n\t\"textmate-languageservice-contributes\": {\n\t\t\"languages\": [{\n\t\t\t\"id\": \"typescript\",\n\t\t\t\"aliases\": [\"TypeScript\"],\n\t\t\t\"extensions\": [\".ts\", \".tsx\", \".cts\", \".mts\"]\n\t\t}],\n\t\t\"grammars\": [{\n\t\t\t\"language\": \"typescript\",\n\t\t\t\"scopeName\": \"source.ts\",\n\t\t\t\"path\": \"./syntaxes/TypeScript.tmLanguage.json\"\n\t\t}]\n\t}\n}\n```\n\n### API methods\n\nUsage (example is for getting the token at the current cursor position):\n\n```typescript\nconst { getScopeInformationAtPosition } = TextmateLanguageService.api;\n\nconst editor = vscode.window.activeTextEditor;\nconst document = editor.document;\nconst position = editor.selection.active;\n\nconst token = await getScopeInformationAtPosition(document, position);\n```\n\n#### `getScopeInformationAtPosition`\n\n`getScopeInformationAtPosition(document: vscode.TextDocument, position: vscode.Position): Promise\u003cTextmateToken\u003e`\n\nGet token scope information at a specific position (caret line and character number).\n\n- **Parameter:** *document* - Document to be tokenized (`vscode.TextDocument`).\n- **Parameter:** *position* - Zero-indexed caret position of token in document (`vscode.Position`).\n- **Returns:** Promise resolving to token data for scope selected by caret position (`{Promise\u003cTextmateToken\u003e}`).\n\n#### `getScopeRangeAtPosition`\n\n`getScopeRangeAtPosition(document: vscode.TextDocument, position: vscode.Position): vscode.Range;`\n\nGet matching scope range of the Textmate token intersecting a caret position.\n\n- **Parameter:** *document* - Document to be tokenized (`vscode.TextDocument`).\n- **Parameter:** *position* - Zero-indexed caret position to intersect with (`vscode.Position`).\n- **Returns:** Promise resolving to character and line number of the range (`Promise\u003cvscode.Range\u003e`).\n\n#### `getTokenInformationAtPosition`\n\n`getTokenInformationAtPosition(document: vscode.TextDocument, position: vscode.Position): Promise\u003cvscode.TokenInformation\u003e;`\n\nVS Code compatible performant API for token information at a caret position.\n\n- **Parameter:** *document* - Document to be tokenized (`vscode.TextDocument`).\n- **Parameter:** *position* - Zero-indexed caret position of token in document (`vscode.Position`).\n- **Returns:** Promise resolving to token data compatible with VS Code (`Promise\u003cvscode.TokenInformation\u003e`).\n\n#### `getLanguageConfiguration`\n\n`getLanguageConfiguration(languageId: string): LanguageDefinition;`\n\nGet the language definition point of a language mode identifier.\n\n- **Parameter:** *languageId* - Language ID as shown in brackets in \"Change Language Mode\" panel (`string`).\n- **Returns:** Language contribution as configured in source VS Code extension (`LanguageDefinition`).\n\n#### `getGrammarContribution`\n\n`getGrammarConfiguration(languageId: string): GrammarLanguageDefinition;`\n\nGet the grammar definition point of a language mode identifier.\n\n- **Parameter:** *languageId* - Language identifier, shown in brackets in \"Change Language Mode\" panel (`string`).\n- **Returns:** Grammar contribution as configured in source VS Code extension (`GrammarLanguageDefinition`).\n\n#### `getLanguageContribution`\n\n`getLanguageConfiguration(languageId: string): LanguageDefinition;`\n\nGet the language configuration of a language mode identifier.\n\n- **Parameter:** *languageId* - Language ID as shown in brackets in \"Change Language Mode\" panel (`string`).\n- **Returns:** Language contribution as configured in source VS Code extension (`LanguageDefinition`).\n\n#### `getContributorExtension`\n\n`getContributorExtension(languageId: string): vscode.Extension\u003cunknown\u003e | void;`\n\nGet the VS Code Extension API entry of the extension that contributed a language mode identifier.\n\n- **Parameter:** *languageId* - Language identifier, shown in brackets in \"Change Language Mode\" panel (`string`).\n- **Returns:** Extension API instance that contributed the language - (`vscode.Extension`).\n\n### Use Oniguruma WASM buffer\n\nThis is the `vscode-oniguruma` build of Oniguruma written in C, compiled to WASM format with memory hooks to V8.\n\nThis is not streaming 🙁 but `vscode` libs must bundle WebAssembly deps so as to support web ecosystem.\n\n```typescript\nimport TextmateLanguageService from 'vscode-textmate-languageservice';\nconst onigurumaPromise = TextmateLanguageService.utils.getOniguruma();\n```\n\n\u003c!-- `vscode-textmate-languageservice` --\u003e\n[tree-sitter-parser-guide]: https://tree-sitter.github.io/tree-sitter/using-parsers#pattern-matching-with-queries\n[github-vscode-pull-treesitter]: https://github.com/microsoft/vscode/pull/161479\n[github-vscode-issue-treesitter]: https://github.com/microsoft/vscode/issues/50140\n[macromates-scope-selector-spec]: https://macromates.com/manual/en/language_grammars#naming_conventions\n[vscode-known-language-ids]: https://code.visualstudio.com/docs/languages/identifiers#_known-language-identifiers\n[github-vscode-anycode-issues]: https://github.com/microsoft/vscode-anycode/issues\n[github-epeshkov-syntax-highlighter]: https://github.com/EvgeniyPeshkov/syntax-highlighter\n\u003c!-- Setup --\u003e\n[vscode-extension-manifest]: https://code.visualstudio.com/api/references/extension-manifest\n[vscode-language-contributions]: https://code.visualstudio.com/api/references/contribution-points#contributes.languages\n[vscode-grammar-contributions]: https://code.visualstudio.com/api/references/contribution-points#contributes.grammars\n\u003c!-- Configuration --\u003e\n[vscode-api-symbolkind]: https://code.visualstudio.com/api/references/vscode-api#SymbolKind\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fvsce-toolroom%2Fvscode-textmate-languageservice","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fvsce-toolroom%2Fvscode-textmate-languageservice","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fvsce-toolroom%2Fvscode-textmate-languageservice/lists"}