{"id":15669103,"url":"https://github.com/dcdunkan/deno_textmate","last_synced_at":"2025-05-06T20:05:43.761Z","repository":{"id":114588025,"uuid":"533760078","full_name":"dcdunkan/deno_textmate","owner":"dcdunkan","description":"🦕 Deno port of Microsoft/vscode-textmate: A library that helps tokenize text using TextMate grammars.","archived":false,"fork":false,"pushed_at":"2022-09-11T14:01:36.000Z","size":2019,"stargazers_count":5,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-03-31T02:22:15.961Z","etag":null,"topics":["deno","textmate","textmate-grammar","typescript","vscode"],"latest_commit_sha":null,"homepage":"","language":"TypeScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/dcdunkan.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2022-09-07T12:40:16.000Z","updated_at":"2023-02-06T00:33:20.000Z","dependencies_parsed_at":null,"dependency_job_id":"c8b57d1d-dfb4-4519-8620-62750fdd302b","html_url":"https://github.com/dcdunkan/deno_textmate","commit_stats":null,"previous_names":[],"tags_count":2,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dcdunkan%2Fdeno_textmate","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dcdunkan%2Fdeno_textmate/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dcdunkan%2Fdeno_textmate/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dcdunkan%2Fdeno_textmate/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/dcdunkan","download_url":"https://codeload.github.com/dcdunkan/deno_textmate/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":252761140,"owners_count":21800124,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["deno","textmate","textmate-grammar","typescript","vscode"],"created_at":"2024-10-03T14:21:32.637Z","updated_at":"2025-05-06T20:05:43.714Z","avatar_url":"https://github.com/dcdunkan.png","language":"TypeScript","funding_links":[],"categories":[],"sub_categories":[],"readme":"\u003cdiv align=\"center\"\u003e\n\n# TextMate\n\n\u003c/div\u003e\n\nA library that helps tokenize text using TextMate grammars.\n\nDeno port of\n[Microsoft/vscode-textmate](https://github.com/Microsoft/vscode-textmate). See\noriginal license [here](./LICENSE-MICROSOFT).\n\nAn interpreter for grammar files as defined by TextMate. TextMate grammars use\nthe oniguruma dialect (https://github.com/kkos/oniguruma). Supports loading\ngrammar files from JSON or PLIST format. Cross-grammar injections are currently\nnot supported.\n\nYou can import this module from\n\n- https://ghc.deno.dev/dcdunkan/deno_textmate@VERSION/mod.ts\n- or from GitHub raw links.\n\nTests:\n\n```shell\ndeno test -A\n```\n\nFor any other information: https://github.com/Microsoft/vscode-textmate\n\n## Example\n\n```ts\nimport {\n  INITIAL,\n  parseRawGrammar,\n  Registry,\n} from \"https://ghc.deno.dev/dcdunkan/deno_textmate/mod.ts\";\nimport {\n  loadWASM,\n  OnigScanner,\n  OnigString,\n} from \"https://esm.sh/vscode-oniguruma@1.6.2\";\n\n// https://github.com/microsoft/vscode-oniguruma/blob/main/out/onig.wasm\nconst wasmFile = await Deno.readFile(\"./onig.wasm\");\nconst wasmBin = wasmFile.buffer;\nconst vscodeOnigurumaLib = loadWASM(wasmBin).then(() =\u003e {\n  return {\n    createOnigScanner(patterns: string[]) {\n      return new OnigScanner(patterns);\n    },\n    createOnigString(str: string) {\n      return new OnigString(str);\n    },\n  };\n});\n\n// Create a registry that can create a grammar from a scope name.\nconst registry = new Registry({\n  onigLib: vscodeOnigurumaLib,\n  loadGrammar: async (scopeName) =\u003e {\n    if (scopeName === \"source.js\") {\n      // https://github.com/textmate/javascript.tmbundle/blob/master/Syntaxes/JavaScript.plist\n      const grammarFileContent = await Deno.readTextFile(\"./JavaScript.plist\");\n      return parseRawGrammar(grammarFileContent);\n    }\n\n    console.log(`Unknown scope name: ${scopeName}`);\n  },\n});\n\n// Load the JavaScript grammar and any other grammars included by it async.\nconst grammar = await registry.loadGrammar(\"source.js\");\n\nconst text = [\n  `function sayHello(name) {`,\n  `\\treturn \"Hello, \" + name;`,\n  `}`,\n];\nlet ruleStack = INITIAL;\nfor (let i = 0; i \u003c text.length; i++) {\n  const line = text[i];\n  const lineTokens = grammar!.tokenizeLine(line, ruleStack);\n  console.log(`\\nTokenizing line: ${line}`);\n  for (let j = 0; j \u003c lineTokens.tokens.length; j++) {\n    const token = lineTokens.tokens[j];\n    console.log(\n      ` - token from ${token.startIndex} to ${token.endIndex} ` +\n        `(${line.substring(token.startIndex, token.endIndex)}) ` +\n        `with scopes ${token.scopes.join(\", \")}`,\n    );\n  }\n  ruleStack = lineTokens.ruleStack;\n}\n```\n\nOutput:\n\n```\nUnknown scope name: source.js.regexp\n\nTokenizing line: function sayHello(name) {\n - token from 0 to 8 (function) with scopes source.js, meta.function.js, storage.type.function.js\n - token from 8 to 9 ( ) with scopes source.js, meta.function.js\n - token from 9 to 17 (sayHello) with scopes source.js, meta.function.js, entity.name.function.js\n - token from 17 to 18 (() with scopes source.js, meta.function.js, punctuation.definition.parameters.begin.js\n - token from 18 to 22 (name) with scopes source.js, meta.function.js, variable.parameter.function.js\n - token from 22 to 23 ()) with scopes source.js, meta.function.js, punctuation.definition.parameters.end.js\n - token from 23 to 24 ( ) with scopes source.js\n - token from 24 to 25 ({) with scopes source.js, punctuation.section.scope.begin.js\n\nTokenizing line: \treturn \"Hello, \" + name;\n - token from 0 to 1 (\t) with scopes source.js\n - token from 1 to 7 (return) with scopes source.js, keyword.control.js\n - token from 7 to 8 ( ) with scopes source.js\n - token from 8 to 9 (\") with scopes source.js, string.quoted.double.js, punctuation.definition.string.begin.js\n - token from 9 to 16 (Hello, ) with scopes source.js, string.quoted.double.js\n - token from 16 to 17 (\") with scopes source.js, string.quoted.double.js, punctuation.definition.string.end.js\n - token from 17 to 18 ( ) with scopes source.js\n - token from 18 to 19 (+) with scopes source.js, keyword.operator.arithmetic.js\n - token from 19 to 20 ( ) with scopes source.js\n - token from 20 to 24 (name) with scopes source.js, support.constant.dom.js\n - token from 24 to 25 (;) with scopes source.js, punctuation.terminator.statement.js\n\nTokenizing line: }\n - token from 0 to 1 (}) with scopes source.js, punctuation.section.scope.end.js\n```\n\nRun the example using Deno CLI:\n\n```shell\ndeno run --allow-env=VSCODE_TEXTMATE_DEBUG --allow-read --allow-net https://ghc.deno.dev/dcdunkan/deno_textmate/example.ts\n```\n\n---\n\nCredits and copyright of the code goes to Microsoft.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdcdunkan%2Fdeno_textmate","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fdcdunkan%2Fdeno_textmate","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdcdunkan%2Fdeno_textmate/lists"}