{"id":19639440,"url":"https://github.com/decaffeinate/coffee-lex","last_synced_at":"2025-04-28T10:31:00.987Z","repository":{"id":3995838,"uuid":"51632968","full_name":"decaffeinate/coffee-lex","owner":"decaffeinate","description":"Stupid lexer for CoffeeScript.","archived":false,"fork":false,"pushed_at":"2023-10-18T04:10:22.000Z","size":47284,"stargazers_count":4,"open_issues_count":2,"forks_count":4,"subscribers_count":3,"default_branch":"main","last_synced_at":"2025-04-20T15:07:25.609Z","etag":null,"topics":["coffeescript","decaffeinate","lex"],"latest_commit_sha":null,"homepage":null,"language":"TypeScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/decaffeinate.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2016-02-13T05:00:58.000Z","updated_at":"2022-06-08T12:05:28.000Z","dependencies_parsed_at":"2024-06-18T18:35:27.791Z","dependency_job_id":"78028c0b-bed1-4ef6-8376-ec73dbd60f62","html_url":"https://github.com/decaffeinate/coffee-lex","commit_stats":{"total_commits":656,"total_committers":10,"mean_commits":65.6,"dds":0.5076219512195121,"last_synced_commit":"85c17e563810fc70ca160e6b268f7ec826739be0"},"previous_names":[],"tags_count":73,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/decaffeinate%2Fcoffee-lex","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/decaffeinate%2Fcoffee-lex/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/decaffeinate%2Fcoffee-lex/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/decaffeinate%2Fcoffee-lex/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/decaffeinate","download_url":"https://codeload.github.com/decaffeinate/coffee-lex/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":251295231,"owners_count":21566421,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["coffeescript","decaffeinate","lex"],"created_at":"2024-11-11T13:01:02.013Z","updated_at":"2025-04-28T10:31:00.980Z","avatar_url":"https://github.com/decaffeinate.png","language":"TypeScript","funding_links":[],"categories":[],"sub_categories":[],"readme":"# coffee-lex [![CI](https://github.com/decaffeinate/coffee-lex/actions/workflows/ci.yml/badge.svg)](https://github.com/decaffeinate/coffee-lex/actions/workflows/ci.yml) [![package version](https://badge.fury.io/js/coffee-lex.svg)](https://badge.fury.io/js/coffee-lex)\n\nStupid lexer for CoffeeScript.\n\n## Install\n\n```bash\n# via yarn\n$ yarn add coffee-lex\n# via npm\n$ npm install coffee-lex\n```\n\n## Usage\n\nThe main `lex` function simply returns a list of tokens:\n\n```js\nimport lex, { SourceType } from 'coffee-lex'\n\nconst source = 'a?(b: c)'\nconst tokens = lex(source)\n\n// Print tokens along with their source.\ntokens.forEach((token) =\u003e\n  console.log(\n    SourceType[token.type],\n    JSON.stringify(source.slice(token.start, token.end)),\n    `${token.start}→${token.end}`\n  )\n)\n// IDENTIFIER \"a\" 0→1\n// EXISTENCE \"?\" 1→2\n// CALL_START \"(\" 2→3\n// IDENTIFIER \"b\" 3→4\n// COLON \":\" 4→5\n// IDENTIFIER \"c\" 6→7\n// CALL_END \")\" 7→8\n```\n\nYou can also get more fine control of what you'd like to lex by using the\n`stream` function:\n\n```js\nimport { stream, SourceType } from 'coffee-lex';\n\nconst source = 'a?(b: c)';\nconst step = stream(source);\nconst location;\n\ndo {\n  location = step();\n  console.log(location.index, SourceType[location.type]);\n} while (location.type !== SourceType.EOF);\n// 0 IDENTIFIER\n// 1 EXISTENCE\n// 2 CALL_START\n// 3 IDENTIFIER\n// 4 COLON\n// 5 SPACE\n// 6 IDENTIFIER\n// 7 CALL_END\n// 8 EOF\n```\n\nThis function not only lets you control how far into the source you'd like to\ngo, it also gives you information about source code that wouldn't become part of\na token, such as spaces.\n\nNote that the input source code should have only UNIX line endings (LF). If you\nwant to process a file with Windows line endings (CRLF), you should convert to\nUNIX line endings first, then use coffee-lex, then convert back if necessary.\n\n## Why?\n\nThe official CoffeeScript lexer does a lot of pre-processing, even with\n`rewrite: false`. That makes it good for building an AST, but bad for\nidentifying parts of the source code that aren't part of the final AST, such as\nthe location of operators. One good example of this is string interpolation. The\nofficial lexer turns it into a series of string tokens separated by (virtual)\n`+` tokens, but they have no reality in the original source code. Here's what\nthe official lexer generates for `\"a#{b}c\"`:\n\n```js\n;[\n  [\n    'STRING_START',\n    '(',\n    { first_line: 0, first_column: 0, last_line: 0, last_column: 0 },\n    (origin: ['STRING', null, [Object]]),\n  ],\n  [\n    'STRING',\n    '\"a\"',\n    { first_line: 0, first_column: 0, last_line: 0, last_column: 1 },\n  ],\n  ['+', '+', { first_line: 0, first_column: 3, last_line: 0, last_column: 3 }],\n  ['(', '(', { first_line: 0, first_column: 3, last_line: 0, last_column: 3 }],\n  [\n    'IDENTIFIER',\n    'b',\n    { first_line: 0, first_column: 4, last_line: 0, last_column: 4 },\n    (variable: true),\n  ],\n  [\n    ')',\n    ')',\n    { first_line: 0, first_column: 5, last_line: 0, last_column: 5 },\n    (origin: ['', 'end of interpolation', [Object]]),\n  ],\n  ['+', '+', { first_line: 0, first_column: 6, last_line: 0, last_column: 6 }],\n  [\n    'STRING',\n    '\"c\"',\n    { first_line: 0, first_column: 6, last_line: 0, last_column: 7 },\n  ],\n  [\n    'STRING_END',\n    ')',\n    { first_line: 0, first_column: 7, last_line: 0, last_column: 7 },\n  ],\n  [\n    'TERMINATOR',\n    '\\n',\n    { first_line: 0, first_column: 8, last_line: 0, last_column: 8 },\n  ],\n]\n```\n\nHere's what coffee-lex generates for the same source:\n\n```js\n[ SourceToken { type: DSTRING_START, start: 0, end: 1 },\n  SourceToken { type: STRING_CONTENT, start: 1, end: 2 },\n  SourceToken { type: INTERPOLATION_START, start: 2, end: 4 },\n  SourceToken { type: IDENTIFIER, start: 4, end: 5 },\n  SourceToken { type: INTERPOLATION_END, start: 5, end: 6 },\n  SourceToken { type: STRING_CONTENT, start: 6, end: 7 },\n  SourceToken { type: DSTRING_END, start: 7, end: 8 } ]\n```\n\n## License\n\nMIT\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdecaffeinate%2Fcoffee-lex","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fdecaffeinate%2Fcoffee-lex","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdecaffeinate%2Fcoffee-lex/lists"}