{"id":13528097,"url":"https://github.com/begin/parsers-compilers","last_synced_at":"2025-04-01T11:30:58.463Z","repository":{"id":87813933,"uuid":"68046495","full_name":"begin/parsers-compilers","owner":"begin","description":"Lexers, tokenizers, parsers, compilers, renderers, stringifiers... What's the difference, and how do they work? ","archived":false,"fork":false,"pushed_at":"2017-04-26T20:11:12.000Z","size":12,"stargazers_count":22,"open_issues_count":0,"forks_count":1,"subscribers_count":4,"default_branch":"master","last_synced_at":"2025-03-23T21:37:10.326Z","etag":null,"topics":["ast","compiler","guide","lexer","node","parse","parsers-compilers","syntax-tree","token","token-stream","tokenize"],"latest_commit_sha":null,"homepage":null,"language":null,"has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"cc-by-4.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/begin.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null}},"created_at":"2016-09-12T20:49:33.000Z","updated_at":"2023-09-08T17:14:44.000Z","dependencies_parsed_at":"2024-01-13T22:33:13.955Z","dependency_job_id":null,"html_url":"https://github.com/begin/parsers-compilers","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/begin%2Fparsers-compilers","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/begin%2Fparsers-compilers/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/begin%2Fparsers-compilers/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/begin%2Fparsers-compilers/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/begin","download_url":"https://codeload.github.com/begin/parsers-compilers/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":246631676,"owners_count":20808732,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["ast","compiler","guide","lexer","node","parse","parsers-compilers","syntax-tree","token","token-stream","tokenize"],"created_at":"2024-08-01T06:02:12.767Z","updated_at":"2025-04-01T11:30:58.185Z","avatar_url":"https://github.com/begin.png","language":null,"readme":"# parsers-compilers\n\n\u003e Lexers, tokenizers, parsers, compilers, renderers, stringifiers... What's the difference, and how do they work? \n\n- [lexer](#lexer)\n- [parser](#parser)\n- [compiler](#compiler)\n- [renderer](#renderer)\n- [related](#related)\n\n## Lexer\n\n\u003e Sifts, or \"tokenizes\" the characters in a string to create an array of objects, referred to as a \"token stream\"\n\n**Returns**: Token stream.\n\n**Concepts**\n\n- token stream\n- token\n- lexical scoping\n- lexical context\n\n**Description**\n\nA token stream is an array of \"tokens\", where each \"token\" is an object that contain details about a specific substring that was \"captured\", such as column and row, or line number and character position.\n\n**Example token**\n\n```js\n{\n  type: 'text',\n  value: 'abc',\n  position: {\n    start: {\n      column: 1 \n      line: 1\n    }, \n    end: {\n      column: 3, \n      line: 1\n    }\n  }\n}\n```\n\nA token should also (and only) attempt to describe _basic lexical context_, such as character \"type\", which might be something like `text`, `number`, `escaped`, `delimiter`, or something similar.\n\nLexers should not attempt to describe dynamic scope, like where a bracketed section begins or ends, this kind of thing is left to the parser and is better represented by an Abstract Syntax Tree (AST).\n\n**Example token stream**\n\nA JavaScript token stream for the string `abc{foo}xyz` might look something like this:\n\n```js\n[\n  {\n    type: 'text',\n    value: 'abc',\n    position: {start: {column: 1 line: 1}, end: {column: 3, line: 1}}\n  },\n  {\n    type: 'left-brace',\n    value: '{',\n    position: {start: {column: 4 line: 1}, end: {column: 4, line: 1}}\n  },\n  {\n    type: 'text',\n    value: 'foo',\n    position: {start: {column: 5 line: 1}, end: {column: 7, line: 1}}\n  },\n  {\n    type: 'right-brace',\n    value: '}',\n    position: {start: {column: 8 line: 1}, end: {column: 8, line: 1}}\n  },\n  {\n    type: 'text',\n    value: 'xyz',\n    position: {start: {column: 9 line: 1}, end: {column: 11, line: 1}}\n  }\n]\n```\n\n## Parser\n\n\u003e Parses a stream of tokens into an Abstract Syntax Tree (AST)\n\n**Returns**: AST object\n\n**Concepts**\n\n- Abstract Syntax Tree (AST)\n- nodes\n- node\n- dynamic scoping\n- dynamic context\n\n**Description**\n\nWhereas a token stream is a \"flat\" array, the _Abstract Ayntax Tree_ generated by a parser gives the tokens a dynamic, or global, context. \n\nThus, an AST is represented as an object, versus an array.\n\n**Example**\n\nA JavaScript AST for the string `abc{foo}xyz` might look something like this:\n\n```js\n{\n  type: 'root',\n  nodes: [\n    {\n      type: 'text',\n      value: 'abc',\n      position: {start: {column: 1 line: 1}, end: {column: 3, line: 1}}\n    },\n    {\n      type: 'brace',\n      nodes: [\n        {\n          type: 'left-brace',\n          value: '{',\n          position: {start: {column: 4 line: 1}, end: {column: 4, line: 1}}\n        },\n        {\n          type: 'text',\n          value: 'foo',\n          position: {start: {column: 5 line: 1}, end: {column: 7, line: 1}}\n        },\n        {\n          type: 'right-brace',\n          value: '}',\n          position: {start: {column: 8 line: 1}, end: {column: 8, line: 1}}\n        }\n      ]\n    },\n    {\n      type: 'text',\n      value: 'xyz',\n      position: {start: {column: 9 line: 1}, end: {column: 11, line: 1}}\n    }\n  ]\n}\n```\n\n## Compiler\n\n\u003e Creates a function by converting an AST into a string of function statements and wrapping it with a boilerplate function body that defines the arguments the function can take. This generated function is then cached for re-use before being returned.\n\n**Returns**: Function\n\n**Concepts**\n\n- function body\n- function statements\n- caching\n\n**Notes**\n\nThe goal of a compiler is to create a cached function that can be invoked one or more times, on-demand, with the same or different arguments. The arguments passed to a compiled function are referred to as \"context\".\n\n## Renderer\n\n\u003e Invokes the function returned from a compiler with a given \"context\", producing a string where any placeholders or variables that may have been defined are replaced with actual values.\n\n**Returns**: String\n\n**Concepts**\n\n- context\n- variables\n\n## Related\n\n- [the-super-tiny-compiler](https://github.com/thejameskyle/the-super-tiny-compiler)\n- **stringify**: typically refers to converting an object to a string representation of the object. Example: `{foo: 'bar'}` would convert to the string `'{\"foo\": \"bar\"}'`.\n- **assembler**: todo\n- **interpreter**: todo\n- **translater**: todo\n","funding_links":[],"categories":["Others"],"sub_categories":[],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fbegin%2Fparsers-compilers","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fbegin%2Fparsers-compilers","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fbegin%2Fparsers-compilers/lists"}