{"id":30796975,"url":"https://github.com/ghost-language/toy-lexer","last_synced_at":"2025-10-19T19:21:56.611Z","repository":{"id":119763240,"uuid":"303212106","full_name":"ghost-language/toy-lexer","owner":"ghost-language","description":"Notes and working example of a lexer written in Go.","archived":false,"fork":false,"pushed_at":"2020-10-13T10:35:29.000Z","size":9,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"nightly","last_synced_at":"2023-03-06T23:56:08.449Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/ghost-language.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2020-10-11T21:02:58.000Z","updated_at":"2024-06-19T10:21:35.837Z","dependencies_parsed_at":null,"dependency_job_id":"f1b8e8ec-3094-4953-a4af-157511da983b","html_url":"https://github.com/ghost-language/toy-lexer","commit_stats":null,"previous_names":[],"tags_count":0,"template":null,"template_full_name":null,"purl":"pkg:github/ghost-language/toy-lexer","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ghost-language%2Ftoy-lexer","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ghost-language%2Ftoy-lexer/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ghost-language%2Ftoy-lexer/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ghost-language%2Ftoy-lexer/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/ghost-language","download_url":"https://codeload.github.com/ghost-language/toy-lexer/tar.gz/refs/heads/nightly","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ghost-language%2Ftoy-lexer/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":273790262,"owners_count":25168669,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-09-05T02:00:09.113Z","response_time":402,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2025-09-05T17:09:17.829Z","updated_at":"2025-10-19T19:21:56.514Z","avatar_url":"https://github.com/ghost-language.png","language":"Go","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Toy Lexer\nThese are notes and a working example of a lexer written in Go. This was developed by going through the **Lexing** portion of [Writing An Interpreter In Go](https://interpreterbook.com/).\n\n**This does not contain a working programming language.** This repository only contains an example toy lexical analyzer used as the first step in creating a programming language. **A lexer takes source code as input and outputs the tokens that represent it.** In our toy lexer, you will find the following subset tokens:\n\n| Token | Description |\n|-------|-------------|\n| `ILLEGAL` | Denotes a value or set of characters that the lexer does not recognize. |\n| `EOF` | Denotes the **end of file**. Every program will end with an `EOF` token. |\n| `IDENT` | Denotes a user-defined **identifier**. |\n| `INT` | Denotes an **integer**. |\n\n## Notes\n- A **lexer** takes source code as input and output the tokens that represent it. It is also known as **lexical analysis**, **lexing**, or **tokenization**.\n  - Lexing is the first step in transforming text written by humans to a computer-readable format that is easy to work with.\n    ```\n    Source code:\n    let x = 5 + 5;\n\n    Lexical output:\n    [\n        LET,\n        IDENTIFIER(\"x\"),\n        EQUAL_SIGN,\n        INTEGER(5),\n        PLUS_SIGN,\n        INTEGER(5),\n        SEMICOLON\n    ]\n    ```\n- **Tokens** are small, categorizable data structures that are fed to a **parser**, which handles the second transformation step, turning our tokens into an **Abstract Syntax Tree** (AST).\n  - A token is a string with an assigned and identifiable meaning. It is structured as a pair consisting of a token **name** and an optional token **value**.\n  - The token name is a category of lexical unit. Common token names are\n    - **identifier:** names the programmar chooses\n    - **keyword:** names already in the programming language\n    - **separator:** puncuation characters and paired-delimiters\n    - **operator:** symbols that operate on arguments and produce results\n    - **literal:** numeric, logical, textual, reference literals\n    - **comment:** line, block\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fghost-language%2Ftoy-lexer","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fghost-language%2Ftoy-lexer","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fghost-language%2Ftoy-lexer/lists"}