{"id":15064349,"url":"https://github.com/specy/tokeko","last_synced_at":"2025-04-10T11:51:46.290Z","repository":{"id":257025342,"uuid":"857096992","full_name":"Specy/tokeko","owner":"Specy","description":"LR(1) and LALR parsing tools app to learn compilers and language theory","archived":false,"fork":false,"pushed_at":"2024-11-17T13:02:03.000Z","size":2098,"stargazers_count":21,"open_issues_count":0,"forks_count":1,"subscribers_count":2,"default_branch":"main","last_synced_at":"2025-03-24T10:45:35.664Z","etag":null,"topics":["lalr","lr1","parsers","rust","svelte","sveltekit","typescript","wasm"],"latest_commit_sha":null,"homepage":"https://tokeko.specy.app/","language":"Svelte","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/Specy.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-09-13T19:50:24.000Z","updated_at":"2025-03-04T02:43:58.000Z","dependencies_parsed_at":"2024-09-14T12:04:38.490Z","dependency_job_id":"62d29de5-4bb8-43e3-ae85-188a68fca229","html_url":"https://github.com/Specy/tokeko","commit_stats":{"total_commits":57,"total_committers":1,"mean_commits":57.0,"dds":0.0,"last_synced_commit":"a202636710d43635baefc61b237ba7b6363270f6"},"previous_names":["specy/parsers"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Specy%2Ftokeko","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Specy%2Ftokeko/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Specy%2Ftokeko/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Specy%2Ftokeko/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/Specy","download_url":"https://codeload.github.com/Specy/tokeko/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248215191,"owners_count":21066622,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["lalr","lr1","parsers","rust","svelte","sveltekit","typescript","wasm"],"created_at":"2024-09-25T00:15:57.189Z","updated_at":"2025-04-10T11:51:46.267Z","avatar_url":"https://github.com/Specy.png","language":"Svelte","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Tokeko\n**A website to help learn parsers**\n\n[Tokeko](https://tokeko.specy.app) is an educational platform designed to help users understand and learn about different types of parsers, grammar structures, and parsing techniques. With interactive features, you can explore complex parsing concepts in a visual and intuitive way.\n\n[try a very simple calculator here!](https://tokeko.specy.app/projects/share?project=N4IgbgpgTgzglgewHYgFwEYA0I4BM0gwAWAhlBCNkiQLYWogDCJANgMYCuLJALglJRC4IMNlDgAHHohQNBYiLwi4AgjzToA7ACYAbLoAM6ACwBWA9oCcAZmwcJuJavUYd+gBwHru7+9PY2ZB4IJBcQdAMAam0DAFprA3kEYQIAHShUpECkGB4AAnIYLnyAXjyABRUAJQBlAFEAClTwqJjY4wNmgEp0zLgAMzyGgEJC4oA6BABrLryeIigEAHcCkQmwVl6s5Fy58gg8srGWHnGNli2t7JgEFghxlgQAcwaIc4aeffHPiAgunoySC2-Q4WWkyDyb1YH32qDyABUiHAYPD9l04UgODQAEbQPLALZ5Il5AZDH73HgATwkBxKdLyAHJ4dAaHBqCwGbMCYDibySYMYb8zqwOBTpiFvtSDsN6QyqhAnhAAB6cuYLZZ5JAQFZ1KCLKANBkAVS1SppbGCuDmLLZrDmUs5AG5CXyieQeBwoEg8hIyDAIABJUKC+7nUXjGAsOBsP7OnnEgC+l3jRNJIclNMOsoAcshmVBWezVdzMq6idd8sA8jBKTjbpgfbxgl68gnDnshWGIHHMi7XWma3WWFmygy6sW+2WiQB6ad5AODebQA5kA7WPJsJEsXDkJANpYHUiQSFwJdQPJ1PKRBF5fgXvKxBGTqdp308ZtIB4hJ7zEd5axcs+U7EhWeQANp1A2CASA28IALrtm+H5AcBG47Pkdz9KUkLvHUXQ9t6qF8qB4hPEQ2FQiwDTwvhKHAaB0HQLwd5lL6sAQAA8jSUDMQa0EAr2KZEamAqMTxfDnnSo6RBOQnCby7qet6mH5NepHkQR8mJpCLD+vyQxibxf4MrEsmllpfKKS2KkPgUcBkTwmkWUmclli55moVZ3qUQ0SHQEgYEGHB+F5LOT6uUSbYQLpBwDA0g7Yrcf7NOU3QloRnkQB6LY+X5XqBcFdEJtFekDrWiXDlJjLwmZGXAV5OHQnlAVBbREWtpO7lIF1wKghaMiNuxXFMRJIZwoiyKor8XTpbycXkhmtL0s0+aFpsIB5AAZJtHahiKYpTBKVKZlVzSMOhJChGldENQtXbfOKn5dp1k7zIsKxajqer8A0AAGJpTEgyzeoZEl5AAJMA5IJr9AndYITw8TQNBkAQ5QPgAfBevZIJesRY5eDIyU+uOY-eJkMiTeNY-COPwmTACkmI0DjTNYmT06BbElhwZE06CGx-pQPCUoEAAMioYtVIIh0QBIw2yKAiO0CjAioP0rD+tg-RwLAPAqEguAAGK3I8SxoBrMXYCQHB8CjfCyJbWsgILbJPPCJDYncFuaxA2CCxAqIkDGPsxQm2DQeCORoKAgQsFiSBG7ruQG8bpvLKHztAyoEgIDAQblH6gc8SHqCfKKVAIDnedBiotsIPbyBoOXfsgNnuf50ghewG7Hte-QLfYMQyx13bzFICoMAAOI8RIRCZ63NDJPQICBCkCYJkAA)\n\n![Tokeko](./static/images/tokeko-editor-wide.webp)\n![Tokeko](./static/images/tokeko-result-wide.webp)\n\n## Features\n\n- [x] **LR(1) parser** — Supports LR(1) grammar parsing\n- [x] **LALR parser** — Handles LALR parsing\n- [ ] **LR(0) parser** — Coming soon\n- [ ] **LL parser** — Planned for future updates\n- [x] **FIRST table** — Displays the FIRST set for grammar symbols\n- [x] **FOLLOW table** — Visualizes the FOLLOW set for grammar symbols\n- [x] **Partial tokenization** — Basic tokenization support\n- [x] **GOTO/Action table** — Comprehensive table to guide the parsing process\n- [x] **Automaton table** — Shows the automaton states and transitions\n- [x] **Automaton graph** — Graphical representation of the automaton\n- [x] **Parsing steps** — Visual breakdown of the parsing process\n- [ ] **Custom syntax highlighting** — Planned feature that will use random colors for non-terminals, making grammar and parse trees easier to understand\n- [x] **Parse tree (AST) viewer** — Visualize the abstract syntax tree (AST)\n- [x] **Typescript runner** — Run typescript code implementing your parser directly in the app\n## Writing a Grammar\n\nTokeko allows you to define the grammar of the language you wish to recognize by specifying terminals, non-terminals, and regex patterns.\n\n### Example Grammar\n\n```\nP -\u003e E\n\nE -\u003e E '+' T\nE -\u003e E '-' T\nE -\u003e T\nT -\u003e %num\n\n%num -\u003e /[0-9]+/\n```\n\n- Terminals are enclosed in single quotes (`'...'`).\n- Regex tokens are prefixed with a `%` (e.g., `%num`).\n- Non-terminals are written without any special symbols.\n\n### Example of a Recognized Expression\n\nGiven the grammar above, the following expression can be parsed:\n```plaintext\n10+20-40\n```\n\n## How It Works\n\nWith Tokeko, you can:\n1. **Explore Language Properties**: View various properties of the language, such as FIRST and FOLLOW sets, automaton tables, and more.\n2. **Parse a String**: Enter a string to see if it is recognized by the defined grammar. If it is, you’ll be able to view the corresponding parse tree.\n\n### Parsing Steps\nAfter compiling a language or parsing a string, Tokeko breaks down the process step by step. You can click on each step to see the details and gain deeper insights into how the parsing works.\n\n### More info\nFor more info about the parser itself and how to use the grammar, please [visit the dotlr](https://github.com/umut-sahin/dotlr?tab=readme-ov-file#how-does-it-work) library, the rust library that this website uses\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fspecy%2Ftokeko","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fspecy%2Ftokeko","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fspecy%2Ftokeko/lists"}