{"id":21195601,"url":"https://github.com/openpeeps/toktok","last_synced_at":"2026-04-02T02:10:29.470Z","repository":{"id":52636775,"uuid":"460743521","full_name":"openpeeps/toktok","owner":"openpeeps","description":"Generic tokenizer written in Nim language 👑  Powered by std/lexbase and Nim's Macros","archived":false,"fork":false,"pushed_at":"2024-04-18T18:31:13.000Z","size":516,"stargazers_count":31,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2026-01-02T23:13:55.680Z","etag":null,"topics":["awesome-nim","generic-library","hacktoberfest","lex","lexer","lexer-generator","lexical","nim","nim-lang","nim-language","parser","programming-language","tokenizer","tokens"],"latest_commit_sha":null,"homepage":"https://openpeeps.github.io/toktok/","language":"Nim","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/openpeeps.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2022-02-18T06:56:04.000Z","updated_at":"2024-09-12T13:20:32.000Z","dependencies_parsed_at":"2024-04-18T19:53:41.035Z","dependency_job_id":null,"html_url":"https://github.com/openpeeps/toktok","commit_stats":null,"previous_names":["openpeeps/toktok"],"tags_count":3,"template":false,"template_full_name":"openpeeps/pistachio","purl":"pkg:github/openpeeps/toktok","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/openpeeps%2Ftoktok","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/openpeeps%2Ftoktok/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/openpeeps%2Ftoktok/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/openpeeps%2Ftoktok/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/openpeeps","download_url":"https://codeload.github.com/openpeeps/toktok/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/openpeeps%2Ftoktok/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":31294416,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-04-02T01:43:37.129Z","status":"online","status_checked_at":"2026-04-02T02:00:08.535Z","response_time":89,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["awesome-nim","generic-library","hacktoberfest","lex","lexer","lexer-generator","lexical","nim","nim-lang","nim-language","parser","programming-language","tokenizer","tokens"],"created_at":"2024-11-20T19:29:05.651Z","updated_at":"2026-04-02T02:10:29.439Z","avatar_url":"https://github.com/openpeeps.png","language":"Nim","funding_links":["https://www.paypal.com/donate/?hosted_button_id=RJK3ZTDWPL55C"],"categories":[],"sub_categories":[],"readme":"\u003cp align=\"center\"\u003e\n  \u003cimg src=\"https://raw.githubusercontent.com/openpeep/toktok/main/.github/logo.png\" width=\"140px\"\u003e\u003cbr\u003e\n  Generic tokenizer written in Nim language, powered by Nim's Macros 👑\n\u003c/p\u003e\n\n\u003cp align=\"center\"\u003e\n  \u003ccode\u003enimble install toktok\u003c/code\u003e\n\u003c/p\u003e\n\n\u003cp align=\"center\"\u003e\n  \u003ca href=\"https://openpeeps.github.io/toktok\"\u003eAPI reference\u003c/a\u003e \u003cbr\u003e\u003cbr\u003e\n  \u003cimg src=\"https://github.com/openpeeps/toktok/workflows/test/badge.svg\" alt=\"Github Actions\"\u003e  \u003cimg src=\"https://github.com/openpeeps/toktok/workflows/docs/badge.svg\" alt=\"Github Actions\"\u003e\n\u003c/p\u003e\n\n\n## 😍 Key Features\n- ✨ Powered by Nim's Macros\n- 🪄 Based on `std/lexbase` / Zero Regular Expression\n- Compile-time generation using macro-based **TokenKind** `enum`, `lexbase`\n- Runtime generation using **TokenKind** `tables`, `lexbase`\n- Open Source | `MIT`\n\n\u003e [!NOTE]\n\u003e This is a generic Lexer, based on std/ `streams`, `lexbase` and `macros`. It is meant to be used by higher level parsers for writing any kind of tools or programs.\n\n\u003e [!NOTE]\n\u003e Compile with `-d:toktokdebug` to inspect the generated code.\n\n## Quick Example\n\n```nim\n# Register your custom handlers\nhandlers:\n  proc handleImport(lex: var Lexer, kind: TokenKind) =\n    # tokenize `import x, y, z`\n    lex.kind = kind\n\n# Register your tokens\nregisterTokens defaultSettings:\n  `const` = \"const\"\n  `echo` = \"hello\"\n  asgn = '=':   # `=` is tkAsgn\n    eq = '='    # `==` is tkEQ\n  excl = '!':\n    ne = '='\n  at = '@':\n    import = tokenizer(handleImport, \"import\") \n\n\n# Tokenizing...\nvar\n  tok = lexer.init(sample)\n  prev: TokenTuple\n  curr: TokenTuple = tok.getToken\n  next: TokenTuple = tok.getToken \n\nproc walk(tok: var Lexer) =\n  prev = curr\n  curr = next\n  next = tok.getToken\n\nwhile likely(curr.kind != tkEOF):\n  if tok.hasError: break\n  echo curr # use `getToken` consumer to get token by token\n  walk tok\n```\n\n## TODO\n- [ ] Runtime Token generation using tables/critbits\n\n### ❤ Contributions \u0026 Support\n- 🐛 Found a bug? [Create a new Issue](https://github.com/openpeeps/toktok/issues)\n- 👋 Wanna help? [Fork it!](https://github.com/openpeeps/toktok/fork)\n- 😎 [Get €20 in cloud credits from Hetzner](https://hetzner.cloud/?ref=Hm0mYGM9NxZ4)\n- 🥰 [Donate to OpenPeeps via PayPal address](https://www.paypal.com/donate/?hosted_button_id=RJK3ZTDWPL55C)\n\n### 🎩 License\n`MIT` license. [Made by Humans from OpenPeeps](https://github.com/openpeeps).\u003cbr\u003e\nCopyright \u0026copy; 2023 OpenPeeps \u0026 Contributors \u0026mdash; All rights reserved.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fopenpeeps%2Ftoktok","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fopenpeeps%2Ftoktok","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fopenpeeps%2Ftoktok/lists"}