{"id":24864485,"url":"https://github.com/bsfishy/zig-lexer","last_synced_at":"2025-03-26T18:42:30.195Z","repository":{"id":274564318,"uuid":"923277748","full_name":"BSFishy/zig-lexer","owner":"BSFishy","description":"lexer generator written in zig","archived":false,"fork":false,"pushed_at":"2025-03-19T03:31:45.000Z","size":209,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-03-19T04:20:01.897Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Zig","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"other","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/BSFishy.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2025-01-27T23:54:58.000Z","updated_at":"2025-03-19T03:31:47.000Z","dependencies_parsed_at":"2025-01-28T04:19:39.910Z","dependency_job_id":"2611cf5d-12e7-405d-b35a-9629a9008ade","html_url":"https://github.com/BSFishy/zig-lexer","commit_stats":null,"previous_names":["bsfishy/zig-lexer"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/BSFishy%2Fzig-lexer","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/BSFishy%2Fzig-lexer/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/BSFishy%2Fzig-lexer/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/BSFishy%2Fzig-lexer/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/BSFishy","download_url":"https://codeload.github.com/BSFishy/zig-lexer/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":245717197,"owners_count":20661078,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2025-01-31T23:54:00.797Z","updated_at":"2025-03-26T18:42:30.188Z","avatar_url":"https://github.com/BSFishy.png","language":"Zig","funding_links":[],"categories":[],"sub_categories":[],"readme":"# zig lexer\n\n\u003e [!CAUTION]\n\u003e I don't recommend that you use this project. I have built it for myself, and\n\u003e as such there are certainly holes in the implementation that I am aware about\n\u003e but will be very difficult to reason about in the spaghetti that I have built.\n\u003e\n\u003e I have open sourced the code for the culture. If you ARE interested in\n\u003e investing some blood, sweat, and tears you are free to fork the project and\n\u003e patch the wholes that I have left. This is and will remain a project that I\n\u003e have built for myself, and as such I won't be paying attention to issues and\n\u003e PRs in this repo.\n\nThis is the home of my lexer. It uses Zig's comptime to generate jump tables at\ncompile time for super easy maintenance and pretty decent performance.\n\nI primarily made this project because I want to use lexers but I am bad at\nwriting them and I don't really care about the implementation. I kinda just care\nabout the tokens and their patterns. And this is exactly that.\n\n```zig\nconst lexer = @import(\"lexer\");\n\nconst token_patterns = [_]lexer.TokenPattern{\n    .Comment = .{ .pattern = \"//([^\\n])*\", .skip = true },\n    .Division = .{ .pattern = \"/\" },\n    .Func = .{ .pattern = \"func\" },\n    .String = .{ .pattern = \"\\\"([^\\\"]|\\\\\\\\\\\")*\\\"\" },\n    .Newline = .{ .pattern = \"(\\n|\\r\\n)\", .skip = true },\n    .Ident = .{ .pattern = \"\\\\w\\\\W*\" },\n};\n\nconst l = lexer.Lexer(\u0026token_patterns).init(allocator);\nconst tokens = try l.lex(input, .{});\nfor (tokens) |token| {\n  _ = token.token_type; // enum of all the variant names\n  _ = token.source; // slice into the input\n}\n```\n\nFeatures:\n\n- Generates jump tables at compile time\n- Supports UTF-8 out of the box\n- Allows matching specific token variants into exhaustive enums\n- Regular expression-esque patterns for matching\n- Linear time complexity lexing\n- Token variants that can be skipped and won't appear in the token enum\n\n## Implementation details\n\nThe first step is parsing the regular expressions. This is a small pattern\nlanguage taken after regular expressions, that only has the functionality that I\nfind useful in making token patterns. You can see every feature that it supports\nin the example above.\n\nOnce tokens are parsed into a tree that represents the structure of the pattern,\nit is passed into the parser. The parser creates a jump table, which is a list\nof tables. The tables are a few maps of kvs. These match a specific character to\neither a next table or a leaf token. There are separate maps for direct matches,\nsequence matches (something like a `\\w`), and fallthroughs (if the character\ndoesn't match anything else and isn't a specific character).\n\nThese are the available sequences:\n\n- **`\\w`** - any alphabetic character\n- **`\\W`** - any alphanumeric character\n- **`\\0`** - any numeric character\n\nThis structure must be recursively constructed due to the fact that the regular\nexpression grammar returns a hierarchical structure. All that code is super\ncomplicated and I quite frankly only have a loose grasp on what is actually\ngoing on. But it works, and that's all I really care about.\n\nOnce the jump table has been generate, we convert it into static tables. This is\nsuper important at the comptime -\u003e runtime boundary. After that, once we have\nUTF-8 input to lex, we just jump around the jump tables, reading through the\ninput.\n\n## Visualizing the tables\n\nBecause the tables are just data in memory, we also have the ability to export\ngraphs representing the tables. This is available through the following API:\n\n```zig\nconst writer = std.io.getStdOut().writer();\ntry l.to_graph(writer);\n```\n\nI found this tremendously useful in building this project. I implemented lexer\nfor a small example language to test out its features. Here is the graphviz of\nthe lexer:\n\n[![Graphviz visualization of the lexer](./.github/graphviz.svg)][graphviz]\n\n## License\n\nThis project is licensed under either:\n\n- MIT License ([LICENSE-MIT](./LICENSE-MIT))\n- Apache License 2.0 ([LICENSE-APACHE](./LICENSE-APACHE))\n\nYou may choose either license.\n\n[graphviz]: https://edotor.net/?engine=dot#digraph%20%7B%0A%20%20start%20-%3E%201%20%5Blabel%3D%22%2F%22%5D%3B%0A%20%201%20-%3E%20Division%20%5Blabel%3D%22super%20%2F%20leaf%22%20color%3Dblue%5D%3B%0A%20%20start%20-%3E%203%20%5Blabel%3D%22f%22%5D%3B%0A%20%203%20-%3E%20Ident%20%5Blabel%3D%22super%20f%20leaf%22%20color%3Dblue%5D%3B%0A%20%20start%20-%3E%206%20%5Blabel%3D%22%5C%22%22%5D%3B%0A%20%20start%20-%3E%20Newline%20%5Blabel%3D%22%5C%5Cn%20leaf%22%20color%3Dblue%5D%3B%0A%20%20start%20-%3E%208%20%5Blabel%3D%22%5C%5Cr%22%5D%3B%0A%20%20start%20-%3E%209%20%5Blabel%3D%22%5C%5Cw%22%20color%3Dorange%5D%3B%0A%20%209%20-%3E%20Ident%20%5Blabel%3D%22super%20%5C%5Cw%20leaf%22%20color%3Dgreen%5D%3B%0A%20%201%20-%3E%202%20%5Blabel%3D%22%2F%22%5D%3B%0A%20%202%20-%3E%20Comment%20%5Blabel%3D%22super%20%2F%20leaf%22%20color%3Dblue%5D%3B%0A%20%202%20-%3E%202%20%5Blabel%3D%22%5C%5Cn%22%20color%3Dred%5D%3B%0A%20%202%20-%3E%20Comment%20%5Blabel%3D%22super%20%5C%5Cn%20leaf%22%20color%3Dpurple%5D%3B%0A%20%203%20-%3E%204%20%5Blabel%3D%22u%22%5D%3B%0A%20%204%20-%3E%20Ident%20%5Blabel%3D%22super%20u%20leaf%22%20color%3Dblue%5D%3B%0A%20%203%20-%3E%2010%20%5Blabel%3D%22%5C%5CW%22%20color%3Dorange%5D%3B%0A%20%2010%20-%3E%20Ident%20%5Blabel%3D%22super%20%5C%5CW%20leaf%22%20color%3Dgreen%5D%3B%0A%20%204%20-%3E%205%20%5Blabel%3D%22n%22%5D%3B%0A%20%205%20-%3E%20Ident%20%5Blabel%3D%22super%20n%20leaf%22%20color%3Dblue%5D%3B%0A%20%204%20-%3E%2011%20%5Blabel%3D%22%5C%5CW%22%20color%3Dorange%5D%3B%0A%20%2011%20-%3E%20Ident%20%5Blabel%3D%22super%20%5C%5CW%20leaf%22%20color%3Dgreen%5D%3B%0A%20%205%20-%3E%2013%20%5Blabel%3D%22c%22%5D%3B%0A%20%2013%20-%3E%20Func%20%5Blabel%3D%22super%20c%20leaf%22%20color%3Dblue%5D%3B%0A%20%205%20-%3E%2012%20%5Blabel%3D%22%5C%5CW%22%20color%3Dorange%5D%3B%0A%20%2012%20-%3E%20Ident%20%5Blabel%3D%22super%20%5C%5CW%20leaf%22%20color%3Dgreen%5D%3B%0A%20%206%20-%3E%207%20%5Blabel%3D%22%5C%5C%22%5D%3B%0A%20%206%20-%3E%20%22String%22%20%5Blabel%3D%22%5C%22%20leaf%22%20color%3Dblue%5D%3B%0A%20%206%20-%3E%206%20%5Blabel%3D%22%5C%22%22%20color%3Dred%5D%3B%0A%20%207%20-%3E%206%20%5Blabel%3D%22%5C%22%22%5D%3B%0A%20%207%20-%3E%206%20%5Blabel%3D%22%5C%22%22%20color%3Dred%5D%3B%0A%20%208%20-%3E%20Newline%20%5Blabel%3D%22%5C%5Cn%20leaf%22%20color%3Dblue%5D%3B%0A%20%209%20-%3E%209%20%5Blabel%3D%22%5C%5CW%22%20color%3Dorange%5D%3B%0A%20%209%20-%3E%20Ident%20%5Blabel%3D%22super%20%5C%5CW%20leaf%22%20color%3Dgreen%5D%3B%0A%20%2010%20-%3E%2010%20%5Blabel%3D%22%5C%5CW%22%20color%3Dorange%5D%3B%0A%20%2010%20-%3E%20Ident%20%5Blabel%3D%22super%20%5C%5CW%20leaf%22%20color%3Dgreen%5D%3B%0A%20%2011%20-%3E%2011%20%5Blabel%3D%22%5C%5CW%22%20color%3Dorange%5D%3B%0A%20%2011%20-%3E%20Ident%20%5Blabel%3D%22super%20%5C%5CW%20leaf%22%20color%3Dgreen%5D%3B%0A%20%2012%20-%3E%2012%20%5Blabel%3D%22%5C%5CW%22%20color%3Dorange%5D%3B%0A%20%2012%20-%3E%20Ident%20%5Blabel%3D%22super%20%5C%5CW%20leaf%22%20color%3Dgreen%5D%3B%0A%20%2013%20-%3E%2014%20%5Blabel%3D%22%5C%5CW%22%20color%3Dorange%5D%3B%0A%20%2014%20-%3E%20Ident%20%5Blabel%3D%22super%20%5C%5CW%20leaf%22%20color%3Dgreen%5D%3B%0A%20%2014%20-%3E%2014%20%5Blabel%3D%22%5C%5CW%22%20color%3Dorange%5D%3B%0A%20%2014%20-%3E%20Ident%20%5Blabel%3D%22super%20%5C%5CW%20leaf%22%20color%3Dgreen%5D%3B%0A%7D%0A\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fbsfishy%2Fzig-lexer","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fbsfishy%2Fzig-lexer","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fbsfishy%2Fzig-lexer/lists"}