{"id":35205214,"url":"https://github.com/kiblitz/alexis","last_synced_at":"2026-05-17T20:34:20.936Z","repository":{"id":330665843,"uuid":"1122528217","full_name":"kiblitz/alexis","owner":"kiblitz","description":"an approximately single-pass lexer library powered by regex deterministic finite automata","archived":false,"fork":false,"pushed_at":"2025-12-27T16:03:04.000Z","size":1408,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2025-12-28T19:39:38.646Z","etag":null,"topics":["dfa","lexer","library","ocaml","regex"],"latest_commit_sha":null,"homepage":"","language":"OCaml","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/kiblitz.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2025-12-25T00:05:13.000Z","updated_at":"2025-12-27T16:03:07.000Z","dependencies_parsed_at":null,"dependency_job_id":null,"html_url":"https://github.com/kiblitz/alexis","commit_stats":null,"previous_names":["kiblitz/alexis"],"tags_count":null,"template":false,"template_full_name":null,"purl":"pkg:github/kiblitz/alexis","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/kiblitz%2Falexis","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/kiblitz%2Falexis/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/kiblitz%2Falexis/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/kiblitz%2Falexis/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/kiblitz","download_url":"https://codeload.github.com/kiblitz/alexis/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/kiblitz%2Falexis/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":33153850,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-17T09:28:26.183Z","status":"ssl_error","status_checked_at":"2026-05-17T09:27:52.702Z","response_time":107,"last_error":"SSL_connect returned=1 errno=0 peeraddr=140.82.121.6:443 state=error: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["dfa","lexer","library","ocaml","regex"],"created_at":"2025-12-29T13:38:57.014Z","updated_at":"2026-05-17T20:34:20.930Z","avatar_url":"https://github.com/kiblitz.png","language":"OCaml","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Alexis\nA library for writing lexers using regex rules.\n\nAlexis is powered by [DFAs](https://en.wikipedia.org/wiki/Deterministic_finite_automaton) which makes inference approximately a single-pass operation.\n\n# Getting Started\n## Overview\nAlexis provides a functor which requires\n```\ntype token\n\nval token_dfa : token Regex_dfa.t\n```\nand generates a lexer that outputs a `token` list on success.\n## Demo\nCheck out `examples/` for some toy lexers built using Alexis!\n\nYou can also try out the `lexcaster_example_bin` binary, a repl for the lexers in `examples/`:\n\n```\ndune exec bin/alexis_lexcaster_example_bin.exe\n```\n\n*note: whitespace is treated as a special character that is skipped between lexed tokens*\n\nhttps://github.com/user-attachments/assets/a3d0542b-29d2-46b2-b072-c44faa71a370\n\n## Making a `Regex_dfa`\n### Regex Configs\nRegex configs are recursively defined with the following type signature:\n```\ntype t =\n  | Empty\n  | Epsilon\n  | Char of char\n  | Concat of t * t\n  | Or of t * t\n  | Star of t\n```\n*(`Epsilon` is the empty string and `Empty` doesn't match any input)*\n\nAlexis also exposes some common constructors for creating these configs:\n```\nval plus : t -\u003e t\nval opt : t -\u003e t\nval exact : string -\u003e t\nval char_or : char list -\u003e t\nval concat : t list -\u003e t\nval or_ : t list -\u003e t\n```\n*i.e. `exact` matches an exact string; `concat` matches a sequence of regexes; etc.*\n\nEach regex config matches a **single token**.\n\n### Regex DFAs\n\nA single regex config generates a single DFA which accepts if the regex config matches the input.\n```\nval create : ?priority:int -\u003e Config.t -\u003e cont_of_match:(Buffer.t -\u003e 'a) -\u003e 'a t\n```\nThe first thing to note is `cont_of_match` -- this is a continuation that is invoked on a successful match. It takes a `Buffer.t` because it might be the case that the matched input is required to generate the associated token.\n\n*For example, if you want to match a keyword, you probably won't need the buffer and can just apply `Fn.const`. On the other hand, if you want to match a string, you'll need the actual contents of the string to create the token.*\n\nThe second thing to note is the `priority` optional argument. It is necessary to understand the lexer mechanism before this parameter can be explained.\n```\nval merge_list : 'a t Nonempty_list.t -\u003e 'a t\n```\nNotably, a lexer is a collection of DFAs. It does a sliding window (current buffer) which continues to expand *as long as there is at least 1 DFA that is not in a dead state*. In other words, it tries to match the current buffer to a single DFA (and then convert it into the associated token via `cont_of_match`). If the current buffer results in every DFA being in a dead state, it must retrieve the (cached) last state where at least 1 DFA was in an accept state (otherwise, lexing fails due to bad input).\n\nThe `priority` argument (default=`1`) is used in situations where multiple DFAs end up in an accepting state -- highest `priority` DFA is chosen.\n\nThis is especially useful for lexing freeform tokens (i.e. variable names) which should be set to **low** priority so that keywords are chosen ahead of them.\n\n*For instance, `true` is a valid variable name in the sense that it starts with an alpha and contains only alphanumerics, but it should definitely match to a `bool`.*\n\n# Implementation Details\n## Core Concepts\n### DFA Core\nFoundationally, every DFA node is a total function from `char` to another DFA node. Additionally, every node is either an accept node or a fail node -- if the state by \u003ceof\u003e is accept, Alexis will invoke that node's continuation (`cont_of_match`). Without changing semantics, Alexis replaces the concept of fail nodes with incomplete and dead nodes -- where dead nodes are just sink nodes. This allows the optimization of exiting early instead of having to read the entire input.\n\n### DFA Construction\nAs implied by the API, Alexis generates a DFA using a regex config. It does so by (fundamentally) mapping the universe of regex configs to a DFA node and minting edges between nodes via [Brzozowski derivative](https://en.wikipedia.org/wiki/Brzozowski_derivative) application. Intuitively, the derivative is a function applying a `char` to the regex config that outputs another regex config representative of a partial inference on the input regex config.\n\nOne small optimization note: applying the derivative on the entire universe of characters per regex config (node) is computationally expensive, so Alexis first generates the \"next char\" universe of that regex and iterates that set.\n\n### Merge\nMerge is a key part of the Alexis algorithm. Essentially, it collapses a set of DFAs (for each token) into a single mega-DFA. This is what drives the approximately single-pass inference performance -- the lexer only needs to feed the input into this machine rather than to those of each individual token.\n\nThis is done via a \"threading\" algorithm. Mathematically, the combiner (function of 2 DFAs) takes the cross product of the nodes between its inputs. Intuitively, the combined inference is like performing inference on each individual DFA and assigning the current node to be the pair of nodes in each DFA.\n\nIf:\n- both are incomplete nodes -\u003e combined node is incomplete\n- only 1 is an accept node -\u003e combined node is an accept node with the same continuation as the accept input node\n- (at least) 1 is a dead node -\u003e combined is the other node\n- both are accept nodes -\u003e combined node is an accept node whose continuation is that of the input node **with the highest priority**\n\n## Performance\n\nDFA inference is a single-pass algorithm. However, the Alexis lexer continues to perform partial (per-character) inference until it reaches a dead state, **and then backtracks to the last accept state**. In practice, most inputs and grammars shouldn't run into bad versions of this (where this practically dominates the computation time).\n\n### Degenerate Edgecase Example\n- Token Foo: matches on [a]\n- Token Bar: matches on [aa...aab]\n- Input: [aa...aa]\n\nAlexis will read the whole input because [Bar] is still matchable until the very end. After reaching \u003ceof\u003e, Alexis will backtrack to index 0 (which is the last accept state -- accepting [Foo]). At this point, the lexer will continue iteration from index 1 (and repeat the same process) resulting in a quadratic operation.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fkiblitz%2Falexis","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fkiblitz%2Falexis","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fkiblitz%2Falexis/lists"}