https://github.com/jimsynz/lex_luthor

Lexer in Elixir. Lexer in Elixir. Lexer in Elixir. Lexer in Elixir. Lexer in Elixir. Lexer in Elixir. Lexer in Elixir. Lexer in Elixir. Lexer in Elixir. Lexer in Elixir.
https://github.com/jimsynz/lex_luthor

Last synced: about 2 months ago
JSON representation

Lexer in Elixir. Lexer in Elixir. Lexer in Elixir. Lexer in Elixir. Lexer in Elixir. Lexer in Elixir. Lexer in Elixir. Lexer in Elixir. Lexer in Elixir. Lexer in Elixir.

• Host: GitHub
• URL: https://github.com/jimsynz/lex_luthor
• Owner: jimsynz
• Created: 2015-03-05T23:31:06.000Z (over 9 years ago)
• Default Branch: master
• Last Pushed: 2024-04-13T04:56:19.000Z (2 months ago)
• Last Synced: 2024-04-15T14:18:41.626Z (2 months ago)
• Language: Elixir
• Homepage: https://harton.dev/james/lex_luthor
• Size: 64.5 KB
• Stars: 31
• Watchers: 2
• Forks: 5
• Open Issues: 0
• Metadata Files:
  • Readme: README.md
  • Changelog: CHANGELOG.md

Lists

• awesome-elixir - lex_luthor - LexLuthor is a Lexer in Elixir which uses macros to generate a reusable lexers. (Lexical analysis)

README

        
# LexLuthor

[![Build Status](https://drone.harton.dev/api/badges/james/lex_luthor/status.svg)](https://drone.harton.dev/james/lex_luthor)

[![Hex.pm](https://img.shields.io/hexpm/v/lex_luthor.svg)](https://hex.pm/packages/lex_luthor)

[![Hippocratic License HL3-FULL](https://img.shields.io/static/v1?label=Hippocratic%20License&message=HL3-FULL&labelColor=5e2751&color=bc8c3d)](https://firstdonoharm.dev/version/3/0/full.html)

LexLuthor is a Lexer in Elixir (say that 10 times fast) which uses macros to generate a reusable lexers. Good times.

LexLuthor is a state based lexer, meaning that it keeps a state stack which you can push states on and pop states off the stack, which are used to filter the applicable rules for a given state. For example:

```elixir

defmodule StringLexer do

  use LexLuthor

  defrule ~r/^'/,              fn(_) -> :STRING end

  defrule ~r/^[^']+/, :STRING, fn(e) -> { :string, e } end

  defrule ~r/^'/,     :STRING, fn(_) -> nil end

end

```

Rules are defined by a regular expression, an optional state (as an atom) and an action in the form of an anonymous function.

When passed the string `'foo'`, the lexer starts in the `:default` state, so it filters for rules in the default state (the first rule, as it doesn't specify a state), then it filters the available rules by the longest matching regular expression. In this case, since we have only one rule (which happens to match) it's automatically the longest match.

Once the longest match is found, then it's action is executed and the return value matched:

- If the return value is a single atom then that atom is assumed to be a state and push onto the top of the state stack.

- If the return value is a two element tuple then the first element is expected to be an atom (the token name) and the second element a value for this token.

- If the return value is `nil` then the top state is popped off the state stack.

If lexing succeeds then you will receive an `:ok` tuple with the second value being a list of `LexLuthor.Token` structs.

If lexing fails then you will receive an `:error` tuple which a reason and position.

## Installation

LexLuthor is [available in Hex](https://hex.pm/packages/lex_luthor), the package

can be installed by adding `lex_luthor` to your list of dependencies in

`mix.exs`:

```elixir

def deps do

  [

    {:lex_luthor, "~> 0.1.2"}

  ]

end

```

Documentation for the latest release can be found on

[HexDocs](https://hexdocs.pm/lex_luthor) and for the `main` branch on

[docs.harton.nz](https://docs.harton.nz/james/lex_luthor).

## Github Mirror

This repository is mirrored [on Github](https://github.com/jimsynz/lex_luthor)

from it's primary location [on my Forgejo instance](https://harton.dev/james/lex_luthor).

Feel free to raise issues and open PRs on Github.