https://github.com/qix-/flexicon

A lightweight, regex-based lexer framework for Python
https://github.com/qix-/flexicon

Last synced: 28 days ago
JSON representation

A lightweight, regex-based lexer framework for Python

• Host: GitHub
• URL: https://github.com/qix-/flexicon
• Owner: Qix-
• License: mit
• Created: 2017-05-16T06:21:45.000Z (over 7 years ago)
• Default Branch: master
• Last Pushed: 2017-05-16T06:29:56.000Z (over 7 years ago)
• Last Synced: 2024-10-03T20:35:02.450Z (about 2 months ago)
• Language: Python
• Size: 5.86 KB
• Stars: 12
• Watchers: 3
• Forks: 1
• Open Issues: 2
• Metadata Files:
  • Readme: README.md
  • License: LICENSE

Awesome Lists containing this project

README

        
# Flexicon

Flexicon is a simple [regex](https://pypi.python.org/pypi/regex/)-based lexer and tokenizer.

## Installation

```console

$ pip install flexicon

```

## Usage

```python

from flexicon import Lexer

# Simple Expression Lexer

EXPRESSION_LEXER = Lexer().simple(

    (r'[ \t]+', lambda: None),

    (r'\+', lambda: ('ADD',)),

    (r'\/', lambda: ('DIVIDE',)),

    (r'\-', lambda: ('SUBTRACT',)),

    (r'\*', lambda: ('MULTIPLY',)),

    (r'\(', lambda: ('OPAREN',)),

    (r'\)', lambda: ('CPAREN',)),

    (r'([0-9]+)', lambda n: ('NUMBER', int(n))),

    (r'([a-zA-Z])', lambda c: ('VARIABLE', c))

)

print(EXPRESSION_LEXER.lex(u'1 + 2a(4 / b)'))

# Outputs:

# [

#      ('NUMBER', 1),

#      ('ADD',),

#      ('NUMBER', 2),

#      ('VARIABLE', 'a'),

#      ('OPAREN',),

#      ('NUMBER', 4),

#      ('DIVIDE',),

#      ('VARIABLE', 'b'),

#      ('CPAREN',)

# ]

```

# License

Copyright © 2017, Josh Junon. Released under the [MIT License](LICENSE).