https://github.com/perrawd/tokenizer.js
A lexical analysis parser module built with JS.
https://github.com/perrawd/tokenizer.js
javascript nodejs parser
Last synced: 2 months ago
JSON representation
A lexical analysis parser module built with JS.
- Host: GitHub
- URL: https://github.com/perrawd/tokenizer.js
- Owner: perrawd
- Created: 2021-09-01T13:06:07.000Z (almost 5 years ago)
- Default Branch: master
- Last Pushed: 2021-10-21T23:51:24.000Z (over 4 years ago)
- Last Synced: 2025-08-20T13:55:49.002Z (10 months ago)
- Topics: javascript, nodejs, parser
- Language: JavaScript
- Homepage:
- Size: 1.09 MB
- Stars: 0
- Watchers: 1
- Forks: 0
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
Awesome Lists containing this project
README
# Lexical Analysis
A module that performs a lexical analysis for a given string with a lexical grammar.
## 🔨 How to use
1. Instantiate a new lexical grammar object for the grammar to be used for lexical analysis. [Read more](./src/lib/lexical-grammar/README.md)
2. Instantiate a new LexicalAnalysis object with a lexical grammar type and the string that is to be analysed.
Example:
```
// Import modules.
import GrammaticType from '../src/lib/lexical-grammar/lexical-grammar.js'
import GRAMMAR_NAME from '../src/lib/lexical-grammar/grammar-name.js'
import LexicalAnalysis from '../src/lexical-analysis.js'
// Assign grammar type (optional).
const { ARITHMETIC } = GRAMMAR_NAME
// Instantiate a new object for the grammar type to be used.
const arithmeticGrammar = new GrammaticType(ARITHMETIC)
// Create a new lexical analysis.
const aritmeticAnalysis = new LexicalAnalysis(arithmeticGrammar, '38 + 4')
// Get active token
console.log(aritmeticAnalysis.getActiveToken()) // Token { tokenMatch: 0, tokenType: 'NUMBER', value: '38' }
// Set the current active token to next.
aritmeticAnalysis.setActiveTokenToNext()
console.log(aritmeticAnalysis.getActiveToken()) // Token { tokenMatch: 1, tokenType: 'ADD', value: '+' }
// Set the current active token to next (twice).
aritmeticAnalysis.setActiveTokenToNext()
aritmeticAnalysis.setActiveTokenToNext()
console.log(aritmeticAnalysis.getActiveToken()) // Token { tokenMatch: 3, tokenType: 'END', value: 'END' }
// Set the current active token to previous.
aritmeticAnalysis.setActiveTokenToPrevious()
console.log(aritmeticAnalysis.getActiveToken()) // Token { tokenMatch: 2, tokenType: 'NUMBER', value: '4' }
console.table(aritmeticAnalysis.getTokenList()) // Table of token list.
```
## 🧰 Public methods
### getActiveToken()
Returns the currently active token.
### setActiveTokenToPrevious()
Sets the pointer of the current token to the previous one in the token list.
Throws error if first token has been reached.
### setActiveTokenToNext()
Sets the pointer of the current token to the next one in the token list.
Creates the next token if needed.
Throws error if END token has been reached.
### getTokenList()
Returns an array of the created tokens.
## ✏️ Add your own grammar
[Read more](./src/lib/lexical-grammar/README.md)
## ⛩️ Classes
### LexicalAnalysis
Instantiate a new ```LexicalAnalysis``` object.
Constructor arguments:
- {LexicalGrammar} ```LexicalGrammar``` The lexical grammar to be used.
- {string} The string to be analysed/tokenized.
### LexicalGrammar
Instantiate a new ```LexicalGrammar``` object.
Constructor arguments:
- {string} ```GRAMMAR_NAME``` name of the grammar to be used.
### Token
Used by ```LexicalAnalysis```.
Instantiate a new ```Token``` object.
Constructor arguments:
- {string} ```tokenMatch``` The index of token.
- {string} ```tokenType``` The tokenType.
- {string} ```value``` The subString.
## 🚀 Built with:
- JavaScript (ES6)
- Node.js
🧪 Tested with:
- Jest
✍️ Coding standard:
- @lnu/eslint-config