https://github.com/robinweser/tokenize-sync

Simple synchronous string tokenizer using Regex
https://github.com/robinweser/tokenize-sync

compiler synchronous tokenization tokenizer tokenizing-parser

Last synced: 23 days ago
JSON representation

Simple synchronous string tokenizer using Regex

• Host: GitHub
• URL: https://github.com/robinweser/tokenize-sync
• Owner: robinweser
• License: mit
• Created: 2017-06-17T16:45:02.000Z (over 7 years ago)
• Default Branch: master
• Last Pushed: 2017-11-13T21:32:45.000Z (almost 7 years ago)
• Last Synced: 2024-10-13T13:38:57.020Z (25 days ago)
• Topics: compiler, synchronous, tokenization, tokenizer, tokenizing-parser
• Language: JavaScript
• Size: 35.2 KB
• Stars: 1
• Watchers: 2
• Forks: 0
• Open Issues: 0
• Metadata Files:
  • Readme: README.md
  • License: LICENSE

Awesome Lists containing this project

README

        
# Synchronous Tokenizer

A simple synchronous string tokenizer using Regex.

    

## Installation

```sh

yarn add tokenize-sync

```

## Usage

The package ships a single `tokenize` function that takes an (*string*) input and a (*Object*) ruleMap that maps (*string*) token names to Regexes.

```javascript

import tokenize from 'tokenize-sync'

const ruleMap = {

  identifier: /^[a-z-]+$/i,

  number: /^\d+$/,

  whitespace: /^\s+$/

}

const input = 'test 12  foobar3'

const tokens = tokenize(input, ruleMap)

tokens === [{

  type: 'identifier',

  value: 'test',

  start: 0,

  end: 4

}, {

  type: 'whitespace',

  value: ' ',

  start: 4,

  end: 5

}, {

  type: 'number',

  value: '12',

  start: 5,

  end: 7

}, {

  type: 'whitespace',

  value: '  ',

  start: 7,

  end: 9

}, {

  type: 'identifier',

  value: 'foobar',

  start: 9,

  end: 15

},  {

  type: 'number',

  value: '3',

  start: 15,

  end: 16

}]

```

## License

tokenize-sync is licensed under the [MIT License](http://opensource.org/licenses/MIT).


Documentation is licensed under [Creative Common License](http://creativecommons.org/licenses/by/4.0/).


Created with ♥ by [@rofrischmann](http://rofrischmann.de).