https://github.com/chengluyu/tokenizer

A JavaScript tokenizer.
https://github.com/chengluyu/tokenizer

Last synced: 2 months ago
JSON representation

A JavaScript tokenizer.

• Host: GitHub
• URL: https://github.com/chengluyu/tokenizer
• Owner: chengluyu
• License: mit
• Created: 2016-02-24T16:57:42.000Z (over 9 years ago)
• Default Branch: master
• Last Pushed: 2016-02-26T01:49:51.000Z (over 9 years ago)
• Last Synced: 2025-02-08T16:14:31.678Z (4 months ago)
• Language: JavaScript
• Size: 6.84 KB
• Stars: 0
• Watchers: 3
• Forks: 0
• Open Issues: 0
• Metadata Files:
  • Readme: README.md
  • License: LICENSE

Awesome Lists containing this project

README

        
# tokenizer

A JavaScript tokenizer.

The tokenizer takes a pattern object and produces a function that converts string into token stream.

## Usage

To use tokenizer, you need a array containing pattern objects of token. Each pattern object is in following form:

``` json

{

  "name": "integer",

  "pattern": "[+-]?\\d+",

  "ignorable": false

}

```

The `name` field specify the token name. The `pattern` field specify the pattern of token in regular expression. And the optional `ignorable` field specify whether this token is ignorable, or in other word, if the value of `ignorable` is true, it won't appear in the output token stream. The default value of `ignorable` field is `false`.

By following example you will know how to use it.

``` javascript

let Tokenizer = require("tokenizer").Tokenizer;

let tokenizer = new Tokenizer([

  {

    "name": "integer",

    "pattern": "[+-]?\\d+"

  },

  {

    "name": "operator",

    "pattern": "[+-*\\()]"

  },

  {

    "name": "whitespace",

    "pattern": "\\s+",

    "ignorable": true

  }

]);

let advance = tokenizer("1+2 *(3-4)");

while (true) {

  let token = advance();

  console.log(`type = ${token.type} ; value = '${token.value}'`);

  if (token.type === "eof")

    break;

}

```

And you will get the following output:

``` 

type = integer ; value = 1

type = operator ; value = '+'

type = integer ; value = 2

type = operator ; value = '*'

type = operator ; value = '('

type = integer ; value = 3

type = operator ; value = '-'

type = integer ; value = 4

type = operator ; value = ')'

```