https://github.com/ertrzyiks/pawn-tokenizer

Pawn language tokenizer
https://github.com/ertrzyiks/pawn-tokenizer

Last synced: about 1 month ago
JSON representation

Pawn language tokenizer

• Host: GitHub
• URL: https://github.com/ertrzyiks/pawn-tokenizer
• Owner: ertrzyiks
• License: mit
• Created: 2014-09-15T16:50:05.000Z (over 10 years ago)
• Default Branch: master
• Last Pushed: 2014-11-09T16:22:46.000Z (about 10 years ago)
• Last Synced: 2024-12-07T19:36:04.211Z (about 1 month ago)
• Language: JavaScript
• Homepage:
• Size: 254 KB
• Stars: 1
• Watchers: 3
• Forks: 0
• Open Issues: 0
• Metadata Files:
  • Readme: README.md
  • License: LICENSE

Awesome Lists containing this project

README

        
pawn-tokenizer [![Build Status](https://travis-ci.org/ertrzyiks/pawn-tokenizer.svg?branch=master)](https://travis-ci.org/ertrzyiks/pawn-tokenizer)

==============

Based on [c-tokenizer](https://github.com/substack/c-tokenizer)

# example

``` js

var tokenize = require('../lib');

var t = tokenize(function (src, token) {

    console.log(token.type + ' => ' + JSON.stringify(src));

});

process.stdin.pipe(t);

```

For the input file plugin.sma:

``` c

#include 

public plugin_init(){

    register("Plugin Name", "1.0", "Author");

}

```

output:

```

$ node example/tokens.js < example/plugin.sma

directive => "#include"

whitespace => " "

operator => "<"

identifier => "amxmodx"

operator => ">"

whitespace => "\n"

whitespace => "\n"

keyword => "public"

whitespace => " "

identifier => "plugin_init"

open paren => "("

close paren => ")"

open curly => "{"

whitespace => "\n"

whitespace => " "

whitespace => " "

whitespace => " "

whitespace => " "

identifier => "register"

open paren => "("

quote => "\"Plugin Name\""

operator => ","

whitespace => " "

quote => "\"1.0\""

operator => ","

whitespace => " "

quote => "\"Author\""

close paren => ")"

operator => ";"

whitespace => "\n"

close curly => "}"

whitespace => "\n"

```

# install

With [npm](https://npmjs.org) do:

```

npm install pawn-tokenizer

```

# license

MIT