https://github.com/clitetailor/tokenize-monster

Yet another powerful tokenizer in js.
https://github.com/clitetailor/tokenize-monster

functional-programming tokenize tokenizer

Last synced: about 2 months ago
JSON representation

Yet another powerful tokenizer in js.

• Host: GitHub
• URL: https://github.com/clitetailor/tokenize-monster
• Owner: clitetailor
• License: mit
• Created: 2017-05-09T16:54:57.000Z (over 7 years ago)
• Default Branch: master
• Last Pushed: 2017-06-30T11:11:54.000Z (over 7 years ago)
• Last Synced: 2024-12-02T05:16:37.169Z (about 2 months ago)
• Topics: functional-programming, tokenize, tokenizer
• Language: JavaScript
• Size: 101 KB
• Stars: 0
• Watchers: 2
• Forks: 0
• Open Issues: 0
• Metadata Files:
  • Readme: README.md
  • License: LICENSE

Awesome Lists containing this project

README

        
Tokenize Monster

================

> Yet another powerful tokenizer in **JS**.

Installation

------------

```bash

npm install --save tokenize-monster

```

Getting Started

---------------

```javascript

const tokenizer = require('tokenize-monster');

const tokens = tokenizer.matchAndMap(

	'i\'d like to  some ',

	/<[^><]+>/,

	text => [[text]],

	text => [text]

);

console.log(tokens);

// => [ 'i\'d like to ', [ '' ], ' some ', [ '' ], '' ]

const nTokens = tokenizer.matchAndMap(

	'i\'d like to  some ',

	/<[^><]+>/,

	text => [text],

	text => []

);

console.log(nTokens);

// => [ '', '' ]

```

Example

-------

- [Simple XML Parser](https://github.com/clitetailor/simple-xml-parser)

Documentation

-------------

Our docs are available at [Tokenize Monster Wiki](https://github.com/clitetailor/tokenize-monster/wiki).

Changelog

---------

Visit [Github Releases](https://github.com/clitetailor/tokenize-monster/releases) page for more information.