https://github.com/kessler/tokenize

personal tokenize util
https://github.com/kessler/tokenize

nodejs nodejs-modules tokenizer

Last synced: 7 months ago
JSON representation

personal tokenize util

• Host: GitHub
• URL: https://github.com/kessler/tokenize
• Owner: kessler
• License: mit
• Created: 2019-01-12T09:11:26.000Z (about 7 years ago)
• Default Branch: master
• Last Pushed: 2020-04-15T14:32:56.000Z (almost 6 years ago)
• Last Synced: 2025-06-15T16:08:02.109Z (7 months ago)
• Topics: nodejs, nodejs-modules, tokenizer
• Language: JavaScript
• Size: 7.81 KB
• Stars: 3
• Watchers: 1
• Forks: 1
• Open Issues: 0
• Metadata Files:
  • Readme: README.md
  • License: LICENSE

Awesome Lists containing this project

README

          
# tokenize

**personal tokenize util**

## example

`npm i -S @kessler/tokenize`

```js

const tokenize = require('@kessler/tokenize')

let tokens1 = tokenize('text')

// with non default tokenization rules

const rules = {

    tokenChars: ['(', ')', ',', '[', ']'],

    globbingChars:  ['"', '\''],

    separators: [' ']

}

let tokens2 = tokenize('(text) "asd" blabla', rules)

```

## license

[MIT](http://opensource.org/licenses/MIT) © Yaniv Kessler