https://github.com/arlac77/chunk-tokenizer

Transform stream that emits tokens
https://github.com/arlac77/chunk-tokenizer
Last synced: 10 months ago
JSON representation
Transform stream that emits tokens
Host: GitHub
URL: https://github.com/arlac77/chunk-tokenizer
Owner: arlac77
License: 0bsd
Created: 2017-08-06T11:01:24.000Z (almost 9 years ago)
Default Branch: master
Last Pushed: 2025-04-24T12:35:33.000Z (about 1 year ago)
Last Synced: 2025-04-24T13:36:22.587Z (about 1 year ago)
Language: JavaScript
Homepage:
Size: 1.72 MB
Stars: 0
Watchers: 2
Forks: 0
Open Issues: 2
Metadata Files:
- Readme: README.md
- License: LICENSE
Awesome Lists containing this project

README

          [![npm](https://img.shields.io/npm/v/chunk-tokenizer.svg)](https://www.npmjs.com/package/chunk-tokenizer)

[![License](https://img.shields.io/badge/License-0BSD-blue.svg)](https://spdx.org/licenses/0BSD.html)

[![bundlejs](https://deno.bundlejs.com/?q=chunk-tokenizer\&badge=detailed)](https://bundlejs.com/?q=chunk-tokenizer)

[![downloads](http://img.shields.io/npm/dm/chunk-tokenizer.svg?style=flat-square)](https://npmjs.org/package/chunk-tokenizer)

[![GitHub Issues](https://img.shields.io/github/issues/arlac77/chunk-tokenizer.svg?style=flat-square)](https://github.com/arlac77/chunk-tokenizer/issues)

[![Build Status](https://img.shields.io/endpoint.svg?url=https%3A%2F%2Factions-badge.atrox.dev%2Farlac77%2Fchunk-tokenizer%2Fbadge\&style=flat)](https://actions-badge.atrox.dev/arlac77/chunk-tokenizer/goto)

[![Styled with prettier](https://img.shields.io/badge/styled_with-prettier-ff69b4.svg)](https://github.com/prettier/prettier)

[![Commitizen friendly](https://img.shields.io/badge/commitizen-friendly-brightgreen.svg)](http://commitizen.github.io/cz-cli/)

[![Known Vulnerabilities](https://snyk.io/test/github/arlac77/chunk-tokenizer/badge.svg)](https://snyk.io/test/github/arlac77/chunk-tokenizer)

[![Coverage Status](https://coveralls.io/repos/arlac77/chunk-tokenizer/badge.svg)](https://coveralls.io/github/arlac77/chunk-tokenizer)

# chunk-tokenizer

Transform stream that emits tokens

# API

### Table of Contents

*   [EOFToken](#eoftoken)

*   [KeywordToken](#keywordtoken)

*   [makeKeywordTokens](#makekeywordtokens)

    *   [Parameters](#parameters)

*   [LineCommentIgnoreToken](#linecommentignoretoken)

*   [makeLineCommentToken](#makelinecommenttoken)

    *   [Parameters](#parameters-1)

*   [makeOperatorTokens](#makeoperatortokens)

    *   [Parameters](#parameters-2)

*   [StringChunk](#stringchunk)

    *   [Parameters](#parameters-3)

    *   [Properties](#properties)

    *   [append](#append)

        *   [Parameters](#parameters-4)

    *   [appendLast](#appendlast)

        *   [Parameters](#parameters-5)

    *   [extractFromMarkedPosition](#extractfrommarkedposition)

    *   [markPosition](#markposition)

        *   [Parameters](#parameters-6)

    *   [peek](#peek)

    *   [advance](#advance)

    *   [advanceBy](#advanceby)

        *   [Parameters](#parameters-7)

*   [parse](#parse)

    *   [Parameters](#parameters-8)

*   [TokenMatcher](#tokenmatcher)

    *   [Parameters](#parameters-9)

    *   [Properties](#properties-1)

*   [Token](#token)

    *   [possibleFirstChars](#possiblefirstchars)

    *   [register](#register)

        *   [Parameters](#parameters-10)

*   [characterSetFromString](#charactersetfromstring)

    *   [Parameters](#parameters-11)

*   [WhitespaceIgnoreToken](#whitespaceignoretoken)

## EOFToken

**Extends Token**

Token representing EOF

## KeywordToken

**Extends Token**

## makeKeywordTokens

Creates a new token class for each token definition.

### Parameters

*   `tokenDefinitions` **[Object](https://developer.mozilla.org/docs/Web/JavaScript/Reference/Global_Objects/Object)** keys are the operator names

*   `baseToken` **[KeywordToken](#keywordtoken)**  (optional, default `KeywordToken`)

Returns **[Array](https://developer.mozilla.org/docs/Web/JavaScript/Reference/Global_Objects/Array)<[KeywordToken](#keywordtoken)>** newly created KeywordToken classes

## LineCommentIgnoreToken

**Extends Token**

Token to skip until end of line

## makeLineCommentToken

### Parameters

*   `prefix` **[string](https://developer.mozilla.org/docs/Web/JavaScript/Reference/Global_Objects/String)** 

*   `baseToken` **Class**  (optional, default `LineCommentIgnoreToken`)

## makeOperatorTokens

Creates a new token class for each token definition.

### Parameters

*   `tokenDefinitions`  {Object} keys are the operator names

*   `baseToken`  {OperatorToken} (optional, default `OperatorToken`)

Returns **[Array](https://developer.mozilla.org/docs/Web/JavaScript/Reference/Global_Objects/Array)\** newly created OperatorToken classes

## StringChunk

### Parameters

*   `buffer`   (optional, default `""`)

*   `isLast`   (optional, default `false`)

### Properties

*   `position` **[Number](https://developer.mozilla.org/docs/Web/JavaScript/Reference/Global_Objects/Number)** current peek position in the buffer

*   `buffer` **[string](https://developer.mozilla.org/docs/Web/JavaScript/Reference/Global_Objects/String)** 

*   `currentLine` **[Number](https://developer.mozilla.org/docs/Web/JavaScript/Reference/Global_Objects/Number)** 

### append

append content of buffer

and reset the position(s)

#### Parameters

*   `buffer` **[string](https://developer.mozilla.org/docs/Web/JavaScript/Reference/Global_Objects/String)** 

### appendLast

Indicate that this will be the last chunk

append content of buffer

and reset the position(s)

#### Parameters

*   `buffer` **[string](https://developer.mozilla.org/docs/Web/JavaScript/Reference/Global_Objects/String)** 

### extractFromMarkedPosition

### markPosition

mark position and prepserve state

#### Parameters

*   `state` **[object](https://developer.mozilla.org/docs/Web/JavaScript/Reference/Global_Objects/Object)** 

Returns **any** former preseved or newly set state

### peek

Returns **[Number](https://developer.mozilla.org/docs/Web/JavaScript/Reference/Global_Objects/Number)** char at the current position

### advance

Advance current position by one (after delivring the current char)

Returns **[Number](https://developer.mozilla.org/docs/Web/JavaScript/Reference/Global_Objects/Number)** char at the current position

### advanceBy

Advance current position by numberOfChars

#### Parameters

*   `numberOfChars` **[Number](https://developer.mozilla.org/docs/Web/JavaScript/Reference/Global_Objects/Number)** 

## parse

0 -> skip leading "

1 -> copy chars

2 -> escape

### Parameters

*   `chunk`  

## TokenMatcher

Holds a Set of tokens and identifies them based on the longest matching character string

### Parameters

*   `tokens` **[Array](https://developer.mozilla.org/docs/Web/JavaScript/Reference/Global_Objects/Array)<[Token](#token)>** 

### Properties

*   `tokens` **[Array](https://developer.mozilla.org/docs/Web/JavaScript/Reference/Global_Objects/Array)<[Token](#token)>** 

*   `registeredTokens` **[Map](https://developer.mozilla.org/docs/Web/JavaScript/Reference/Global_Objects/Map)<[string](https://developer.mozilla.org/docs/Web/JavaScript/Reference/Global_Objects/String), [Token](#token)>** 

*   `maxTokenLengthForFirstChar` **[Map](https://developer.mozilla.org/docs/Web/JavaScript/Reference/Global_Objects/Map)\** 

## Token

Abstract base token

### possibleFirstChars

Possible first chars

Returns **any** Set() all possible first chars for the token

### register

register the token in the TokenMatcher

#### Parameters

*   `tokenMatcher` **[TokenMatcher](#tokenmatcher)** 

## characterSetFromString

### Parameters

*   `str` **[string](https://developer.mozilla.org/docs/Web/JavaScript/Reference/Global_Objects/String)** 

Returns **[Set](https://developer.mozilla.org/docs/Web/JavaScript/Reference/Global_Objects/Set)<[string](https://developer.mozilla.org/docs/Web/JavaScript/Reference/Global_Objects/String)>** all characters from the string

## WhitespaceIgnoreToken

**Extends Token**

Token to consume all consecutive whitespace

# install

With [npm](http://npmjs.org) do:

```shell

npm install chunk-tokenizer

```

# license

BSD-2-Clause
ecosyste.ms

Data

Tools

Indexes

Applications

Experiments

Awesome

https://github.com/arlac77/chunk-tokenizer

Awesome Lists containing this project

README
