{"id":13528315,"url":"https://github.com/Borewit/strtok3","last_synced_at":"2025-04-01T11:31:22.465Z","repository":{"id":21997442,"uuid":"94679236","full_name":"Borewit/strtok3","owner":"Borewit","description":"Promise based streaming tokenizer","archived":false,"fork":false,"pushed_at":"2024-10-28T06:48:34.000Z","size":3263,"stargazers_count":37,"open_issues_count":3,"forks_count":15,"subscribers_count":3,"default_branch":"master","last_synced_at":"2024-10-29T16:00:07.336Z","etag":null,"topics":["binary","decoder","decoding","encoder","encoding","stream","streaming-tokenizer","tokenizer"],"latest_commit_sha":null,"homepage":null,"language":"TypeScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/Borewit.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2017-06-18T10:06:00.000Z","updated_at":"2024-10-26T12:28:41.000Z","dependencies_parsed_at":"2024-06-03T23:43:45.061Z","dependency_job_id":null,"html_url":"https://github.com/Borewit/strtok3","commit_stats":{"total_commits":738,"total_committers":6,"mean_commits":123.0,"dds":0.5555555555555556,"last_synced_commit":"480bd621b7beb834066f80ac0d8494aebba37717"},"previous_names":[],"tags_count":94,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Borewit%2Fstrtok3","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Borewit%2Fstrtok3/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Borewit%2Fstrtok3/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Borewit%2Fstrtok3/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/Borewit","download_url":"https://codeload.github.com/Borewit/strtok3/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":222721797,"owners_count":17028600,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["binary","decoder","decoding","encoder","encoding","stream","streaming-tokenizer","tokenizer"],"created_at":"2024-08-01T06:02:25.316Z","updated_at":"2025-04-01T11:31:22.449Z","avatar_url":"https://github.com/Borewit.png","language":"TypeScript","funding_links":["https://github.com/sponsors/Borewit","https://www.buymeacoffee.com/borewit"],"categories":["TypeScript"],"sub_categories":[],"readme":"[![Node.js CI](https://github.com/Borewit/strtok3/actions/workflows/ci.yml/badge.svg)](https://github.com/Borewit/strtok3/actions/workflows/ci.yml)\n[![CodeQL](https://github.com/Borewit/strtok3/actions/workflows/codeql.yml/badge.svg?branch=master)](https://github.com/Borewit/strtok3/actions/workflows/codeql.yml)\n[![NPM version](https://badge.fury.io/js/strtok3.svg)](https://npmjs.org/package/strtok3)\n[![npm downloads](http://img.shields.io/npm/dm/strtok3.svg)](https://npmcharts.com/compare/strtok3,token-types?start=1200\u0026interval=30)\n[![DeepScan grade](https://deepscan.io/api/teams/5165/projects/8526/branches/103329/badge/grade.svg)](https://deepscan.io/dashboard#view=project\u0026tid=5165\u0026pid=8526\u0026bid=103329)\n[![Known Vulnerabilities](https://snyk.io/test/github/Borewit/strtok3/badge.svg?targetFile=package.json)](https://snyk.io/test/github/Borewit/strtok3?targetFile=package.json)\n[![Codacy Badge](https://api.codacy.com/project/badge/Grade/59dd6795e61949fb97066ca52e6097ef)](https://www.codacy.com/app/Borewit/strtok3?utm_source=github.com\u0026amp;utm_medium=referral\u0026amp;utm_content=Borewit/strtok3\u0026amp;utm_campaign=Badge_Grade)\n# strtok3\n\nA promise based streaming [*tokenizer*](#tokenizer-object) for [Node.js](http://nodejs.org) and browsers.\n\nThe `strtok3` module provides several methods for creating a [*tokenizer*](#tokenizer-object) from various input sources. \nDesigned for:\n* Seamless support in streaming environments.\n* Efficiently decode binary data, strings, and numbers.\n* Reading [predefined](https://github.com/Borewit/token-types) or custom tokens.\n* Offering [*tokenizers*](#tokenizer-object) for reading from [files](#method-strtok3fromfile), [streams](#fromstream-function) or [Uint8Arrays](#frombuffer-function).\n\n### Features\n`strtok3` can read from:\n* Files, using a file path as input.\n* Node.js [streams](https://nodejs.org/api/stream.html).\n* [Buffer](https://nodejs.org/api/buffer.html) or [Uint8Array](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Uint8Array).\n* HTTP chunked transfer provided by [@tokenizer/http](https://github.com/Borewit/tokenizer-http).\n* [Amazon S3](https://aws.amazon.com/s3) chunks with [@tokenizer/s3](https://github.com/Borewit/tokenizer-s3).\n\n## Installation\n\n```sh\nnpm install strtok3\n```\n\n### Compatibility\n\nStarting with version 7, the module has migrated from [CommonJS](https://en.wikipedia.org/wiki/CommonJS) to [pure ECMAScript Module (ESM)](https://gist.github.com/sindresorhus/a39789f98801d908bbc7ff3ecc99d99c).\nThe distributed JavaScript codebase is compliant with the [ECMAScript 2020 (11th Edition)](https://en.wikipedia.org/wiki/ECMAScript_version_history#11th_Edition_%E2%80%93_ECMAScript_2020) standard.\n\nRequires a modern browser, Node.js (V8) ≥ 18 engine or Bun (JavaScriptCore) ≥ 1.2.\n\nFor TypeScript CommonJs backward compatibility, you can use [load-esm](https://github.com/Borewit/load-esm).\n\n\u003e [!NOTE]\n\u003e This module requires a [Node.js ≥ 16](https://nodejs.org/en/about/previous-releases) engine.\n\u003e It can also be used in a browser environment when bundled with a module bundler.\n\n## Support the Project\nIf you find this project useful and would like to support its development, consider sponsoring or contributing:\n\n- [Become a sponsor to Borewit](https://github.com/sponsors/Borewit)\n\n- Buy me a coffee:\n\n  \u003ca href=\"https://www.buymeacoffee.com/borewit\" target=\"_blank\"\u003e\u003cimg src=\"https://cdn.buymeacoffee.com/buttons/default-orange.png\" alt=\"Buy me A coffee\" height=\"41\" width=\"174\"\u003e\u003c/a\u003e\n\n## API Documentation\n\n### strtok3 methods\n\nUse one of the methods to instantiate an [*abstract tokenizer*](#tokenizer-object):\n- [fromFile](#fromfile-function)*\n- [fromStream](#fromstream-function)*\n- [fromWebStream](#fromwebstream-function)\n- [fromBuffer](#frombuffer-function)\n\n\u003e **_NOTE:_**  * `fromFile` and `fromStream`  only available when importing this module with Node.js\n\nAll methods return a [`Tokenizer`](#tokenizer-object), either directly or via a promise.\n\n#### `fromFile` function\n\nCreates a [*tokenizer*](#tokenizer-object) from a local file.\n\n```ts\nfunction fromFile(sourceFilePath: string): Promise\u003cFileTokenizer\u003e\n```  \n\n| Parameter      | Type     | Description                |\n|----------------|----------|----------------------------|\n| sourceFilePath | `string` | Path to file to read from  |\n\n\u003e [!NOTE]\n\u003e - Only available for Node.js engines\n\u003e - `fromFile` automatically embeds [file-information](#file-information)\n\nReturns, via a promise, a [*tokenizer*](#tokenizer-object) which can be used to parse a file.\n\n```js\nimport * as strtok3 from 'strtok3';\nimport * as Token from 'token-types';\n\n(async () =\u003e {\n\n  const tokenizer = await strtok3.fromFile(\"somefile.bin\");\n         try {\n    const myNumber = await tokenizer.readToken(Token.UINT8);\n    console.log(`My number: ${myNumber}`);\n  } finally {\n    tokenizer.close(); // Close the file\n  }\n})();\n```\n\n#### `fromStream` function\n\nCreates a [*tokenizer*](#tokenizer-object) from a Node.js [readable stream](https://nodejs.org/api/stream.html#stream_class_stream_readable).\n\n```ts\nfunction fromStream(stream: Readable, options?: ITokenizerOptions): Promise\u003cReadStreamTokenizer\u003e\n```\n\n| Parameter |  Optional | Type                                                                                 | Description              |\n|-----------|-----------|-------------------------|--------------------------|\n| stream    | no        | [Readable](https://nodejs.org/api/stream.html#stream_class_stream_readable)         | Stream to read from      |\n| fileInfo  | yes       | [IFileInfo](#IFileInfo) | Provide file information |\n\nReturns a Promise providing a [*tokenizer*](#tokenizer-object).\n\n\u003e [!NOTE]\n\u003e - Only available for Node.js engines\n\n#### `fromWebStream` function\n\nCreates [*tokenizer*](#tokenizer-object) from a [WHATWG ReadableStream](https://nodejs.org/api/webstreams.html#web-streams-api).\n\n```ts\nfunction fromWebStream(webStream: AnyWebByteStream, options?: ITokenizerOptions): ReadStreamTokenizer\n```\n\n| Parameter      |  Optional | Type                                                                     | Description                        |\n|----------------|-----------|--------------------------------------------------------------------------|------------------------------------|\n| readableStream | no        | [ReadableStream](https://nodejs.org/api/webstreams.html#web-streams-api) | WHATWG ReadableStream to read from |\n| fileInfo       | yes       | [IFileInfo](#IFileInfo)                                                  | Provide file information           |\n\nReturns a Promise providing a [*tokenizer*](#tokenizer-object)\n\n```js\nimport strtok3 from 'strtok3';\nimport * as Token from 'token-types';\n\nstrtok3.fromWebStream(readableStream).then(tokenizer =\u003e {\n  return tokenizer.readToken(Token.UINT8).then(myUint8Number =\u003e {\n    console.log(`My number: ${myUint8Number}`);\n  });\n});\n```\n\n#### `fromBuffer()` function\n\nCreate a tokenizer from memory ([Uint8Array](https://nodejs.org/api/buffer.html)).\n\n```ts\nfunction fromBuffer(uint8Array: Uint8Array, options?: ITokenizerOptions): BufferTokenizer\n```\n\n| Parameter  | Optional | Type                                             | Description                            |\n|------------|----------|--------------------------------------------------|----------------------------------------|\n| uint8Array | no       | [Uint8Array](https://nodejs.org/api/buffer.html) | Uint8Array or Buffer to read from      |\n| fileInfo   | yes      | [IFileInfo](#IFileInfo)                          | Provide file information               |\n\nReturns a Promise providing a [*tokenizer*](#tokenizer-object).\n\n```js\nimport * as strtok3 from 'strtok3';\n\nconst tokenizer = strtok3.fromBuffer(buffer);\n\ntokenizer.readToken(Token.UINT8).then(myUint8Number =\u003e {\n  console.log(`My number: ${myUint8Number}`);\n});\n```\n\n### `Tokenizer` object\nThe *tokenizer* is an abstraction of a [stream](https://nodejs.org/api/stream.html), file or [Uint8Array](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Uint8Array), allowing _reading_ or _peeking_ from the stream.\nIt can also be translated in chunked reads, as done in [@tokenizer/http](https://github.com/Borewit/tokenizer-http);\n\n#### Key Features:\n\n- Supports seeking within the stream using `tokenizer.ignore()`.\n- Offers `peek` methods to preview data without advancing the read pointer.\n- Maintains the read position via tokenizer.position.\n\n#### Tokenizer functions\n\n_Read_ methods advance the stream pointer, while _peek_ methods do not.\n\nThere are two kind of functions:\n1. *read* methods: used to read a *token* of [Buffer](https://nodejs.org/api/buffer.html) from the [*tokenizer*](#tokenizer-object). The position of the *tokenizer-stream* will advance with the size of the token.\n2. *peek* methods: same as the read, but it will *not* advance the pointer. It allows to read (peek) ahead.\n\n#### `readBuffer` function\n\nRead data from the _tokenizer_ into provided \"buffer\" (`Uint8Array`).\n`readBuffer(buffer, options?)`\n\n```ts\nreadBuffer(buffer: Uint8Array, options?: IReadChunkOptions): Promise\u003cnumber\u003e;\n```\n\n| Parameter  | Type                                                           | Description                                                                                                                                                                                                                            |\n|------------|----------------------------------------------------------------|----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|\n| buffer     | [Buffer](https://nodejs.org/api/buffer.html) \u0026#124; Uint8Array | Target buffer to write the data read to                                                                                                                                                                                                |\n| options    | [IReadChunkOptions](#ireadchunkoptions)                        | An integer specifying the number of bytes to read                                                                                                                                                                                      |\n\nReturn promise with number of bytes read.\nThe number of bytes read maybe if less, *mayBeLess* flag was set.\n\n#### `peekBuffer` function\n\nPeek (read ahead), from [*tokenizer*](#tokenizer-object), into the buffer without advancing the stream pointer.\n\n```ts\npeekBuffer(uint8Array: Uint8Array, options?: IReadChunkOptions): Promise\u003cnumber\u003e;\n```\n\n| Parameter  | Type                                    | Description                                         |\n|------------|-----------------------------------------|-----------------------------------------------------|\n| buffer     | Buffer \u0026#124; Uint8Array                | Target buffer to write the data read (peeked) to.   |\n| options    | [IReadChunkOptions](#ireadchunkoptions) | An integer specifying the number of bytes to read.  |                                                                                                                           |\n\nReturn value `Promise\u003cnumber\u003e` Promise with number of bytes read. The number of bytes read maybe if less, *mayBeLess* flag was set.\n\n#### `readToken` function\n\nRead a *token* from the tokenizer-stream.\n\n```ts\nreadToken\u003cValue\u003e(token: IGetToken\u003cValue\u003e, position: number = this.position): Promise\u003cValue\u003e\n```  \n\n| Parameter  | Type                    | Description                                                                                                           |\n|------------|-------------------------|---------------------------------------------------------------------------------------------------------------------- |\n| token      | [IGetToken](#IGetToken) | Token to read from the tokenizer-stream.                                                                              |\n| position?  | number                  | Offset where to begin reading within the file. If position is null, data will be read from the current file position. |\n\nReturn value `Promise\u003cnumber\u003e`. Promise with number of bytes read. The number of bytes read maybe if less, *mayBeLess* flag was set.\n\n#### `peek` function\n\nPeek a *token* from the [*tokenizer*](#tokenizer-object).\n\n```ts\npeekToken\u003cValue\u003e(token: IGetToken\u003cValue\u003e, position: number = this.position): Promise\u003cValue\u003e\n```\n\n| Parameter  | Type                       | Description                                                                                                             |\n|------------|----------------------------|-------------------------------------------------------------------------------------------------------------------------|\n| token      | [IGetToken\u003cT\u003e](#IGetToken) | Token to read from the tokenizer-stream.                                                                                |\n| position?  | number                     | Offset where to begin reading within the file. If position is null, data will be read from the current file position.   |\n\nReturn a promise with the token value peeked from the [*tokenizer*](#tokenizer-object).\n\n#### `readNumber` function\n\nPeek a numeric [*token*](#token) from the [*tokenizer*](#tokenizer-object).\n\n```ts\nreadNumber(token: IToken\u003cnumber\u003e): Promise\u003cnumber\u003e\n```\n\n| Parameter  | Type                            | Description                                        |\n|------------|---------------------------------|----------------------------------------------------|\n| token      | [IGetToken\u003cnumber\u003e](#IGetToken) | Numeric token to read from the tokenizer-stream.   |\n\nReturns a promise with the decoded numeric value from the *tokenizer-stream*.\n\n#### `ignore` function\n\nAdvance the offset pointer with the token number of bytes provided.\n\n```ts\nignore(length: number): Promise\u003cnumber\u003e\n```\n\n| Parameter  | Type   | Description                                                          |\n|------------|--------|----------------------------------------------------------------------|\n| ignore     | number | Numeric of bytes to ignore. Will advance the `tokenizer.position`    |\n\nReturns a promise with the decoded numeric value from the *tokenizer-stream*.\n\n#### `close` function\nClean up resources, such as closing a file pointer if applicable.\n\n#### `Tokenizer` attributes\n\n- `fileInfo`\n\n  Optional attribute describing the file information, see [IFileInfo](#IFileInfo)\n\n- `position`\n\n  Pointer to the current position in the [*tokenizer*](#tokenizer-object) stream.\n  If a *position* is provided to a _read_ or _peek_ method, is should be, at least, equal or greater than this value.\n\n### `IReadChunkOptions` interface\n\nEach attribute is optional:\n\n| Attribute | Type    | Description                                                                                                                                                                                                                   |\n|-----------|---------|-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|\n| length    | number  | Requested number of bytes to read.                                                                                                                                                                                            |\n| position  | number  | Position where to peek from the file. If position is null, data will be read from the [current file position](#attribute-tokenizerposition). Position may not be less then [tokenizer.position](#attribute-tokenizerposition) |\n| mayBeLess | boolean | If and only if set, will not throw an EOF error if less then the requested *mayBeLess* could be read.                                                                                                                         |\n\nExample usage:\n```js\n  tokenizer.peekBuffer(buffer, {mayBeLess: true});\n```\n\n### `IFileInfo` interface\n\nProvides optional metadata about the file being tokenized.\n\n| Attribute | Type    | Description                                                                                       |\n|-----------|---------|---------------------------------------------------------------------------------------------------|\n| size      | number  | File size in bytes                                                                                |\n| mimeType  | number  | [MIME-type](https://developer.mozilla.org/en-US/docs/Web/HTTP/Basics_of_HTTP/MIME_types) of file. |\n| path      | number  | File path                                                                                         |\n| url       | boolean | File URL                                                                                          |\n\n### `Token` object\n\nThe *token* is basically a description what to read form the [*tokenizer-stream*](#tokenizer-object).\nA basic set of *token types* can be found here: [*token-types*](https://github.com/Borewit/token-types).\n\nA token is something which implements the following interface:\n```ts\nexport interface IGetToken\u003cT\u003e {\n\n  /**\n   * Length in bytes of encoded value\n   */\n  len: number;\n\n  /**\n   * Decode value from buffer at offset\n   * @param buf Buffer to read the decoded value from\n   * @param off Decode offset\n   */\n  get(buf: Uint8Array, off: number): T;\n}\n```\nThe *tokenizer* reads `token.len` bytes from the *tokenizer-stream* into a Buffer.\nThe `token.get` will be called with the Buffer. `token.get` is responsible for conversion from the buffer to the desired output type.\n\n### Working with Web-API readable stream\nTo convert a [Web-API readable stream](https://developer.mozilla.org/en-US/docs/Web/API/ReadableStreamDefaultReader) into a [Node.js readable stream]((https://nodejs.org/api/stream.html#stream_readable_streams)), you can use [readable-web-to-node-stream](https://github.com/Borewit/readable-web-to-node-stream) to convert one in another.\n\n```js\nimport { fromWebStream } strtok3 from 'strtok3';\nimport { ReadableWebToNodeStream } from 'readable-web-to-node-stream';\n\n(async () =\u003e {\n\n  const response = await fetch(url);\n  const readableWebStream = response.body; // Web-API readable stream\n  const webStream = new ReadableWebToNodeStream(readableWebStream); // convert to Node.js readable stream\n\n  const tokenizer = fromWebStream(webStream); // And we now have tokenizer in a web environment\n})();\n```\n\n## Dependencies\n\nThe diagram below illustrates the primary dependencies of `strtok3`:\n\n```mermaid\ngraph TD;\n  S(strtok3)--\u003eP(peek-readable)\n  S(strtok3)--\u003eTO(\"@tokenizer/token\")\n```\n \n- [peek-readable](https://github.com/Borewit/peek-readable): Manages reading operations with peeking capabilities, allowing data to be previewed without advancing the read pointer.\n- [@tokenizer/token](https://github.com/Borewit/tokenizer-token): Provides token definitions and utilities used by `strtok3` for interpreting binary data.\n\n## Licence\n\nThis project is licensed under the [MIT License](LICENSE.txt). Feel free to use, modify, and distribute as needed.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FBorewit%2Fstrtok3","html_url":"https://awesome.ecosyste.ms/projects/github.com%2FBorewit%2Fstrtok3","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FBorewit%2Fstrtok3/lists"}