{"id":19430200,"url":"https://github.com/etherdream/quickreader","last_synced_at":"2025-03-17T12:10:52.942Z","repository":{"id":37700868,"uuid":"505875458","full_name":"EtherDream/QuickReader","owner":"EtherDream","description":"An ultra-high performance stream reader for browser and Node.js","archived":false,"fork":false,"pushed_at":"2023-04-26T03:07:33.000Z","size":200,"stargazers_count":167,"open_issues_count":1,"forks_count":4,"subscribers_count":4,"default_branch":"main","last_synced_at":"2025-03-04T09:40:31.680Z","etag":null,"topics":["await-async","browser","nodejs","performance","reader","selective-await","stream"],"latest_commit_sha":null,"homepage":"","language":"TypeScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/EtherDream.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2022-06-21T14:21:32.000Z","updated_at":"2024-08-22T16:50:36.000Z","dependencies_parsed_at":"2024-12-26T15:11:06.640Z","dependency_job_id":"38be79cc-113c-4dc4-b89e-e76bf0972a59","html_url":"https://github.com/EtherDream/QuickReader","commit_stats":{"total_commits":9,"total_committers":1,"mean_commits":9.0,"dds":0.0,"last_synced_commit":"022ac4baed04cc95979f62b808e3e19ac6917279"},"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/EtherDream%2FQuickReader","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/EtherDream%2FQuickReader/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/EtherDream%2FQuickReader/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/EtherDream%2FQuickReader/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/EtherDream","download_url":"https://codeload.github.com/EtherDream/QuickReader/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":244031033,"owners_count":20386534,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["await-async","browser","nodejs","performance","reader","selective-await","stream"],"created_at":"2024-11-10T14:23:56.016Z","updated_at":"2025-03-17T12:10:52.920Z","avatar_url":"https://github.com/EtherDream.png","language":"TypeScript","funding_links":[],"categories":[],"sub_categories":[],"readme":"# QuickReader\n\nAn ultra-high performance stream reader for browser and Node.js, easy-to-use, zero dependency.\n\n[![NPM Version](https://badgen.net/npm/v/quickreader)](https://npmjs.org/package/quickreader)\n[![NPM Install Size](https://badgen.net/packagephobia/install/quickreader)](https://packagephobia.com/result?p=quickreader)\n![GitHub CI](https://github.com/EtherDream/quickreader/actions/workflows/ci.yml/badge.svg)\n\n\n# Install\n\n```bash\nnpm i quickreader\n```\n\n\n# Demo\n\n```js\nimport {QuickReader, A} from 'quickreader'\n\nconst res = await fetch('https://unpkg.com/quickreader-demo/demo.bin')\nconst stream = res.body   // ReadableStream\nconst reader = new QuickReader(stream)\n\ndo {\n  const id   = reader.u32() ?? await A\n  const name = reader.txt() ?? await A\n  const age  = reader.u8()  ?? await A\n\n  console.log(id, name, age)\n\n} while (!reader.eof)\n```\n\nhttps://jsbin.com/loyuxad/edit?html,console\n\nWith a stream reader, you can read the data in the specified types while downloading, which makes the user experience better. You don't have to do the chunk slicing and buffering yourself, the reader does all that.\n\nWithout it, you would have to wait for all the data to be downloaded before you could read it (e.g., via DataView). Since JS doesn't support structures, you have to pass in an offset parameter for each read, which is inconvenient to use.\n\n\n# Why Quick\n\nWe used two tricks to improve performance:\n\n* selective await\n\n* synchronized EOF\n\n## selective await\n\nThe overhead of await is considerable, here is a test:\n\n```js\nlet s1 = 0, s2 = 0\n\nconsole.time('no-await')\nfor (let i = 0; i \u003c 1e7; i++) {\n  s1 += i\n}\nconsole.timeEnd('no-await')   // ~15ms\n\nconsole.time('await')\nfor (let i = 0; i \u003c 1e7; i++) {\n  s2 += await i\n}\nconsole.timeEnd('await')      // Chrome: ~800ms, Safari: ~3000ms\n```\n\nhttps://jsbin.com/gehazin/edit?html,output\n\nThe above two cases do the same thing, but the await one is 50x to 200x slower than the no-await. On Chrome it's even ~2000x slower if the console is open (only ~500 await/ms).\n\nThis test seems meaningless, but in fact, sometimes we call await heavily in an almost synchronous logic, such as an async query function that will mostly hit the memory cache and return.\n\n```js\nasync function query(key) {\n  if (cacheMap.has(key)) {\n    return ...  // 99.9%\n  }\n  await ...\n}\n```\n\nReading data from a stream has the same issue. For a single integer or tiny text, it takes only a few bytes, in most cases, it can be read directly from the buffer without I/O calls, so await is unnecessary; await is only needed when the buffer is not enough.\n\nIf await is called only when needed, the overhead can be reduced many times.\n\n```js\nconsole.time('selective-await')\nfor (let i = 0; i \u003c 1e7; i++) {\n  const value = (i % 1000)  // buffer enough?\n    ? i         // 99.9%\n    : await i   //  0.1%\n}\nconsole.timeEnd('selective-await')  // ~40ms 🚀\n```\n\nFor `QuickReader`, when its buffer is enough, it returns the result immediately; otherwise, it returns nothing (`undefined`), and the result can be obtained by `await A`.\n\n```js\nfunction readBytes(len) {\n  if (len \u003c availableLen) {\n    return buf.subarray(offset, offset + len)   // likely\n  }\n  A = readBytesAsync(len)\n}\n\nasync function readBytesAsync(len) {\n  await stream.read()\n  ...\n}\n```\n\nThe calling logic can be simplified into one line using the [nullish coalescing operator](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Operators/Nullish_coalescing_operator):\n\n```js\nresult = readBytes(10) ?? await A\n```\n\nThis is both high-performance and easy-to-use. \n\n\u003e Note: The `A` is not a global variable in the real code, it's just an imported object that implements [thenable](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Operators/await#thenable_objects), you can rename it on import.\n\n## synchronized EOF\n\n`QuickReader` always keeps its buffer at least 1 byte, this means when the available buffer length is 4 and call `reader.u32()`, the result will not be returned immediately, but requires await. The buffer will not be fully read until the stream is closed.\n\nIn this way, the EOF state can be detected synchronously, nearly zero overhead.\n\n\u003e Note: If the data must be fully read before continuing (e.g. reading a handshake command from a socket stream and replying), do not use `QuickReader`, otherwise it will wait forever. `QuickReader` assumes that data can be read all the time, e.g. a file stream.\n\n# Node.js\n\nNodeStream is also supported:\n\n```js\nconst stream = fs.createReadStream('/path/to')\nconst reader = new QuickReader(stream)\n// ...\n```\n\nIf the stream provides data as `Buffer`, buffer-related methods like `bytes`, `bytesTo`, etc. return `Buffer` as well, otherwise they return `Uint8Array`.\n\n\u003e [Buffer](https://nodejs.org/api/buffer.html#buffer) is a subclass of `Uint8Array`.\n\n\n# API\n\n## Class\n\n* QuickReader`\u003cT extends Uint8Array = Uint8Array\u003e`\n\n## Constructor\n\n* new(stream: `AsyncIterable\u003cT\u003e` | `ReadableStream\u003cT\u003e`)\n\n## By length\n\n* bytes(len: `number`) : `T` | `undefined`\n\n* skip(len: `number`) : `number` | `undefined`\n\n* txtNum(len: `number`) : `string` | `undefined`\n\n## By delimiter\n\n* bytesTo(delim: `number`) : `T` | `undefined`\n\n* skipTo(delim: `number`) : `number` | `undefined`\n\n* txtTo(delim: `number`) : `string` | `undefined`\n\n## Helper\n\n* txt() : `string` - Equivalent to `txtTo(0)`.\n\n* txtLn() : `string` - Equivalent to `txtTo(10)`.\n\n## Number\n\n* {u, i}{8, 16, 32, 16be, 32be}() : `number` | `undefined`\n\n* {u, i}{64, 64be}() : `bigint` | `undefined`\n\n* f{32, 64, 32be, 64be}() : `number` | `undefined`\n\n## Chunk\n\n* chunk(): `Promise\u003cT\u003e`\n\n* chunks(len: number) : `AsyncGenerator\u003cT\u003e`\n\n* chunksToEnd(len: number) : `AsyncGenerator\u003cT\u003e`\n\n## Property\n\n* eof: `boolean`\n\n* eofAsDelim: `boolean` (default `false`)\n\n## More\n\nSee [index.d.ts](typings/index.d.ts)\n\n\u003e Note: The program will not check the parameter `len`, but converts it to u32,  which may cause negative values become large numbers. So the caller needs to ensure that `len \u003e= 0`. Similarly, the parameter `delim` will not be checked, but converted to u8.\n\n\n# EOF State\n\nSince the `eof` property is synchronized, it's meaningless until the first read.\n\nIf the stream is expected to be non-empty, you can read before detecting:\n\n```js\nconst reader = new QuickReader(stream)\ndo {\n  const line = reader.txtLn() ?? await A\n} while (!reader.eof)\n```\n\nHowever, an error will be thrown when the stream is empty.\n\nIf the empty stream needs to be considered, you can call `pull` method first, then the `eof` will be meaningful:\n\n```js\nconst reader = new QuickReader(stream)\nawait reader.pull()\n\nwhile (!reader.eof) {\n  const line = reader.txtLn() ?? await A\n}\n```\n\nIn this way, no error will be thrown if the stream is empty.\n\n\n# Buffer Type\n\nThe generic type `T` can be used as a type hint for the buffer:\n\n```ts\nclass QuickReader\u003cT extends Uint8Array = Uint8Array\u003e {\n\n  new(stream: AsyncIterable\u003cT\u003e | ReadableStream\u003cT\u003e)\n\n  public bytes(len: number) : T | undefined\n  public bytesTo(delim: number) : T | undefined\n\n  public chunk() : Promise\u003cT\u003e\n  public chunks(len: number) : AsyncGenerator\u003cT\u003e\n  public chunksToEnd(len: number) : AsyncGenerator\u003cT\u003e\n}\n```\n\n`T` is based on the type of the construction parameter `stream`, if it is not clear, you need to specify `T` manually:\n\n```ts\n{\n  const reader = new QuickReader\u003cBuffer\u003e(fs.createReadStream('/path/to'))\n  const buffer = reader.bytes(10) ?? await A\n  buffer  // Type Hint: Buffer\n}\n{\n  const reader = new QuickReader\u003cUint8Array\u003e(getStreamSomeHow())\n  const buffer = reader.bytes(10) ?? await A\n  buffer  // Type Hint: Uint8Array\n}\n```\n\nWhen `T` is explicit, buffer-related methods can get the expected return type hint.\n\n\n# Chunk Reading\n\n## chunk\n\nWhen processing a large file, after reading the header, sometimes we want to read the remaining data chunk by chunk instead of all at once. In this case, we can use the `chunk` method:\n\n```js\nconst header = reader.byte(10) ?? await A\ndo {\n  const chunk = await reader.chunk()\n  // ...\n} while (!reader.eof)\n```\n\nOnce this method is called, the amount of remaining data will be unpredictable, you can only call this method repeatedly.\n\n## chunks\n\nUsing the `chunks` method, you can specify the read length:\n\n```js\nconst ver = reader.u32() ?? await A\nconst len = reader.u32() ?? await A\n\nfor await (const chunk of reader.chunks(len)) {\n  // ...\n}\n\nconst trailer = reader.bytes(8) ?? await A\n```\n\nIn this way, after reading some chunks, you can continue to read data with types.\n\n\u003e During iteration, calling other methods to read data is not allowed.\n\n## chunksToEnd\n\nThis method will read all data until `len` bytes remaining, so that the trailer can be excluded.\n\n```ts\nconst header = reader.bytes(10) ?? await A\n\nfor await (const chunk of reader.chunksToEnd(8)) {\n  // ...\n}\n\nconst trailer = reader.bytes(8) ?? await A\n```\n\nAfter calling this method, the stream is closed, and the buffer has `len` bytes.\n\nIf `len` is `0`, it is similar to calling `chunk` method repeatedly.\n\n# Type Check\n\nIt is better to use TypeScript. When you forget to add `?? await A`, the type of result will be unioned with `undefined`, which makes it easier to expose the issue.\n\n```js\nconst id = reader.u32()   // number | undefined\nid.toString()             // ❌\n```\n\n\n# Concurrency\n\nThe same reader is not allowed to be called by multiple co-routines in parallel, as this would break the waiting order. Therefore, the following logic should not be used:\n\n```js\nconst reader = new QuickReader(stream)\n\nasync function routine() {\n  do {\n    const id = reader.u32() ?? await A\n    const name = reader.txt() ?? await A\n    // ...\n  } while (!reader.eof)\n}\n\n// ❌\nfor (let i = 0; i \u003c 10; i++) {\n  routine()\n}\n```\n\n\n# Read Line\n\n`QuickReader` is also a high performance line reader. It reduces the overhead by ~60% compared to the Node.js' native `readline` module, because its parsing logic is simpler, e.g. using only `\\n` delimiter (ignoring `\\r`).\n\n```js\nconst stream = fs.createReadStream('log.txt')\nconst reader = new QuickReader(stream)\nawait reader.pull()\n\n// no error if the file does not end with '\\n'\nreader.eofAsDelim = true\n\nwhile (!reader.eof) {\n  const line = reader.txtLn() ?? await A\n  // ...\n}\n```\n\nOf course, as mentioned above, concurrency is not supported. If there are multiple co-routines reading the same file, it is better to use the native `readline` module:\n\n```js\nimport fs from 'node:fs'\nimport readline from 'node:readline'\n\nconst stream = fs.createReadStream('urls.txt')\nconst rl = readline.createInterface({input: stream})\nconst iter = rl[Symbol.asyncIterator]()\n\nasync function routine() {\n  for (;;) {\n    const {value: url} = await iter.next()\n    if (!url) { \n      break\n    }\n    const res = await fetch(url)\n    // ...\n  }\n}\n\nfor (let i = 0; i \u003c 100; i++) {\n  routine()\n}\n```\n\n\n# About\n\nThe idea of this project was born when the `await` keyword was introduced. The earliest solution was:\n\n```js\nconst result = reader.read() || await A\n```\n\nSince the `||` operator will also short-circuit `0` and `''`, so it was not perfect, until the `??` was introduced in ES2020.\n\nHowever, the performance of await had been greatly improved compared to the past, so it was not as meaningful as it was then. Anyway, I still share this idea, even if it is 2022 now, after all, performance optimization is never-ending.\n\nDue to limited time and English, the document and some code comments (e.g. index.d.ts) were translated via Google, hopefully someone will improve it.\n\n\n# License\n\nMIT\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fetherdream%2Fquickreader","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fetherdream%2Fquickreader","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fetherdream%2Fquickreader/lists"}