{"id":23282102,"url":"https://github.com/bitpatty/node-buffered-file-reader","last_synced_at":"2025-09-14T04:04:57.107Z","repository":{"id":61012140,"uuid":"546611987","full_name":"BitPatty/node-buffered-file-reader","owner":"BitPatty","description":"A NodeJS TypeScript library for reading a file in buffered chunks","archived":false,"fork":false,"pushed_at":"2025-04-01T13:47:15.000Z","size":1373,"stargazers_count":1,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"master","last_synced_at":"2025-04-02T23:42:38.698Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":"TypeScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/BitPatty.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2022-10-06T11:06:32.000Z","updated_at":"2025-04-01T13:47:12.000Z","dependencies_parsed_at":"2024-06-26T17:02:54.329Z","dependency_job_id":"f7b033df-2262-4dde-8eda-67f6c177a627","html_url":"https://github.com/BitPatty/node-buffered-file-reader","commit_stats":{"total_commits":189,"total_committers":2,"mean_commits":94.5,"dds":0.05291005291005291,"last_synced_commit":"08b3aab9e10af305c302e6aa9bb41c7f87232ae2"},"previous_names":["bitpatty/node-buffered-file-reader","bitpatty/buffered-file-reader"],"tags_count":8,"template":false,"template_full_name":"BitPatty/ts-library-template","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/BitPatty%2Fnode-buffered-file-reader","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/BitPatty%2Fnode-buffered-file-reader/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/BitPatty%2Fnode-buffered-file-reader/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/BitPatty%2Fnode-buffered-file-reader/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/BitPatty","download_url":"https://codeload.github.com/BitPatty/node-buffered-file-reader/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":247492544,"owners_count":20947545,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-12-20T00:14:38.308Z","updated_at":"2025-09-14T04:04:57.095Z","avatar_url":"https://github.com/BitPatty.png","language":"TypeScript","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Buffered File Reader\n\nA NodeJS library for reading a file in buffered chunks, allowing you to finish the processing of a chunk before reading the next one.\n\n## Usage\n\nThe library by default exports a function that instantiates a `BufferedFileReader` and returns its [generator](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Statements/function*) function. The base skeleton for use within an application could look as follows:\n\n```typescript\nimport createReader from '@bitpatty/buffered-file-reader';\n\n// Create the reader instance for the target file\nconst reader = createReader('path-to-your-file');\n\nfor (\n  let chunk = await reader.next();\n  !chunk.done;\n  chunk = await reader.next()\n) {\n  // Do something with chunk.value.data\n}\n```\n\n### Chunk Value\n\nThe chunk value is passed in the format below - whereas `Buffer` is a [`NodeJS Buffer`](https://nodejs.org/api/buffer.html). The typing of the iterator and its result can also be imported from the library as follows: `import { Iterator, IteratorResult } from '@bitpatty/buffered-file-reader'`. Note: There is a builtin `IteratorResult` type in NodeJS, make sure you import it from the library if you intend to use it.\n\n```typescript\ntype IteratorResult = {\n  /**\n   * The state of the chunk cursor for this data buffer.\n   *\n   * Note: Indexing starts at 0\n   */\n  chunkCursor: {\n    /**\n     * The cursor position at the start of where the initial chunk\n     * of the current buffer was read.\n     *\n     * This number is inclusive, meaning that the byte at this\n     * position has been read into the current data buffer.\n     */\n    start: number;\n\n    /**\n     * The cursor position at the end of where the last chunk of\n     * the current buffer was read.\n     *\n     * This number is exclusive, meaning the the byte at this\n     * position has not been read into the current data buffer.\n     */\n    end: number;\n  };\n\n  /**\n   * Peeks at the next data buffer following the result of this\n   * iteration without moving the cursor forward.\n   *\n   * @returns  The next data buffer\n   */\n  peekNext: () =\u003e Promise\u003cOmit\u003cIteratorResult, 'peekNext'\u003e | null\u003e;\n\n  /**\n   * The data in the current chunk\n   */\n  data: Buffer;\n};\n```\n\n### End of file\n\nTo know whether the reader reached the end of the file (no more to read), you can either check whether the value is `null` or whether the `done` flag is set:\n\n```typescript\nconst chunkA = await reader.next();\nif (chunkA.value == null) console.log('done');\n\n// .. or ..\nconst chunkA = await reader.next();\nif (chunkA.done) console.log('done');\n```\n\n### Exiting early\n\n\u003e Note: If you don't read the whole file you should always call the return function to close the open handles.\n\nIf you want to stop further reads you can call `return` on the reader:\n\n```typescript\nimport createReader from '@bitpatty/buffered-file-reader';\n\n// Create the reader instance for the target file\nconst reader = createReader('path-to-your-file');\n\n// ...\n\nreader.return(null);\n```\n\n## Configuration Options\n\nThe following options can be passed as second argument when instantiating the reader:\n\n```typescript\n{\n  // The offset (in Bytes) from which to start reading the file\n  //\n  // If set to a number greater than 0, that number of bytes\n  // will be skipped when reading the first chunk.\n  //\n  // Defaults to `0`\n  startOffset: 123;\n\n  // The default chunk size of the reader.\n  //\n  // If provided, the specified number of bytes is read on each\n  // read operation. If no separator is configured, this will\n  // also be the size of the buffer returned on each iteration.\n  //\n  // Note that when the reader reaches the end of the file,\n  // the buffer size may be less than the chunk size.\n  //\n  // Defaults to `100`\n  chunkSize: 100;\n\n  //\n  // Whether to fail if the file is modified while its being\n  // processed by the file reader.\n  //\n  // If set to `true`, the file will be tracked for changes\n  // by `watchFile`. If any changes are done to the file\n  // the following read will throw an error.\n  //\n  // See: https://nodejs.org/docs/latest/api/fs.html#fs_fs_watchfile_filename_options_listener\n  //\n  // Defaults to `true`.\n  //\n  throwOnFileModification: boolean;\n\n  // The interval of file modification checks in milliseconds.\n  //\n  // The interval specifies at which rate the file stats should\n  // be checked for any modifications.\n  //\n  // Note that the actual detection frequency is still relying\n  // on the NodeJS file watcher.\n  //\n  // Defaults to `1000`\n  //\n  fileModificationPollInterval: number;\n\n  // The byte pattern to identify the end of a section.\n  //\n  // If provided, the reader continues to read bytes into the buffer\n  // until it encounters the separator and continues reading from\n  // the end of the separator during the next iteration.\n  //\n  // This is especially useful when dealing with text files with\n  // a delimiter (such as newlines). Note that the chunk size\n  // configuration still applies for read operations. However,\n  // the returned buffer size may be less or greater than the\n  // configured size, depending on where it encounters the\n  // separator pattern.\n  //\n  // Defaults to `undefined`\n  separator: new Uint8Array([0x01, 0x02, 0x03]);\n\n  // Whether the encountered separator should be trimmed from\n  // The returned buffer. This configuration is ignored if no\n  // separator is specified.\n  //\n  // If this is set to true the separator will be removed from\n  // the end of the buffer before being returned as result of\n  // the current iteration, else it is kept in the buffer.\n  //\n  // E.g. The file 0x12345678 with the separator 0x56 will return\n  //      0x1234 if this is set to `true`, else it will return\n  //      0x123456.\n  //\n  // Defaults to `false`\n  trimSeparator: true;\n}\n```\n\n### Separators\n\nThe library provides a set of separators signatures which you can provide in the `separator` configuration (see example below). Note that you can always define your own separator by passing an `Uint8Array`.\n\n```typescript\nimport createReader, { Separator } '@bitpatty/buffered-file-reader'\n\n// Create the reader instance for the target file\nconst reader = createReader('path-to-your-file', {\n  separator: Separator.CRLF\n});\n\n```\n\nThe following separators are available within the library itself:\n\n```typescript\nconst Separator = {\n  /**\n   * Matches the carriage return character `\\r`\n   */\n  CR: new Uint8Array([0x0d]),\n  /**\n   * Matches the newline character `\\n`\n   */\n  LF: new Uint8Array([0x0a]),\n  /**\n   * Matches the carriage return newline characters `\\r\\n`\n   */\n  CRLF: new Uint8Array([0x0d, 0x0a]),\n  /**\n   * Matches the NULL terminator `\\0`\n   */\n  NULL_TERMINATOR: new Uint8Array([0x00]),\n};\n```\n\n## Examples\n\nThis section provides a few examples on how the library can be used.\n\n### Reading a text file line by line\n\nThe following snippet reads a text file containing lines separated with a newline character `\\n`.\n\n\u003e Note that since the separator is a new line character (`\\n`) and you choose to trim the separator\n\u003e a potential carriage return (`\\r`) is not trimmed.\n\n```typescript\n// my-file.txt:\n// lorem ipsum\n// dolor sit amet\n\nimport createReader, { Separator } from '@bitpatty/buffered-file-reader';\n\nconst reader = createReader('my-file.txt', {\n  separator: Separator.LF,\n  trimSeparator: true,\n});\n\nconst line1 = await reader.next();\nconst line2 = await reader.next();\nconst line3 = await reader.next();\n\nconsole.log(line1.value.data.toString()); // lorem ipsum\nconsole.log(line2.value.data.toString()); // dolor sit amet\n```\n\n### Reading a binary file byte by byte\n\nThe following snippet reads a binary file byte-by-byte\n\n```typescript\n// my-file.bin:\n// 0x0011\n\nimport createReader from '@bitpatty/buffered-file-reader';\n\nconst reader = createReader('my-file.txt', { chunkSize: 1 });\n\nconst byte1 = await reader.next();\nconst byte2 = await reader.next();\nconst byte3 = await reader.next();\nconst byte4 = await reader.next();\n\nconsole.log(byte1.value.data); // \u003cBuffer 00\u003e\nconsole.log(byte2.value.data); // \u003cBuffer 11\u003e\nconsole.log(byte3.value.data); // \u003cnull\u003e\nconsole.log(byte4.value.data); // \u003cundefined\u003e\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fbitpatty%2Fnode-buffered-file-reader","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fbitpatty%2Fnode-buffered-file-reader","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fbitpatty%2Fnode-buffered-file-reader/lists"}