{"id":22541331,"url":"https://github.com/kayahr/datastream","last_synced_at":"2025-08-01T05:04:31.653Z","repository":{"id":50675024,"uuid":"425084639","full_name":"kayahr/datastream","owner":"kayahr","description":"Data stream classes for writing and reading all kinds of data types, even single bits","archived":false,"fork":false,"pushed_at":"2025-07-31T13:51:34.000Z","size":2329,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":2,"default_branch":"main","last_synced_at":"2025-07-31T17:39:15.237Z","etag":null,"topics":["data","datastream","input","output","stream","typescript"],"latest_commit_sha":null,"homepage":"https://kayahr.github.io/datastream/","language":"TypeScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/kayahr.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":".github/funding.yml","license":"LICENSE.md","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null},"funding":{"github":"kayahr","custom":"https://paypal.me/kayaahr/"}},"created_at":"2021-11-05T21:08:00.000Z","updated_at":"2025-07-31T13:51:38.000Z","dependencies_parsed_at":"2024-01-26T00:28:02.051Z","dependency_job_id":"74f3af63-be66-4958-a6f5-4c8937e401f8","html_url":"https://github.com/kayahr/datastream","commit_stats":{"total_commits":52,"total_committers":1,"mean_commits":52.0,"dds":0.0,"last_synced_commit":"98d2e14941a24507a6dbdb8184f8531dfbfa7d6d"},"previous_names":[],"tags_count":8,"template":false,"template_full_name":null,"purl":"pkg:github/kayahr/datastream","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/kayahr%2Fdatastream","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/kayahr%2Fdatastream/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/kayahr%2Fdatastream/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/kayahr%2Fdatastream/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/kayahr","download_url":"https://codeload.github.com/kayahr/datastream/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/kayahr%2Fdatastream/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":268171959,"owners_count":24207436,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-08-01T02:00:08.611Z","response_time":67,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["data","datastream","input","output","stream","typescript"],"created_at":"2024-12-07T12:15:29.697Z","updated_at":"2025-08-01T05:04:31.585Z","avatar_url":"https://github.com/kayahr.png","language":"TypeScript","funding_links":["https://github.com/sponsors/kayahr","https://paypal.me/kayaahr/"],"categories":[],"sub_categories":[],"readme":"Datastream\n==========\n\n[GitHub] | [NPM] | [API Doc]\n\nData stream classes for reading and writing all kinds of data types. For input and output the data stream classes uses very simple source and sink interfaces which are compatible to the readers and writers provided by [ReadableStream] and [WritableStream] from the [Streams API]. But this library also provides some easy to use stream implementations to use Node.js files and Byte Arrays as source/sink.\n\nThe following data types are currently supported:\n\n* Single bits\n* Arrays of bits\n* Unsigned and signed bytes\n* Unsigned and signed 16, 32 and 64 bit values (little and big endian)\n* Unsigned and signed byte arrays\n* Unsigned and signed 16, 32 and 64 bit arrays (little and big endian)\n* Fixed-length strings\n* Null-terminated strings\n* String lines (LF or CRLF terminated)\n\nSee [text-encodings] for a list of supported text encodings.\n\nUsage\n-----\n\nInstall the library as a dependency in your project:\n\n```\nnpm install @kayahr/datastream\n```\n\nAnd then use it like in this example:\n\n```typescript\nimport { DataReader, DataWriter, Uint8ArraySource, Uint8ArraySink } from \"@kayahr/datastream\";\n\nconst sink = new Uint8ArraySink();\nconst writer = new DataWriter(sink);\nawait writer.writeBit(1);\nawait writer.writeBit(2);\nawait writer.writeUint16(0x1234);\nawait writer.flush();\nconst data = sink.getData();\n\nconst source = new Uint8ArraySource(data);\nconst reader = new DataReader(source);\nconst bit1 = await reader.readBit();\nconst bit2 = await reader.readBit();\nconst u16 = await reader.readUint16();\n```\n\nInstead of simply using a Uint8Array as sink and source you can also read from streams and write to streams which is explained in the next sections.\n\nDataReaderSource\n----------------\n\nTo read data you first have to create a source. A source is simply an object providing the following method:\n\n```typescript\nread(): Promise\u003cReadableStreamReadResult\u003cUint8Array\u003e\u003e;\n```\n\nYou might recognize this signature as it is provided by a reader from a [ReadableStream]. So if you have a standard readable stream from the [Streams API] then you can simply use `stream.getReader()` as a data source (Remember to release the lock on the reader with `reader.releaseLock()` when you no longer need it).\n\nThe datastream library also provides a `FileInputStream` implementation for reading from Node.js files and a `Uint8ArraySource` class which can be used to read directly from a static byte array.\n\nDataReader\n----------\n\nTo read data from a stream create a new DataReader instance and pass the source to it.\n\n\n```typescript\nimport { DataReader } from \"@kayahr/datastream\";\n\nconst reader = new DataReader(source);\n```\n\nBy default the data reader uses the native endianness for reading multi-byte values and utf-8 encoding for reading strings. You can set a specific endianness and/or encoding as options:\n\n```typescript\nimport { Endianness } from \"@kayahr/datastream\";\n\nconst reader = new DataReader(source, {\n    endianness: Endianness.BIG,\n    encoding: \"utf-16be\"\n});\n```\n\n### Reading single values\n\nReading single bits, bytes or multi-byte values works like this:\n\n```typescript\nconst bit = await reader.readBit();\nconst u8 = await reader.readUint8();\nconst s8 = await reader.readInt8();\nconst u16 = await reader.readUint16();\nconst i16 = await reader.readInt16();\nconst u32 = await reader.readUint32();\nconst i32 = await reader.readInt32();\nconst u64 = await reader.readBigUint64();\nconst i64 = await reader.readBigInt64();\n```\n\nThese methods return `null` when the end of the stream is reached. The methods reading multi-byte values also have an endianness parameter which overwrites the endianness of the reader in case you have to read mixed-endianness data. Example:\n\n\n```typescript\nconst u32Big = await reader.readUint32(Endianness.BIG);\n```\n\n### Reading arrays\n\nReading arrays from the stream works like this. The following examples only show reading an unsigned byte array but the same methods are available for signed bytes and 16, 32 and 64 bit values:\n\n```typescript\nconst buffer = new Uint8Array(64);\nconst bytesRead = await reader.readUint8Array(buffer);\n```\n\nThe array methods return the number of read values (bytes in this case) which is 0 when end of stream has been reached.\n\nAll array methods accept a second option parameter with which you can define the offset within the output buffer, the number of values to read and (for multi-byte values) the endianness.\n\nThis example reads 8 signed big endian 32 bit values and writes it at the end (offset 2) of the provided buffer holding 10 values:\n\n```typescript\nconst buffer = new Int32Array(10);\nconst valuesRead = await reader.readInt32Array(buffer, {\n    offset: 2,\n    size: 8,\n    endianness: Endianness.BIG\n});\n```\n\n\n### Reading bit arrays\n\nReading a list of bits is a little bit different then the other array read methods. Instead of passing an array to fill you only specify the number of bits you want to read and you get a number array where each entry represents a single bit. When end-of-stream has been reached without reading any bits then the returned array is empty.\n\n```typescript\nconst bits = await this.readBits(10);\n```\n\n### Reading strings\n\nA fixed-length string can be read like this:\n\n```typescript\nconst text = await reader.readString(128);\n```\n\nThis reads up to 128 bytes from the stream (fewer if end of stream is reached) and converts it to a string. So depending on the text encoding the returned string might be smaller than the given byte size.\n\nAn empty string is returned when end of stream has been reached without reading any character.\n\nBy default UTF-8 encoding is used. You can specify a different one as second parameter if needed:\n\n```typescript\nconst text = await reader.readString(128, \"shift-jis\");\n```\n\nTo read a null-terminated string instead of a fixed-length string do this:\n\n```typescript\nconst text = await reader.readNullTerminatedString();\n```\n\nTo distinguish between end of stream and an empty string (terminated with a null byte) this method returns `null` when end of stream has been reached without reading any character.\n\nThe method supports various options:\n\n```typescript\nconst text = await reader.readNullTerminatedString({\n    // Optional encoding, default is UTF-8\n    encoding: \"shift-jis\",\n    // Optional initial capacity of byte buffer used for building the string, default is 1024\n    initialCapacity: 256,\n    // Optional maximum number of bytes to read, default is unlimited\n    maxBytes: 1024\n);\n```\n\nYou can also read LF or CRLF terminated lines of strings like this:\n\n```typescript\nconst line = await reader.readLine();\n```\n\nThe method returns `null` when end of stream is reached. So wrap it with a while loop if you want to read all lines until the end of stream. `readLine` supports the same options as the `readNullTerminatedString` method (`encoding`, `initialCapacity` and `maxBytes`) with one additional option to include the EOL characters (`\\n` or `\\r\\n`) at the end of the line in the returned string:\n\n```typescript\nconst lineWithEOL = await reader.readLine({ includeEOL: true });\n```\n\n### Reading from streams\n\nIf you work with a readable stream as input source then you have to release the lock on the stream reader acquired from the stream in addition to closing the stream. So you may end up with a nested try..finally structure like this:\n\n```typescript\nconst stream = new FileInputStream(filename);\ntry {\n    const streamReader = stream.getReader();\n    try {\n        const dataReader = new DataReader(streamReader);\n        // Read stuff from data reader\n    } finally {\n        streamReader.releaseLock();\n    }\n} finally {\n    await stream.close();\n}\n```\n\nYou can simplify this structure a little bit with the helper function `readDataFromStream` which creates the data\nreader and also releases the lock on the stream reader. But closing the stream is still your own responsibility:\n\n```typescript\nimport { readDataFromStream } from \"@kayahr/datastream\";\n\nconst stream = new FileInputStream(filename);\ntry {\n    await readDataFromStream(stream, async reader =\u003e {\n        // Read stuff from data reader\n    });\n} finally {\n    await stream.close();\n}\n```\n\n### Skipping data\n\nInstead of reading data you can also simply skip data. This is faster than reading/ignoring data because it fast-forwards through the current and subsequently read buffers without actually looking at the data.\n\n```typescript\nconst skippedBits = await reader.skipBits(30);\nawait fullySkippedBytes = await reader.skipBytes(5);\n```\n\n`skipBits` returns the number of actually skipped bits which may be lower than the requested number of bits when end-of-stream has been reached.\n\n`skipBytes` returns the number of full bytes that have been skipped (not counting partially read bytes). Again this can be lower than the requested number of bytes when end-of-stream has been reached.\n\n### Look-ahead\n\n`DataReader` supports look-ahead operations which remembers the current stream position and restores it after the read-ahead operation is finished.\n\nSome examples:\n\n```typescript\nconst nextLine = await reader.lookAhead(() =\u003e reader.readLine());\n```\n\n```typescript\nconst { foo, bar } = await reader.lookAhead(async () =\u003e {\n    const foo = await reader.readBit();\n    const bar = await reader.readUint8();\n    return { foo, bar };\n});\n```\n\nYou can perform any read operation and as many as you like inside the function passed to the `lookAhead` method. You can even nest a look-ahead inside the look-ahead. But keep in mind that reading large amount of data in a look-ahead results in buffers piling up in memory because they need to be recorded to be able to restore the previous stream position because it is not possible to seek in a stream. So keep your look-ahead operations short so ideally they are performed within the same buffer or simply the next one.\n\nA look-ahead function can also decide to commit a specific amount of bytes or bits as finally read. This is useful for example if you read ahead lets say 20 bytes to scan the content, then you see that it begins with 3 bytes you are looking for so you commit these 3 bytes so the stream pointer only rewinds by 17 bytes:\n\n```typescript\nconst foo = await reader.lookAhead(async commit =\u003e {\n    const string = await.readString(20);\n    if (string.startsWith(\"foo\")) {\n        commit(3);\n        return \"foo\";\n    } else {\n        return null;\n    }\n});\n```\n\nWhen you call `commit()` without parameters then all data read in the look-ahead operation is committed. If you want to commit a number of bits instead of bytes then pass `1` as second parameter. This second parameter defines the bits per value which defaults to 8. So `commit(12)` is the same as `commit(12, 8)` which commits twelve 8-bit values. `commit(12, 1)` commits twelve 1-bit values (12 bits).\n\nDataWriterSink\n--------------\n\nTo write data you first have to create a sink. A sink is simply an object providing the following method:\n\n```typescript\nwrite(chunk: Uint8Array): Promise\u003cvoid\u003e;\n```\n\nYou might recognize this signature as it is provided by a writer from a [WritableStream]. So if you have a standard writable stream from the [Streams API] then you can simply use `stream.getWriter()` as a data sink (Remember to release the lock on the writer with `writer.releaseLock()` when you no longer need it).\n\nThe datastream library also provides a `FileOutputStream` implementation for writing to Node.js files and a `Uint8ArraySink` class which can be used to write directly to a growing byte array.\n\nDataWriter\n----------\n\nTo write data to a stream create a new DataWriter instance and pass the sink to it.\n\n```typescript\nimport { DataWriter } from \"@kayahr/datastream\";\n\nconst writer = new DataWriter(sink);\n```\n\nBy default the data writer uses the native endianness for writing multi-byte values, UTF-8 encoding for writing strings and a buffer size of 64 KiB. You can set specific settings through options:\n\n```typescript\nimport { Endianness } from \"@kayahr/datastream\";\n\nconst writer = new DataWriter(sink, {\n    endianness: Endianness.BIG,\n    encoding: \"utf-16be\",\n    bufferSize: 8192\n});\n```\n\n### Writing values\n\nWriting single bits, bytes, multi-byte values, arrays and strings works like this:\n\n```typescript\nwriter.writeBit(1);\nwriter.writeUint8(5);\nwriter.writeInt8(-3);\nwriter.writeUint16(10000);\nwriter.writeInt16(-5000);\nwriter.writeUint32(5762874);\nwriter.writeInt32(-2357622);\nwriter.writeBigUint64(75721771n);\nwriter.writeBigInt64(-3247534n);\nwriter.writeUint8Array(new Uint8Array(values));\nwriter.writeUint8Array(new Int8Array(values));\nwriter.writeUint16Array(new Uint16Array(values));\nwriter.writeUInt16Array(new Int16Array(values));\nwriter.writeUint32Array(new Uint32Array(values));\nwriter.writeUInt32Array(new Int32Array(values));\nwriter.writeBigUint64Array(new BigUint64Array(values));\nwriter.writeBigInt64Array(new BigInt64Array(values));\nwriter.writeString(\"Foo\");\n```\n\nAll write methods synchronously fills the write buffer and flushes the buffer asynchronously when full.\n\nIn case you are finished writing to the writer you have to flush the buffer a last time and await the returned promise to ensure it is fully written to the sink:\n\n```typescript\nawait writer.flush();\n```\n\nWhen writing multi-bytes you can specify the endianness. If you don't do this then the global endianness of the writer is used which defaults to the native endianness. Example for specifying endianness:\n\n```typescript\nwriter.writeUint32(123456, Endianness.BIG)\n```\n\nWhen writing strings then you can specify the text encoding. If you don't do this then the global encoding of the writer is used which defaults to UTF-8. Example for specifying encoding:\n\n```typescript\nwriter.writeString(\"灯台もと暗し。\", \"Shift-JIS\")\n```\n\nIf you want to write null-terminated strings or lines then append `\"\\0\"` or EOF characters like `\"\\r\\n\"` yourself.\n\n### Text-Encodings\n\nThis project depends on the [text-encodings] project to support a lot of text encodings. But by default no encoding is loaded so only the default UTF-8 encoding is available. If you want to write strings in other encodings then you have to import the specific encodings or all encodings yourself:\n\n```typescript\nimport \"@kayahr/text-encoding/encodings/shift_jis\"; // Imports a specific text encoding\nimport \"@kayahr/text-encoding/encodings\";           // Imports all text encodings\n```\n\n### Writing to streams\n\nIf you work with a writable stream as output sink then you have to release the lock on the stream writer acquired from the stream in addition to closing the stream. So you may end up with a nested try..finally structure like this:\n\n```typescript\nconst stream = new FileOutputStream(filename);\ntry {\n    const streamWriter = stream.getWriter();\n    try {\n        const dataWriter = new DataWriter(streamWriter);\n        // Write stuff to data writer\n        await dataWriter.flush();\n    } finally {\n        streamWriter.releaseLock();\n    }\n} finally {\n    await stream.close();\n}\n```\n\nYou can simplify this structure a little bit with the helper function `writeDataToStream` which creates the data\nwriter and also releases the lock on the stream writer. But closing the stream is still your own responsibility:\n\n```typescript\nimport { writeDataToStream } from \"@kayahr/datastream\";\n\nconst stream = new FileOutputStream(filename);\ntry {\n    await writeDataToStream(stream, async writer =\u003e {\n        // Read stuff from data reader\n        await writer.flush();\n    });\n} finally {\n    await stream.close();\n}\n```\n\n[API Doc]: https://kayahr.github.io/datastream/\n[GitHub]: https://github.com/kayahr/datastream\n[NPM]: https://www.npmjs.com/package/@kayahr/datastream\n[Streams API]: https://developer.mozilla.org/en-US/docs/Web/API/Streams_API\n[ReadableStream]: https://developer.mozilla.org/en-US/docs/Web/API/ReadableStream\n[WritableStream]: https://developer.mozilla.org/en-US/docs/Web/API/WritableStream\n[Encoding API]: https://developer.mozilla.org/en-US/docs/Web/API/Encoding_API\n[encodings]: https://developer.mozilla.org/en-US/docs/Web/API/TextDecoder/encoding\n[text-encodings]: https://www.npmjs.com/package/@kayahr/text-encoding\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fkayahr%2Fdatastream","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fkayahr%2Fdatastream","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fkayahr%2Fdatastream/lists"}