{"id":16821458,"url":"https://github.com/strml/parse-stream","last_synced_at":"2025-04-05T10:12:44.563Z","repository":{"id":31889942,"uuid":"130407969","full_name":"STRML/parse-stream","owner":"STRML","description":"Parse streams of binary data of arbitrary lengths, handling broken/incomplete chunks.","archived":false,"fork":false,"pushed_at":"2023-01-09T16:24:13.000Z","size":118,"stargazers_count":1,"open_issues_count":2,"forks_count":0,"subscribers_count":2,"default_branch":"master","last_synced_at":"2025-03-29T09:36:12.523Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"JavaScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/STRML.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2018-04-20T19:55:18.000Z","updated_at":"2022-11-22T15:14:45.000Z","dependencies_parsed_at":"2023-01-14T20:01:52.325Z","dependency_job_id":null,"html_url":"https://github.com/STRML/parse-stream","commit_stats":null,"previous_names":[],"tags_count":2,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/STRML%2Fparse-stream","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/STRML%2Fparse-stream/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/STRML%2Fparse-stream/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/STRML%2Fparse-stream/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/STRML","download_url":"https://codeload.github.com/STRML/parse-stream/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":247318746,"owners_count":20919484,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-10-13T11:00:15.161Z","updated_at":"2025-04-05T10:12:44.518Z","avatar_url":"https://github.com/STRML.png","language":"JavaScript","funding_links":[],"categories":[],"sub_categories":[],"readme":"# parse-stream\n\nParse streams of binary data of arbitrary lengths, handling broken/incomplete chunks.\n\nThis is useful when procesisng binary data that can be chunked in any way. For example, imagine we're handling some arbitrary IPC format we'll call \"jsonIPC\" through a `net.Socket`.\n\n\"jsonIPC\" is a fake, simple data format that encodes the length of the JSON string as a 32-bit little-endian uint before the JSON string. By default, `net.Socket` may emit 8192 byte chunks. These chunks may contain multiple messages, may be smaller than 8192 bytes, or contain part of a larger message. To illustrate, they may look like this, with `|` indicating a break in chunks:\n\n```\n[len32, ...message], [len32, ...message], [len32, ...mes | sage], [len32, ...message]\n```\n\nBy defining how to get the length of each message from a stream of binary data, `ParseStream` takes care of splitting chunks properly, dealing with:\n\n* Chunks that contain multiple messages\n* Chunks that contain partial messages (e.g. 8192 byte chunks, 1MB message)\n* Chunks that don't contain enough data to even parse the length\n  - Return `Infinity` from `getDataGramLength()` and a larger chunk will be passed back on the next invocation.\n\n\n### Usage\n\n\u003e Notice! Version 2.0 no longer has the `parseDataGram` function, and does not emit `'chunkLen'` anymore.\n\u003e Simply pipe your `ParseStream` into another transform stream to replicate the old behavior.\n\n```js\nconst ParseStream = require('../dist/index.js');\nconst {Transform} = require('stream');\n\n// Get a socket from somewhere\nconst sock = new require('stream').PassThrough();\n\n// Pipe through a ParseStream.\nsock.pipe(\n  new ParseStream({\n    // This is used to slice up buffers. Knowing your data format, return the\n    // length of the message you expect to parse.\n    // IMPORTANT: You may get a buffer of *any length*! Use Infinity as a\n    // sentinel value to tell ParseStream to get another chunk.\n    getDataGramLength(buf) {\n      if (buf.length \u003c 4) return Infinity;\n      return 4 + buf.readUInt32LE(0);\n    },\n  })\n).pipe(\n  new Transform({\n    // Once you have the full datagram, you might want to parse it.\n    //\n    // This defines the transformation from raw buffer data to any type.\n    // The length of the buffer you are passed is defined by getDataGramLength().\n    transform(chunk, encoding, callback) {\n      // Slice off first 4 which is length\n      callback(null, JSON.parse(chunk.slice(4).toString('utf8')));\n    },\n    readableObjectMode: true,\n  })\n).on('data', function(result/*: Object */) {\n  console.log(result, typeof result);\n});\n\nconst testData = JSON.stringify({foo: 'bar', biff: [1,2,3]});\nconst testBuf = Buffer.alloc(4 + testData.length);\ntestBuf.writeUInt32LE(Buffer.byteLength(testData), 0);\ntestBuf.write(testData, 4);\n\nsock.write(testBuf);\n// Logs: \"{foo: 'bar', biff: [1,2,3]}, 'object'\"\n\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fstrml%2Fparse-stream","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fstrml%2Fparse-stream","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fstrml%2Fparse-stream/lists"}