{"id":23246405,"url":"https://github.com/marianmeres/unchunk","last_synced_at":"2025-06-24T00:33:50.237Z","repository":{"id":232673469,"uuid":"784862832","full_name":"marianmeres/unchunk","owner":"marianmeres","description":"Utility for reconstructing chunks into messages.","archived":false,"fork":false,"pushed_at":"2024-04-15T13:19:43.000Z","size":119,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":2,"default_branch":"master","last_synced_at":"2025-06-06T08:05:08.843Z","etag":null,"topics":["parser"],"latest_commit_sha":null,"homepage":"","language":"JavaScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/marianmeres.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-04-10T17:53:02.000Z","updated_at":"2024-04-10T19:52:58.000Z","dependencies_parsed_at":null,"dependency_job_id":"0113e87e-3d05-422e-a7ce-d0fb7de13e8c","html_url":"https://github.com/marianmeres/unchunk","commit_stats":null,"previous_names":["marianmeres/unchunk"],"tags_count":4,"template":false,"template_full_name":null,"purl":"pkg:github/marianmeres/unchunk","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/marianmeres%2Funchunk","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/marianmeres%2Funchunk/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/marianmeres%2Funchunk/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/marianmeres%2Funchunk/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/marianmeres","download_url":"https://codeload.github.com/marianmeres/unchunk/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/marianmeres%2Funchunk/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":261582979,"owners_count":23180639,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["parser"],"created_at":"2024-12-19T07:14:50.204Z","updated_at":"2025-06-24T00:33:50.223Z","avatar_url":"https://github.com/marianmeres.png","language":"JavaScript","funding_links":[],"categories":[],"sub_categories":[],"readme":"# @marianmeres/unchunk\n\nUtility for reconstructing chunks into messages.\n\n## Use case\n\nA server is streaming **multiple** messages (eg `csv` rows, `json` objects...) via `Transfer-Encoding: 'chunked'`...\n\nWhile the client's [`reader.read`](https://developer.mozilla.org/en-US/docs/Web/API/Streams_API/Using_readable_streams) helps us glue the low level transport chunks, we still need to restore the higher level messages ASAP while the chunks are still incoming. This is where this tool comes in handy.\n\n## Delimiter\n\nEach message (not chunk) must have a _delimiter_ appended at the end for this to work. And of course, both sides (client and server) must use the same _delimiter_.\n\nYou can use any _delimiter_ (eg `,`, `\\n`, `\\r\\n`, `\\n\\n`, `💩`, ...) if you are sure that it does not conflict with the message. By default, this utility uses the single [ASCII Record Separator](https://en.wikipedia.org/wiki/C0_and_C1_control_codes#Field_separators) char `\\x1E`, which should be safe for the majority of use cases.\n\n## Usage\n\nInstallation:\n\n```bash\nnpm i @marianmeres/unchunk\n```\n\nSignature:\n\n```typescript\ncreateUnchunk(\n    onMessage: (message: string) =\u003e void,\n    messageDelimiter: string = createUnchunk.DELIMITER\n): (chunk: string) =\u003e void;\n```\n\nIllustration example:\n\n```javascript\nimport { createUnchunk } from '@marianmeres/unchunk';\n\nconst messages = [];\n\n// let's create the worker function with a custom \"\\n\\n\" delimiter\nconst unchunk = createUnchunk((message) =\u003e {\n\t// in the real world, we would probably be updating the UI right now...\n\tmessages.push(message);\n}, '\\n\\n');\n\n// Note that the delimiter is split into multiple chunks here for the real\n// world illustration. Also note, that in the real world, the chunks\n// are streaming in and are not available in advance - which makes this\n// example kind of stupid where one could just simply use:\n// const messages = chunks.join('').split(delimiter)\nconst chunks = ['1', '2\\n', '\\n3', '4', '\\n', '\\n56\\n\\n'];\n\n// actual work\nchunks.forEach(unchunk);\n\n// we have successfully reconstructed 3 messages\nassert(messages.join() === '12,34,56');\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmarianmeres%2Funchunk","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fmarianmeres%2Funchunk","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmarianmeres%2Funchunk/lists"}