{"id":20563840,"url":"https://github.com/worker-tools/json-stream","last_synced_at":"2025-04-14T14:53:23.755Z","repository":{"id":50417347,"uuid":"489034323","full_name":"worker-tools/json-stream","owner":"worker-tools","description":"Utilities for working with streaming JSON in Worker Environments.","archived":false,"fork":false,"pushed_at":"2022-05-27T11:08:19.000Z","size":125,"stargazers_count":13,"open_issues_count":1,"forks_count":1,"subscribers_count":1,"default_branch":"master","last_synced_at":"2025-03-23T10:41:42.136Z","etag":null,"topics":["cloudflare-workers","deno","javascript","json","parser","service-workers","streams","stringify","web-streams","whatwg-streams"],"latest_commit_sha":null,"homepage":"https://workers.tools/json-stream","language":"TypeScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/worker-tools.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE.md","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2022-05-05T15:47:17.000Z","updated_at":"2025-02-12T15:10:13.000Z","dependencies_parsed_at":"2022-09-10T18:21:34.665Z","dependency_job_id":null,"html_url":"https://github.com/worker-tools/json-stream","commit_stats":null,"previous_names":[],"tags_count":13,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/worker-tools%2Fjson-stream","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/worker-tools%2Fjson-stream/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/worker-tools%2Fjson-stream/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/worker-tools%2Fjson-stream/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/worker-tools","download_url":"https://codeload.github.com/worker-tools/json-stream/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248901198,"owners_count":21180368,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["cloudflare-workers","deno","javascript","json","parser","service-workers","streams","stringify","web-streams","whatwg-streams"],"created_at":"2024-11-16T04:21:23.877Z","updated_at":"2025-04-14T14:53:23.713Z","avatar_url":"https://github.com/worker-tools.png","language":"TypeScript","readme":"# JSON Stream\n\nUtilities for working with streaming JSON in Worker Runtimes such as Cloudflare Workers, Deno Deploy and Service Workers.\n\n***\n\n__Work in Progress__: See TODOs \u0026 Deprecations\n\n***\n\n## Base Case\nThe most basic use case is turning a stream of objects into stream of strings that can be sent over the wire.\nOn the other end, it can be turned back into a stream of JSON objects. \nFor this `JSONStringifyStream` and `JSONParseStream` are all that is required. \nThey work practically the same as `TextEncoderStream` and `TextDecoderStream`:\n\n```js\nconst items = [\n  { a: 1 }, \n  { b: 2}, \n  { c: 3 }, \n  'foo', \n  { a: { nested: { object: true }} }\n];\nconst stream = toReadableStream(items)\n  .pipeThrough(new JSONStringifyStream())\n  .pipeThrough(new TextEncoderStream())\n\n// Usage e.g.:\nawait fetch('/endpoint.json', { \n  body: stream, \n  method: 'POST', \n  headers: [['content-type', 'application/json']] \n})\n\n// On the server side:\nconst collected = [];\nawait stream\n  .pipeThrough(new JSONParseStream())\n  .pipeTo(new WritableStream({ write(obj) { collected.push(obj) }}))\n\nassertEquals(items, collected)\n```\n\nNote that standard JSON is used as the transport format. Unlike ND-JSON, \nneither side needs to opt-in using the streaming parser/stringifier to accept data. \nFor example this is just as valid:\n\n```js\nconst collected = await new Response(stream).json()\n```\n\n~~If on the other hand ND-JSON is sufficient for your use case, this module also provides `NDJSONStringifyStream` and `NDJSONParseStream` that work the same way as shown above, but lack the following features.~~ (TODO: make separate module?)\n\n## Using JSON Path to locate nested data \n__JSON Stream__ also supports more complex use cases. Assume JSON of the following structure:\n\n```jsonc\n// filename: \"nested.json\"\n{\n  \"type\": \"foo\",\n  \"items\": [\n    { \"a\": 1 }, \n    { \"b\": 2 }, \n    { \"c\": 3 }, \n    // ...\n    { \"zzz\": 999 }, \n  ]\n}\n```\n\nHere, the example code from above wouldn't work (or at least not as you would expect), \nbecause by default `JSONParseStream` emits the objects that are the immediate children of the root object. \nHowever, the constructor accepts a JSONPath-like string to locate the desired data to parse:\n\n```js\nconst collected = [];\nawait (await fetch('/nested.json')).body\n  .pipeThrough(new JSONParseStream('$.items.*')) // \u003c-- new\n  .pipeTo(new WritableStream({ write(obj) { collected.push(obj) }}))\n```\n\nIt's important to add the `.*` at the end, otherwise the entire items array will arrive in a singe call once it is fully parsed.\n\n`JSONParseStream` only supports a subset of JSONPath, specifically eval (`@`) expressions and negative slices are omitted.\nBelow is a table showing some examples:\n\n| JSONPath                  | Description                                                 |\n|:--------------------------|:------------------------------------------------------------|\n| `$.*`                     | All direct children of the root. Default.                   |\n| `$.store.book[*].author`  | The authors of all books in the store                       |\n| `$..author`               | All authors                                                 |\n| `$.store.*`               | All things in store, which are some books and a red bicycle |\n| `$.store..price`          | The price of everything in the store                        |\n| `$..book[2]`              | The third book                                              |\n| `$..book[0,1]`            | The first two books via subscript union                     |\n| `$..book[:2]`             | The first two books via subscript array slice               |\n| `$..*`                    | All members of JSON structure                               |\n\n## Streaming Complex Data\nYou might also be interested in how to stream complex data such as the one above from memory.\nIn that case `JSONStringifyStream` isn't too helpful, as it only supports JSON arrays (i.e. the root element is an array `[]`). \n\nFor that case __JSON Stream__ provides the `jsonStringifyStream` method (TODO: better name to indicate that it is a readableStream? Change to ReadableStream subclass? Export `JSONStream` object with `stringify` method?) which accepts any JSON-ifiable data as argument. It is mostly compatible with `JSON.stringify` (TODO: replacer \u0026 spaces), but with the important exception that it \"inlines\" any `Promise`, `ReadableStream` and `AsyncIterable` it encounters. Again, an example:\n\n```js\nconst stream = jsonStringifyStream({\n  type: Promise.resolve('foo'),\n  items: (async function* () {\n    yield { a: 1 } \n    yield { b: 2 } \n    yield { c: 3 } \n    // Can also have nested async values:\n    yield Promise.resolve({ zzz: 999 })\n  })(),\n})\n\nnew Response(stream.pipeThrough(new TextEncoderStream()), {\n  headers: [['content-type', 'application/json']] \n})\n```\n\nInspecting this on the network would show the following (where every newline is a chunk):\n```json\n{\n\"type\":\n\"foo\"\n,\n\"items\":\n[\n{\n\"a\":\n1\n}\n,\n{\n\"b\":\n2\n}\n,\n{\n\"c\":\n3\n}\n,\n{\n\"zzz\":\n999\n}\n]\n}\n```\n\n## Retrieving Complex Structures\nBy providing a JSON Path to `JSONParseStream` we can stream the values of a single, nested array. \nFor scenarios where the JSON structure is more complex, there is the `JSONParseNexus` (TODO: better name) class. \nIt provides promise and and stream-based methods that accept JSON paths to retrieve one or multiple values respectively. \nWhile it is much more powerful and can restore arbitrary complex structures, it is also more difficult to use.\n\nIt's best to explain by example. Assuming the data structure from above, we have:\n\n```js\nconst parser = new JSONParseNexus();\nconst data = {\n  type: parser.promise('$.type'),\n  items: parser.stream('$.items.*'),\n}\n(await fetch('/nested.json').body)\n  .pipeThrough(parser)  // \u003c-- new\n\nassertEquals(await data.type, 'foo')\n\n// We can collect the values as before:\nconst collected = [];\nawait data.items\n  .pipeTo(new WritableStream({ write(obj) { collected.push(obj) }}))\n```\n\nWhile this works just fine, it becomes more complicated when there are multiple streams and values involved.\n\n### Managing Internal Queues\nIt's important to understand that `JSONParseNexus` provides mostly pull-based APIs.\nIn the cause of `.stream()` and `.iterable()` no work is being done until a consumer requests a value by calling `.read()` or `.next()` respectively.\nHowever, once a value is requested, `JSONParseNexus` will parse values until the requested JSON path is found. \nAlong the way it will fill up queues for any other requested JSON paths it encounters.\nThis means that memory usage can grow arbitrarily large unless the data is processed in the order it was stringified:\nTake for example the following structure:\n\n```js\nconst parser = new JSONParseNexus();\n\njsonStringifyStream({\n  xs: new Array(10_000).fill({ x: 'x' }),\n  ys: new Array(10_000).fill({ y: 'y' }),\n}).pipeThrough(parser)\n\nfor await (const y of parser.iterable('$.ys.*')) console.log(y)\nfor await (const x of parser.iterable('$.xs.*')) console.log(x)\n```\n\nIn this examples Ys are being processed before Xs, but were stringified in the opposite order. \nThis means the internal queue of Xs grows to 10.000 before it is being processed by the second loop. \nThis can be avoided by changing the order to match the stringification order.\n\n### Single Values and Lazy Promises\nSpecial attention has to be given single values, as Promises in JS are eager by default and have no concept of \"pulling\" data. \n`JSONParseNexus` introduces a lazy promise type that has a different behavior. \nAs with async iterables and streams provided by `.iterable` and `.stream`, it does not pull values form the underlying readable until requested. This happens when `await`ing the promise, i.e. is calling the `.then` instance method, otherwise it stays idle.\n\n```js\nconst parser = new JSONParseNexus();\n\njsonStringifyStream({\n  type: 'items',\n  items: new Array(10_000).fill({ x: 'x' }),\n  trailer: 'trail',\n}).pipeThrough(parser)\n\nconst data = {\n  type: await parser.promise('$.type') // ok\n  items: parser.iterable('$.items.*')\n  trailer: parser.promise('$.trailer') // do not await!\n}\n\nconsole.log(data.type) //=\u003e 'items'\n\n// Now async iteration is in control of parser:\nfor await (const x of data.items) {\n  console.log(x)\n}\n// Now we can await the trailer:\nconsole.log(await data.trailer)\n```\n\nIn the above example, without lazy promises `ctrl.promise('$.trailer')` would immediately parse the entire JSON structure, which involves filling a queue of 10.000 elements.\n\nIn order to transform value without triggering executions, \nthe class provides a `.map` function that works similar to JS arrays:\n\n```js\nconst trailer = ctrl.promise('$.trailer').map(x =\u003e x.toUpperCase())\n```\n\n## Limitations\n**JSON Stream** largely consists of old Node libraries that have been modified to work in Worker Runtimes and the browser. \nCurrently they are not \"integrated\", for example specifying a specific JSON Path does not limit the amount of parsing the parser does.\n\nThe stringification implementation, which is original, relies heavily on async generators, which are \"slow\" but they made the implementation easy and quick to implement.\n\n**JSON Stream** heavily relies on [`TransformStream`](https://developer.mozilla.org/en-US/docs/Web/API/TransformStream), which has only recently shipped in Chrome \u0026 Safari and is still behind a flag in Firefox. However, the latest version of Deno and Cloudflare Workers support it (might require compatibility flags in CF Workers).\n\n\n## Appendix\n### To ReadableStream Function\nAn example above uses a `toReadableStream` function, which can be implemented as follows:\n```ts\nfunction toReadableStream\u003cT\u003e(iter: Iterable\u003cT\u003e) {\n  const xs = [...iter];\n  let x: T | undefined;\n  return new ReadableStream\u003cT\u003e({\n    pull(ctrl) { \n      if (x = xs.shift()) ctrl.enqueue(x); else ctrl.close();\n    },\n  });\n}\n```\n\n--------\n\n\u003cp align=\"center\"\u003e\u003ca href=\"https://workers.tools\"\u003e\u003cimg src=\"https://workers.tools/assets/img/logo.svg\" width=\"100\" height=\"100\" /\u003e\u003c/a\u003e\n\u003cp align=\"center\"\u003eThis module is part of the Worker Tools collection\u003cbr/\u003e⁕\n\n[Worker Tools](https://workers.tools) are a collection of TypeScript libraries for writing web servers in [Worker Runtimes](https://workers.js.org) such as Cloudflare Workers, Deno Deploy and Service Workers in the browser. \n\nIf you liked this module, you might also like:\n\n- 🧭 [__Worker Router__][router] --- Complete routing solution that works across CF Workers, Deno and Service Workers\n- 🔋 [__Worker Middleware__][middleware] --- A suite of standalone HTTP server-side middleware with TypeScript support\n- 📄 [__Worker HTML__][html] --- HTML templating and streaming response library\n- 📦 [__Storage Area__][kv-storage] --- Key-value store abstraction across [Cloudflare KV][cloudflare-kv-storage], [Deno][deno-kv-storage] and browsers.\n- 🆗 [__Response Creators__][response-creators] --- Factory functions for responses with pre-filled status and status text\n- 🎏 [__Stream Response__][stream-response] --- Use async generators to build streaming responses for SSE, etc...\n- 🥏 [__JSON Fetch__][json-fetch] --- Drop-in replacements for Fetch API classes with first class support for JSON.\n- 🦑 [__JSON Stream__][json-stream] --- Streaming JSON parser/stingifier with first class support for web streams.\n\nWorker Tools also includes a number of polyfills that help bridge the gap between Worker Runtimes:\n- ✏️ [__HTML Rewriter__][html-rewriter] --- Cloudflare's HTML Rewriter for use in Deno, browsers, etc...\n- 📍 [__Location Polyfill__][location-polyfill] --- A `Location` polyfill for Cloudflare Workers.\n- 🦕 [__Deno Fetch Event Adapter__][deno-fetch-event-adapter] --- Dispatches global `fetch` events using Deno’s native HTTP server.\n\n[router]: https://workers.tools/router\n[middleware]: https://workers.tools/middleware\n[html]: https://workers.tools/html\n[kv-storage]: https://workers.tools/kv-storage\n[cloudflare-kv-storage]: https://workers.tools/cloudflare-kv-storage\n[deno-kv-storage]: https://workers.tools/deno-kv-storage\n[kv-storage-polyfill]: https://workers.tools/kv-storage-polyfill\n[response-creators]: https://workers.tools/response-creators\n[stream-response]: https://workers.tools/stream-response\n[json-fetch]: https://workers.tools/json-fetch\n[json-stream]: https://workers.tools/json-stream\n[request-cookie-store]: https://workers.tools/request-cookie-store\n[extendable-promise]: https://workers.tools/extendable-promise\n[html-rewriter]: https://workers.tools/html-rewriter\n[location-polyfill]: https://workers.tools/location-polyfill\n[deno-fetch-event-adapter]: https://workers.tools/deno-fetch-event-adapter\n\nFore more visit [workers.tools](https://workers.tools).\n","funding_links":[],"categories":[],"sub_categories":[],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fworker-tools%2Fjson-stream","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fworker-tools%2Fjson-stream","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fworker-tools%2Fjson-stream/lists"}