{"id":50419813,"url":"https://github.com/lcweden/jsontext","last_synced_at":"2026-05-31T08:00:53.392Z","repository":{"id":358712797,"uuid":"1239707907","full_name":"lcweden/jsontext","owner":"lcweden","description":"A state machine for incremental JSON processing.","archived":false,"fork":false,"pushed_at":"2026-05-28T15:50:53.000Z","size":142,"stargazers_count":2,"open_issues_count":0,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2026-05-28T16:24:50.998Z","etag":null,"topics":["json","json-parser","json-path","json-pointer","jsonl","jsontext","parser","state-machine","stream","streaming","web-streams"],"latest_commit_sha":null,"homepage":"","language":"TypeScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/lcweden.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2026-05-15T11:06:47.000Z","updated_at":"2026-05-28T15:49:47.000Z","dependencies_parsed_at":null,"dependency_job_id":null,"html_url":"https://github.com/lcweden/jsontext","commit_stats":null,"previous_names":["lcweden/jsontext"],"tags_count":2,"template":false,"template_full_name":null,"purl":"pkg:github/lcweden/jsontext","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lcweden%2Fjsontext","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lcweden%2Fjsontext/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lcweden%2Fjsontext/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lcweden%2Fjsontext/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/lcweden","download_url":"https://codeload.github.com/lcweden/jsontext/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lcweden%2Fjsontext/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":33723549,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-26T15:22:16.424Z","status":"online","status_checked_at":"2026-05-31T02:00:06.040Z","response_time":95,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["json","json-parser","json-path","json-pointer","jsonl","jsontext","parser","state-machine","stream","streaming","web-streams"],"created_at":"2026-05-31T08:00:52.660Z","updated_at":"2026-05-31T08:00:53.373Z","avatar_url":"https://github.com/lcweden.png","language":"TypeScript","funding_links":[],"categories":[],"sub_categories":[],"readme":"# JSONText\n\n[![license](https://img.shields.io/github/license/lcweden/jsontext.svg)](LICENSE)\n[![npm version](https://img.shields.io/npm/v/jsontext.svg)](https://www.npmjs.com/package/jsontext)\n[![jsr version](https://img.shields.io/jsr/v/@lcweden/jsontext)](https://jsr.io/@lcweden/jsontext)\n\nA state machine for incremental JSON processing.\n\n## Quick Start\n\nThe following example demonstrates how to use `JSONTextSelectorStream` to extract all `address` from\na JSON fetched from [DummyJSON](https://dummyjson.com/).\n\n```javascript\nimport { JSONTextSelectorStream } from \"jsontext\";\n\nconst response = await fetch(\"https://dummyjson.com/users\");\nconst addresses = response.body.pipeThrough(new JSONTextSelectorStream(\"$.users[*].address\"));\n\nfor await (const value of addresses) {\n  console.log(value.json());\n}\n```\n\n## Installation\n\n`jsontext` is an ESM-only package available on both `NPM` and `JSR`. The core decoder and encoder\nrun in any modern JavaScript environment; the optional `*Stream` classes additionally require\n`WHATWG` Streams support:\n\n### NPM\n\nInstall via [npm](https://www.npmjs.com/package/jsontext):\n\n```bash\nnpm install jsontext\n```\n\n### Deno\n\nInstall via [JSR](https://jsr.io/@lcweden/jsontext):\n\n```bash\ndeno add jsr:@lcweden/jsontext\n```\n\n## APIs\n\nSee full reference on [JSR](https://jsr.io/@lcweden/jsontext/doc/).\n\n| Category   | Exports                                                                                                  |\n| :--------- | :------------------------------------------------------------------------------------------------------- |\n| Core       | [`JSONTextDecoder`], [`JSONTextEncoder`]                                                                 |\n| Stream     | [`JSONTextDecoderStream`], [`JSONTextEncoderStream`], [`JSONTextSelectorStream`], [`JSONTextLineStream`] |\n| Components | [`Token`], [`Value`], [`Kind`]                                                                           |\n| Error      | [`SyntacticError`]                                                                                       |\n\n[`JSONTextDecoder`]: https://jsr.io/@lcweden/jsontext/doc/~/JSONTextDecoder\n[`JSONTextEncoder`]: https://jsr.io/@lcweden/jsontext/doc/~/JSONTextEncoder\n[`JSONTextDecoderStream`]: https://jsr.io/@lcweden/jsontext/doc/~/JSONTextDecoderStream\n[`JSONTextEncoderStream`]: https://jsr.io/@lcweden/jsontext/doc/~/JSONTextEncoderStream\n[`JSONTextSelectorStream`]: https://jsr.io/@lcweden/jsontext/doc/~/JSONTextSelectorStream\n[`JSONTextLineStream`]: https://jsr.io/@lcweden/jsontext/doc/~/JSONTextLineStream\n[`Token`]: https://jsr.io/@lcweden/jsontext/doc/~/Token\n[`Value`]: https://jsr.io/@lcweden/jsontext/doc/~/Value\n[`Kind`]: https://jsr.io/@lcweden/jsontext/doc/~/KIND\n[`SyntacticError`]: https://jsr.io/@lcweden/jsontext/doc/~/SyntacticError\n\n### Core\n\nThe core APIs provide more control and flexibility. They are designed for scenarios where Web\nStreams are not available or when you need granular control.\n\n#### JSONTextDecoder\n\nA low-level, stateful JSON decoder that processes bytes incrementally. It is suitable for developing\ncustom JSON processing logic and `TransformStreams`.\n\nUnlike `JSON.parse`, you need to `.push()` bytes into `JSONTextDecoder` as they arrive, and pull\n`Tokens` or `Values`.\n\n##### Basic Usage\n\nThe following example demonstrates how to `.push()` bytes into `JSONTextDecoder` and read tokens one\nby one. The decoder automatically buffers incomplete tokens across bytes.\n\n```javascript\nconst decoder = new JSONTextDecoder();\n\ndecoder.push(new TextEncoder().encode(`{\"name\": \"Al`));\ndecoder.push(new TextEncoder().encode(`ice\", \"age\": 18`));\ndecoder.push(new TextEncoder().encode(`}`));\n\ndecoder.end(); // no more bytes are coming, signal the end of input\n\ndecoder.readToken().kind; // KIND.OBJECT_BEGIN  ('{')\ndecoder.readToken().asString(); // \"name\"\ndecoder.readToken().asString(); // \"Alice\"\ndecoder.readToken().asString(); // \"age\"\ndecoder.readToken().asNumber(); // 18\ndecoder.readToken().kind; // KIND.OBJECT_END    ('}')\n\ndecoder.checkEOF();\n```\n\nYou may want to check the type before parsing a `Token`. `KIND` is a constant enum that can be used\nlike this: `token.kind === KIND.STRING` or `token.kind === KIND.BOOLEAN`.\n\n\u003e [!TIP]\n\u003e `.end()` signals that no more bytes will be pushed. The decoder needs this signal to confirm that\n\u003e a number at the very end of the stream is complete and not just more digits still coming, since\n\u003e there is no delimiter after it. Always call `.end()` when you know the input is done.\n\n\u003e [!TIP]\n\u003e `checkEOF()` asserts that the entire input was consumed and well-formed, no unclosed objects or\n\u003e trailing garbage bytes.\n\n##### Extracting and Skipping Values\n\nOther than reading tokens one by one, you can also read a `Value` with `.readValue()`, which can be\na scalar, an entire object, or an array.\n\n```javascript\nconst decoder = new JSONTextDecoder(new TextEncoder().encode(`{\"id\": 1, \"metadata\": {  }}`));\nlet token;\n\nwhile (true) {\n  token = decoder.readToken();\n\n  if (token === undefined) {\n    break; // need more bytes\n  }\n\n  if (token.asString() === \"metadata\") {\n    const value = decoder.readValue();\n    const metadata = value.json();\n  } else {\n    decoder.skipValue(); // skip the value of this token without parsing it\n  }\n}\n\ndecoder.end();\ndecoder.checkEOF();\n```\n\nThe example above follows the sequence:\n\n| step | action                      |\n| :--- | :-------------------------- |\n| 1    | read a token (`\"id\"`)       |\n| 2    | skip the value (`1`)        |\n| 3    | read a token (`\"metadata\"`) |\n| 4    | read a value (`{ }`)        |\n| 5    | parse the value as JSON     |\n\n\u003e [!TIP]\n\u003e Use `.stackPointer()` to get the [JSON Pointer](https://datatracker.ietf.org/doc/html/rfc6901),\n\u003e which is useful for targeting specific paths in the document like\n\u003e `decoder.stackPointer() === \"/metadata\"`.\n\n##### Web Streams\n\nThe following example demonstrates how to use `JSONTextDecoder` with a `ReadableStream` from\n`fetch`.\n\n```javascript\nconst response = await fetch(\"your.api/endpoint\");\nconst decoder = new JSONTextDecoder();\n\n// Outer loop: wait for new chunks to arrive\nfor await (const chunk of response.body) {\n  decoder.push(chunk);\n\n  // Inner loop: read all decodable tokens from the current buffer\n  for (let token; (token = decoder.readToken()) !== undefined;) {\n    // read token or value...\n  }\n}\n\ndecoder.end();\ndecoder.checkEOF();\n```\n\nRequires the user to manage backpressure and chunk boundaries, it gives you the most control and\nflexibility. Check `JSONTextDecoderStream` to see how to wrap it in a `TransformStream` that handles\nall the stream mechanics for you.\n\n#### JSONTextEncoder\n\nIt is the exact counterpart to `JSONTextDecoder`, which allows you to construct a JSON document\ntoken by token or value.\n\n##### Basic Usage\n\nYou can feel free to write tokens and values in any order using `Token` and `Value` provided\nmethods.\n\n```javascript\nimport { Token, Value } from \"jsontext\";\n\nconst decoder = new TextDecoder();\nconst encoder = new JSONTextEncoder();\n\nencoder.writeToken(Token.ARRAY_BEGIN);\nencoder.writeValue(Value.from({ id: 1, status: \"active\" }));\nencoder.writeValue(Value.from({ id: 2, status: \"pending\" }));\nencoder.writeToken(Token.ARRAY_END);\n\nconst bytes = encoder.takeBytes();\nconst text = decoder.decode(bytes);\n// '[{\"id\":1,\"status\":\"active\"},{\"id\":2,\"status\":\"pending\"}]'\n```\n\n##### Round Trip\n\nA common use case is piping a decoder directly into an encoder to mutate a stream on the fly. In\nthis pattern, you drain tokens from the decoder, modify them if needed, and write them to the\nencoder.\n\n```javascript\nconst decoder = new JSONTextDecoder();\nconst encoder = new JSONTextEncoder();\n\nconst response = await fetch(\"your.api/endpoint\");\n\nfor await (const chunk of response.body) {\n  decoder.push(chunk);\n\n  for (let token; (token = decoder.readToken()) !== undefined;) {\n    encoder.writeToken(token);\n  }\n\n  const bytes = encoder.takeBytes();\n}\n\ndecoder.end();\ndecoder.checkEOF();\n```\n\n\u003e [!IMPORTANT]\n\u003e `takeBytes()` only gives you the encoded bytes and clears the encoder's internal buffer. It does\n\u003e not write them anywhere. You must manually pipe these bytes to your destination, such as a file\n\u003e writer, network socket, or controller.\n\n### Stream\n\nThese classes wrap the core decoder and encoder in `TransformStream` interfaces, making them easy to\nhandle some common use cases and compose with other Web Streams APIs. See the [Examples](#examples)\nsection for more details.\n\n#### JSONTextDecoderStream\n\nWraps a `JSONTextDecoder` and emits `Token`s as they are decoded. Ideal for token-level processing,\nsuch as filtering or transforming tokens. If you need to work with `Value`, use `JSONTextDecoder`\ndirectly.\n\n```javascript\nconst response = await fetch(\"your.api/endpoint\");\nconst tokens = response.body.pipeThrough(new JSONTextDecoderStream());\n\nfor await (const token of tokens) {\n  // ...\n}\n```\n\n#### JSONTextEncoderStream\n\nWraps a `JSONTextEncoder` and accepts `Token` only. While streams like `JSONTextSelectorStream` and\n`JSONTextLineStream` emit `Value`, `Value` provides a `.tokens()` generator that can be used to feed\ntokens into `JSONTextEncoderStream`.\n\nThe following example demonstrates how to write a `TransformStream` that converts `Value` into\n`Token` and pipe it into a `JSONTextEncoderStream`.\n\n```javascript\nconst encoder = new JSONTextEncoderStream();\nconst transformer = new TransformStream({\n  transform(value, controller) {\n    for (const token of value.tokens()) {\n      controller.enqueue(token);\n    }\n  },\n});\n\nstream.pipeThrough(transformer).pipeThrough(encoder);\n```\n\n#### JSONTextSelectorStream\n\n`JSONTextSelectorStream` supports a subset of\n[JSON Path](https://datatracker.ietf.org/doc/html/rfc9535) syntax for selecting specific values from\na JSON document.\n\n| Supported                                                                                       | Syntax                                  |\n| :---------------------------------------------------------------------------------------------- | :-------------------------------------- |\n| [Root Identifier](https://datatracker.ietf.org/doc/html/rfc9535#name-root-identifier)           | `$`                                     |\n| [Child Segment](https://datatracker.ietf.org/doc/html/rfc9535#name-child-segment)               | `.`, `[]`                               |\n| [Descendant Segment](https://datatracker.ietf.org/doc/html/rfc9535#name-descendant-segment)     | `..`                                    |\n| [Name Selector](https://datatracker.ietf.org/doc/html/rfc9535#name-name-selector)               | `.name`, `['name']`, `['name', 'name']` |\n| [Wildcard Selector](https://datatracker.ietf.org/doc/html/rfc9535#name-wildcard-selector)       | `.*`                                    |\n| [Index Selector](https://datatracker.ietf.org/doc/html/rfc9535#name-index-selector)             | `[0]`                                   |\n| [Array Slice Selector](https://datatracker.ietf.org/doc/html/rfc9535#name-array-slice-selector) | `[start:end:step]`                      |\n\n\u003e [!NOTE]\n\u003e Negative numbers in index and slice selectors are not supported.\n\nThe following example extracts all `email` values from `{ \"users\": [ ... ] }`.\n\n```javascript\nconst response = await fetch(\"your.api/endpoint\");\nconst emails = response.body.pipeThrough(new JSONTextSelectorStream(\"$.users[*].email\"));\n\nfor await (const value of emails) {\n  console.log(value.json());\n}\n```\n\n\u003e [!TIP]\n\u003e `Value` has an optional `.pointer` property that returns the\n\u003e [JSON Pointer](https://datatracker.ietf.org/doc/html/rfc6901) of where the value was located in\n\u003e the source document. `JSONTextSelectorStream` sets this automatically, so you can use it to get\n\u003e the exact location of each selected value.\n\n#### JSONTextLineStream\n\n`JSONTextLineStream` is designed for processing JSON Lines (JSONL) format, but it can also handle\nconcatenated JSON documents.\n\n```javascript\nconst response = await fetch(\"your.api/endpoint\");\nconst lines = response.body.pipeThrough(new JSONTextLineStream());\n\nfor await (const value of lines) {\n  console.log(value.json());\n}\n```\n\n### Components\n\n#### Token\n\nA `Token` represents the smallest lexical unit of JSON. It is either a scalar (like `\"Alice\"`,\n`true`, `123`, `null`) or a structural symbol (like `{`, `}`, `[`, `]`), it **never** represents a\nwhole object or array.\n\nSee JSR documentation for all available methods, such as `ARRAY_BEGIN`, `.asNumber()`,\n`.isScalar()`, etc.\n\n\u003e [!IMPORTANT]\n\u003e Tokens and Values returned from a decoder are views into its internal buffer. This buffer is\n\u003e overwritten the next time you `.push()` more bytes.\n\u003e\n\u003e If you need to keep a token or value around for later use, you must copy it using `.clone()`:\n\u003e\n\u003e ```javascript\n\u003e const collected = [];\n\u003e while ((token = decoder.readToken()) !== undefined) {\n\u003e   collected.push(token); // ❌ UNSAFE: all entries will point to the mutated bytes\n\u003e   collected.push(token.clone()); // ✅ SAFE: creates an independent copy\n\u003e }\n\u003e ```\n\n#### Value\n\nA `Value` represents a complete JSON unit. It can be a simple scalar, or it can be an entire\n`object` or `array` including everything nested inside it.\n\nUse `Value` when you need a specific subtree. You can call `value.json()` to materialize it into a\nJavaScript object, or use `decoder.skipValue()` to cheaply discard massive branches you don't need\nwithout ever parsing them.\n\n##### Create a Value instance `from`\n\n`.from()` is a static helper that creates a `Value` instance from any JSON-serializable value.\n\n```javascript\nconst value = Value.from(\"Hello, World!\");\n```\n\n##### Canonicalize\n\n`.canonicalize()` implements the\n[JSON Canonicalization Scheme](https://datatracker.ietf.org/doc/html/rfc8785) by recursively sorting\nobject keys by UTF-16 code unit order and normalizing numbers. The result is deterministic and\nidempotent, making it ideal for hashing or strict comparisons.\n\n```javascript\nconst value = Value.from({ b: 2, a: 1 }).canonicalize(); // {\"a\":1,\"b\":2}\n```\n\n##### Tokenize\n\n`.tokens()` is a generator method that yields each `Token` within this value in document order. This\nallows you to process or transform the value token by token without materializing the whole thing in\nmemory.\n\n```javascript\nconst value = Value.from({ name: \"Alice\", tags: [\"admin\", \"user\"] });\n\nfor (const token of value.tokens()) {\n  if (token.kind === KIND.STRING) {\n    console.log(token.asString());\n  }\n}\n```\n\n#### Kind\n\n`KIND` is a constant object containing string discriminants that identify the structural role of a\nJSON token. Always use these constants for comparisons to avoid typos.\n\n| Kind                | Value      |\n| :------------------ | :--------- |\n| `KIND.NULL`         | `\"null\"`   |\n| `KIND.FALSE`        | `\"false\"`  |\n| `KIND.TRUE`         | `\"true\"`   |\n| `KIND.STRING`       | `\"string\"` |\n| `KIND.NUMBER`       | `\"number\"` |\n| `KIND.OBJECT_BEGIN` | `\"{\"`      |\n| `KIND.OBJECT_END`   | `\"}\"`      |\n| `KIND.ARRAY_BEGIN`  | `\"[\"`      |\n| `KIND.ARRAY_END`    | `\"]\"`      |\n\nYou can check a token's kind with `token.kind === KIND.STRING` or use helper methods like\n`token.isScalar()`, `token.isStructural()`, etc.\n\n### Error\n\n`jsontext` throws standard JavaScript errors (`TypeError`, `RangeError`, `SyntaxError`) for\nprogrammer mistakes such as invalid arguments or type mismatches. For malformed JSON input, it\nthrows the custom `SyntacticError` described below.\n\n#### SyntacticError\n\nWhen input violates [The JavaScript Object Notation](https://datatracker.ietf.org/doc/html/rfc8259),\nit throws a `SyntacticError` carrying both the byte `offset` and the JSON `pointer` to help pinpoint\nthe exact failure.\n\n```javascript\nimport { JSONTextDecoder, SyntacticError } from \"jsontext\";\n\ntry {\n  const encoder = new TextEncoder();\n  const decoder = new JSONTextDecoder(encoder.encode(`{\"a\": 1, \"b\": }`));\n\n  decoder.end();\n\n  while (decoder.readToken() !== undefined) {\n    /* ... */\n  }\n} catch (error) {\n  if (error instanceof SyntacticError) {\n    console.error(error.offset);\n    console.error(error.pointer);\n    console.error(error.message);\n  }\n}\n```\n\n## Performance\n\n`jsontext` is designed for flat memory usage regardless of input size. The following shows a\npassthrough run on a 1 GB file — heap stays near baseline throughout:\n\n![Passthrough Result](https://github.com/user-attachments/assets/6d8d795b-ba11-41c1-8993-ac5e15088524)\n\nFor full profiling results across passthrough, round-trip, and query scenarios, see\n[docs/performance.md](docs/performance.md).\n\n## Examples\n\nBelow are some simple examples demonstrating how to use `jsontext` for common JSON processing tasks.\nFor more examples, see the [docs/](docs/).\n\n### Replace `null` with an empty string\n\nIn this example, we read a JSON stream from an API endpoint, replace all `null` values with empty\nstrings, and write the modified JSON back out as a stream without ever materializing the whole\ndocument in memory.\n\n```javascript\nimport { JSONTextDecoderStream, JSONTextEncoderStream, KIND, Token } from \"jsontext\";\n\nconst response = await fetch(\"your.api/endpoint\");\n\nif (!response.ok || !response.body) {\n  throw new Error(\"Failed to fetch data\");\n}\n\nconst decoder = new JSONTextDecoderStream();\nconst encoder = new JSONTextEncoderStream();\nconst replacer = new TransformStream({\n  transform(token, controller) {\n    if (token.kind === KIND.NULL) { // Detect a `null` token\n      controller.enqueue(Token.fromString(\"\")); // Emit an empty string token instead\n    } else {\n      controller.enqueue(token);\n    }\n  },\n});\n\nconst stream = response.body.pipeThrough(decoder).pipeThrough(replacer).pipeThrough(encoder);\nconst blob = await new Response(stream).blob();\n```\n\n\u003e [!TIP]\n\u003e `JSONTextDecoderStream` supports token-level processing only. If you need to replace values that\n\u003e may be nested inside objects or arrays, you will need to use `JSONTextDecoder` directly.\n\n## License\n\nThis project is licensed under the [MIT](LICENSE) License.\n\n## Acknowledgements\n\nThis project is inspired by Go's\n[`encoding/json/jsontext`](https://pkg.go.dev/encoding/json/jsontext) standard library.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Flcweden%2Fjsontext","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Flcweden%2Fjsontext","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Flcweden%2Fjsontext/lists"}