{"id":13994234,"url":"https://github.com/daggaz/json-stream","last_synced_at":"2025-10-29T01:51:13.017Z","repository":{"id":46210892,"uuid":"283274062","full_name":"daggaz/json-stream","owner":"daggaz","description":"Simple streaming JSON parser and encoder.","archived":false,"fork":false,"pushed_at":"2025-10-26T10:20:48.000Z","size":134,"stargazers_count":171,"open_issues_count":16,"forks_count":17,"subscribers_count":4,"default_branch":"master","last_synced_at":"2025-10-26T11:32:19.255Z","etag":null,"topics":["json","json-parser","parser","python"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/daggaz.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE.txt","code_of_conduct":"CODE_OF_CONDUCT.md","threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2020-07-28T16:53:54.000Z","updated_at":"2025-10-26T10:20:52.000Z","dependencies_parsed_at":"2024-01-18T05:11:22.952Z","dependency_job_id":"6e1239a3-c774-45f6-8e3f-ec706bbbfafc","html_url":"https://github.com/daggaz/json-stream","commit_stats":{"total_commits":62,"total_committers":7,"mean_commits":8.857142857142858,"dds":0.4516129032258065,"last_synced_commit":"eead2261f793964e2ae7914c52383fd91f172500"},"previous_names":[],"tags_count":2,"template":false,"template_full_name":null,"purl":"pkg:github/daggaz/json-stream","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/daggaz%2Fjson-stream","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/daggaz%2Fjson-stream/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/daggaz%2Fjson-stream/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/daggaz%2Fjson-stream/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/daggaz","download_url":"https://codeload.github.com/daggaz/json-stream/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/daggaz%2Fjson-stream/sbom","scorecard":{"id":316790,"data":{"date":"2025-08-11","repo":{"name":"github.com/daggaz/json-stream","commit":"cbc9111d1cb3b55fbcb1829069e5100bd0a8a560"},"scorecard":{"version":"v5.2.1-40-gf6ed084d","commit":"f6ed084d17c9236477efd66e5b258b9d4cc7b389"},"score":4.2,"checks":[{"name":"Maintained","score":0,"reason":"0 commit(s) and 1 issue activity found in the last 90 days -- score normalized to 0","details":null,"documentation":{"short":"Determines if the project is \"actively maintained\".","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#maintained"}},{"name":"Binary-Artifacts","score":10,"reason":"no binaries found in the repo","details":null,"documentation":{"short":"Determines if the project has generated executable (binary) artifacts in the source repository.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#binary-artifacts"}},{"name":"Dangerous-Workflow","score":10,"reason":"no dangerous workflow patterns detected","details":null,"documentation":{"short":"Determines if the project's GitHub Action workflows avoid dangerous patterns.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#dangerous-workflow"}},{"name":"Code-Review","score":0,"reason":"Found 1/20 approved changesets -- score normalized to 0","details":null,"documentation":{"short":"Determines if the project requires human code review before pull requests (aka merge requests) are merged.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#code-review"}},{"name":"Pinned-Dependencies","score":0,"reason":"dependency not pinned by hash detected -- score normalized to 0","details":["Warn: GitHub-owned GitHubAction not pinned by hash: .github/workflows/tests.yml:22: update your workflow using https://app.stepsecurity.io/secureworkflow/daggaz/json-stream/tests.yml/master?enable=pin","Warn: GitHub-owned GitHubAction not pinned by hash: .github/workflows/tests.yml:24: update your workflow using https://app.stepsecurity.io/secureworkflow/daggaz/json-stream/tests.yml/master?enable=pin","Warn: GitHub-owned GitHubAction not pinned by hash: .github/workflows/tests.yml:65: update your workflow using https://app.stepsecurity.io/secureworkflow/daggaz/json-stream/tests.yml/master?enable=pin","Warn: GitHub-owned GitHubAction not pinned by hash: .github/workflows/tests.yml:68: update your workflow using https://app.stepsecurity.io/secureworkflow/daggaz/json-stream/tests.yml/master?enable=pin","Warn: third-party GitHubAction not pinned by hash: .github/workflows/tests.yml:100: update your workflow using https://app.stepsecurity.io/secureworkflow/daggaz/json-stream/tests.yml/master?enable=pin","Warn: pipCommand not pinned by hash: .github/workflows/tests.yml:29","Warn: pipCommand not pinned by hash: .github/workflows/tests.yml:30","Warn: pipCommand not pinned by hash: .github/workflows/tests.yml:31","Warn: pipCommand not pinned by hash: .github/workflows/tests.yml:46","Warn: pipCommand not pinned by hash: .github/workflows/tests.yml:95","Warn: pipCommand not pinned by hash: .github/workflows/tests.yml:96","Info:   0 out of   4 GitHub-owned GitHubAction dependencies pinned","Info:   0 out of   1 third-party GitHubAction dependencies pinned","Info:   0 out of   6 pipCommand dependencies pinned"],"documentation":{"short":"Determines if the project has declared and pinned the dependencies of its build process.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#pinned-dependencies"}},{"name":"Token-Permissions","score":0,"reason":"detected GitHub workflow tokens with excessive permissions","details":["Info: jobLevel 'contents' permission set to 'read': .github/workflows/tests.yml:61","Warn: no topLevel permission defined: .github/workflows/tests.yml:1","Info: no jobLevel write permissions found"],"documentation":{"short":"Determines if the project's workflows follow the principle of least privilege.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#token-permissions"}},{"name":"CII-Best-Practices","score":0,"reason":"no effort to earn an OpenSSF best practices badge detected","details":null,"documentation":{"short":"Determines if the project has an OpenSSF (formerly CII) Best Practices Badge.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#cii-best-practices"}},{"name":"Security-Policy","score":0,"reason":"security policy file not detected","details":["Warn: no security policy file detected","Warn: no security file to analyze","Warn: no security file to analyze","Warn: no security file to analyze"],"documentation":{"short":"Determines if the project has published a security policy.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#security-policy"}},{"name":"Vulnerabilities","score":10,"reason":"0 existing vulnerabilities detected","details":null,"documentation":{"short":"Determines if the project has open, known unfixed vulnerabilities.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#vulnerabilities"}},{"name":"License","score":10,"reason":"license file detected","details":["Info: project has a license file: LICENSE.txt:0","Info: FSF or OSI recognized license: MIT License: LICENSE.txt:0"],"documentation":{"short":"Determines if the project has defined a license.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#license"}},{"name":"Fuzzing","score":0,"reason":"project is not fuzzed","details":["Warn: no fuzzer integrations found"],"documentation":{"short":"Determines if the project uses fuzzing.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#fuzzing"}},{"name":"Signed-Releases","score":-1,"reason":"no releases found","details":null,"documentation":{"short":"Determines if the project cryptographically signs release artifacts.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#signed-releases"}},{"name":"Branch-Protection","score":-1,"reason":"internal error: error during branchesHandler.setup: internal error: githubv4.Query: Resource not accessible by integration","details":null,"documentation":{"short":"Determines if the default and release branches are protected with GitHub's branch protection settings.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#branch-protection"}},{"name":"Packaging","score":10,"reason":"packaging workflow detected","details":["Info: Project packages its releases by way of GitHub Actions.: .github/workflows/tests.yml:51"],"documentation":{"short":"Determines if the project is published as a package that others can easily download, install, easily update, and uninstall.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#packaging"}},{"name":"SAST","score":0,"reason":"SAST tool is not run on all commits -- score normalized to 0","details":["Warn: 0 commits out of 13 are checked with a SAST tool"],"documentation":{"short":"Determines if the project uses static code analysis.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#sast"}}]},"last_synced_at":"2025-08-18T00:22:42.493Z","repository_id":46210892,"created_at":"2025-08-18T00:22:42.493Z","updated_at":"2025-08-18T00:22:42.493Z"},"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":281544222,"owners_count":26519553,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-10-28T02:00:06.022Z","response_time":60,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["json","json-parser","parser","python"],"created_at":"2024-08-09T14:02:46.992Z","updated_at":"2025-10-29T01:51:12.984Z","avatar_url":"https://github.com/daggaz.png","language":"Python","funding_links":["https://www.buymeacoffee.com/daggaz","https://paypal.me/JCockburn307?country.x=GB\u0026locale.x=en_GB"],"categories":["Python"],"sub_categories":[],"readme":"# json-stream\n\n[![Tests](https://github.com/daggaz/json-stream/actions/workflows/tests.yml/badge.svg)](https://github.com/daggaz/json-stream/actions/workflows/tests.yml)\n[![PyPI package and version badge](https://img.shields.io/pypi/v/json-stream)](https://pypi.org/project/json-stream)\n[![Supported Python versions badge](https://img.shields.io/pypi/pyversions/json-stream)](https://pypi.org/project/json-stream/)\n[![Donate](https://img.shields.io/badge/buy%20me%20a%20coffee-donate-blue.svg)](https://www.buymeacoffee.com/daggaz)\n\nSimple streaming JSON parser and encoder.\n\nWhen [reading](#reading) JSON data, `json-stream` can decode JSON data in \na streaming manner, providing a pythonic dict/list-like interface, or a\n[visitor-based interfeace](#visitor). Can stream from files, [URLs](#urls) \nor [iterators](#iterators).\n\nWhen [writing](#writing) JSON data, `json-stream` can stream JSON objects \nas you generate them.\n\nThese techniques allow you to [reduce memory consumption and \nlatency](#standard-json-problems).\n\n# \u003ca id=\"reading\"\u003e\u003c/a\u003e Reading\n\n`json-stream` is a JSON parser just like the standard library's\n [`json.load()`](https://docs.python.org/3/library/json.html#json.load). It \n will read a JSON document and convert it into native python types.\n\n```python\nimport json_stream\ndata = json_stream.load(f)\n```\n\nFeatures:\n* stream all JSON data types (objects, lists and simple types)\n* stream nested data\n* simple pythonic `list`-like/`dict`-like interface\n* stream truncated or malformed JSON data (up to the first error)\n* [native code parsing speedups](#rust-tokenizer) for most common platforms \n* pure python fallback if native extensions not available\n\nUnlike `json.load()`, `json-stream` can _stream_ JSON data from any file-like or\n[iterable](#iterators) object. This has the following benefits:\n\n* it does not require the whole json document to be read into memory up-front\n* it can start producing data before the entire document has finished loading\n* it only requires enough memory to hold the data currently being parsed\n\nThere are specific integrations for streaming JSON data from [URLs](#urls) using the \n[`requests`](#requests), [`httpx`](#httpx), or [`urllib`](#urllib).\n\nThe objects that `json-stream` produces can be [re-output](#encoding-json-stream-objects)\nusing `json.dump()` with a little work.\n\n## Usage\n\n### `json_stream.load()`\n\n`json_stream.load()` has two modes of operation, controlled by\nthe `persistent` argument (default false).\n\nIt is also possible to \"mix\" the modes as you consume the data.\n\n#### Transient mode (default)\n\nThis mode is appropriate if you can consume the data iteratively. You cannot \nmove backwards through the stream to read data that has already been skipped\nover. It is the mode you **must** use if you want to process large amounts of\nJSON data without consuming large amounts of memory.\n\nIn transient mode, only the data currently being read is stored in memory. Any\ndata previously read from the stream is discarded (it's up to you what to do \nwith it) and attempting to access this data results in a\n`TransientAccessException`.\n\n```python\nimport json_stream\n\n# JSON: {\"count\": 3, \"results\": [\"a\", \"b\", \"c\"]}\ndata = json_stream.load(f)  # data is a transient dict-like object \n# stream has been read up to \"{\"\n\n# use data like a dict\nresults = data[\"results\"]  # results is a transient list-like object\n# stream has been read up to \"[\", we now cannot read \"count\"\n\n# iterate transient list\nfor result in results:\n    print(result)  # prints a, b, c\n# stream has been read up to \"]\"\n\n# attempt to read \"count\" from earlier in stream\ncount = data[\"count\"]  # will raise exception\n# stream is now exhausted\n\n# attempt to read from list that has already been iterated\nfor result in results:  # will raise exception\n    pass\n```\n\n#### Persistent mode\n\nIn persistent mode all previously read data is stored in memory as\nit is parsed. The returned `dict`-like or `list`-like objects\ncan be used just like normal data structures.\n\nIf you request an index or key that has already been read from the stream\nthen it is retrieved from memory. If you request an index or key that has\nnot yet been read from the stream, then the request blocks until that item\nis found in the stream.\n\n```python\nimport json_stream\n\n# JSON: {\"count\": 1, \"results\": [\"a\", \"b\", \"c\"]}\ndata = json_stream.load(f, persistent=True)\n# data is a streaming  dict-like object \n# stream has been read up to \"{\"\n\n# use data like a dict\nresults = data[\"results\"]  # results is a streaming list-like object\n# stream has been read up to \"[\"\n# count has been stored data\n\n# use results like a list\na_result = results[1]  # a_result = \"b\"\n# stream has been read up to the middle of list\n# \"a\" and \"b\" have been stored in results\n\n# read earlier data from memory\ncount = data[\"count\"]  # count = 1\n\n# consume rest of list\nresults.read_all()\n# stream has been read up to \"}\"\n# \"c\" is now stored in results too\n# results.is_streaming() == False\n\n# consume everything\ndata.read_all()\n# stream is now exhausted\n# data.is_streaming() == False\n```\n\nPersistent mode is not appropriate if you care about memory consumption, but\nprovides an identical experience compared to `json.load()`.\n\n#### Mixed mode\n\nIn some cases you will need to be able to randomly access some part of the \ndata, but still only have that specific data taking up memory resources.\n\nFor example, you might have a very long list of objects, but you cannot always \naccess the keys of the objects in stream order. You want to be able to iterate\nthe list transiently, but access the result objects persistently.\n\nThis can be achieved using the `persistent()` method of all the `list` or\n`dict`-like objects json_stream produces. Calling `persistent()` causes the existing\ntransient object to produce persistent child objects.\n\nNote that the `persistent()` method makes the children of the object it\nis called on persistent, not the object it is called on.\n\n```python\nimport json_stream\n\n# JSON: {\"results\": [{\"x\": 1, \"y\": 3}, {\"y\": 4, \"x\": 2}]}\n# note that the keys of the inner objects are not ordered \ndata = json_stream.load(f)  # data is a transient dict-like object \n\n# iterate transient list, but produce persistent items\nfor result in data['results'].persistent():\n    # result is a persistent dict-like object\n    print(result['x'])  # print x\n    print(result['y'])  # print y (error on second result without .persistent())\n    print(result['x'])  # print x again (error without .persistent())\n```\n\nThe opposite is also possible, going from persistent mode to transient mode, though \nthe use cases for this are more esoteric.\n\n```python\n# JSON: {\"a\": 1, \"x\": [\"long\", \"list\", \"I\", \"don't\", \"want\", \"in\", \"memory\"], \"b\": 2}\ndata = load(StringIO(json), persistent=True).transient()\n# data is a persistent dict-list object that produces transient children\n\nprint(data[\"a\"])  # prints 1\nx = data[\"x\"]  # x is a transient list, you can use it accordingly\nprint(x[0])  # prints long\n\n# access earlier data from memory\nprint(data[\"a\"])  # this would have raised an exception if data was transient\n\nprint(data[\"b\"])  # prints 2\n\n# we have now moved past all the data in the transient list\nprint(x[0])  # will raise exception\n```\n\n### \u003ca id=\"visitor\"\u003e\u003c/a\u003evisitor pattern\n\nYou can also parse using a visitor-style approach where a function you supply\nis called for each data item as it is parsed (depth-first).\n\nThis uses a transient parser under the hood, so does not consume memory for\nthe whole document.\n\n```python\nimport json_stream\n\n# JSON: {\"x\": 1, \"y\": {}, \"xxxx\": [1,2, {\"yyyy\": 1}, \"z\", 1, []]}\n\ndef visitor(item, path):\n    print(f\"{item} at path {path}\")\n\njson_stream.visit(f, visitor)\n```\n\nOutput:\n```\n1 at path ('x',)\n{} at path ('y',)\n1 at path ('xxxx', 0)\n2 at path ('xxxx', 1)\n1 at path ('xxxx', 2, 'yyyy')\nz at path ('xxxx', 3)\n1 at path ('xxxx', 4)\n[] at path ('xxxx', 5)\n```\n\n### \u003ca id=\"urls\"\u003e\u003c/a\u003e Stream a URL\n\n`json_stream` knows how to stream directly from a URL using a variety of packages.\nSupported packages include:\n- Python's batteries-included [`urllib`](#urllib) package\n- The popular [`requests`](#requests) library\n- The newer [`httpx`](#httpx) library\n\n#### \u003ca id=\"urllib\"\u003e\u003c/a\u003e urllib\n\n[`urllib`](https://docs.python.org/3/library/urllib.html)'s response objects are already\nfile-like objects, so we can just pass them directly to `json-stream`.\n\n```python\nimport urllib.request\nimport json_stream\n\nwith urllib.request.urlopen('http://example.com/data.json') as response:\n    data = json_stream.load(response)\n```\n\n#### \u003ca id=\"requests\"\u003e\u003c/a\u003erequests\n\nTo stream JSON data from [`requests`](https://requests.readthedocs.io/en/latest/), you must\npass `stream=True` when making a request, and call `json_stream.requests.load()` passing the response. \n\n```python\nimport requests\nimport json_stream.requests\n\nwith requests.get('http://example.com/data.json', stream=True) as response:\n    data = json_stream.requests.load(response)\n```\n\n\u003ca id=\"requests-chunk-size\"\u003e\u003c/a\u003e\nNote: these functions use\n[`response.iter_content()`](https://requests.readthedocs.io/en/latest/api/#requests.Response.iter_content) under the\nhood with a `chunk_size` of 10k bytes. This default allows us to perform effective reads from the response stream and \nlower CPU usage. The drawback to this is that `requests` will buffer each read until up to 10k bytes have been read \nbefore passing the data back to `json_stream`. If you need to consume data more responsively the only option is to tune\n`chunk_size` back to 1 to disable buffering.\n\n#### \u003ca id=\"httpx\"\u003e\u003c/a\u003e httpx\n\nTo stream JSON data from [`httpx`](https://www.python-httpx.org/), you must call\n[`stream()`](https://www.python-httpx.org/quickstart/#streaming-responses) when\nmaking your request, and call `json_stream.httpx.load()` passing the response.\n\n```python\nimport httpx\nimport json_stream.httpx\n\nwith httpx.Client() as client, client.stream('GET', 'http://example.com/data.json') as response:\n    data = json_stream.httpx.load(response)\n```\n\nUnder the hood, this works similarly to the [`requests`](#requests) version above, including \nthe caveat about [`chunk_size`](#requests-chunk-size).\n\n### Stream a URL (with visitor)\n\nThe visitor pattern also works with URL streams.\n\n#### urllib\n\n```python\nimport urllib.request\nimport json_stream\n\ndef visitor(item, path):\n    print(f\"{item} at path {path}\")\n    \nwith urllib.request.urlopen('http://example.com/data.json') as response:\n    json_stream.visit(response, visitor)\n```\n\n#### requests\n\n```python\nimport requests\nimport json_stream.requests\n\ndef visitor(item, path):\n    print(f\"{item} at path {path}\")\n    \nwith requests.get('http://example.com/data.json', stream=True) as response:\n    json_stream.requests.visit(response, visitor)\n```\n\nThe [`chunk_size`](#requests-chunk-size) note also applies to `visit()`.\n\n#### httpx\n\n```python\nimport httpx\nimport json_stream.httpx\n\ndef visitor(item, path):\n    print(f\"{item} at path {path}\")\n    \nwith httpx.Client() as client, client.stream('GET', 'http://example.com/data.json') as response:\n    json_stream.httpx.visit(response, visitor)\n```\n\n### \u003ca id=\"iterators\"\u003e\u003c/a\u003e Stream an iterable\n\n`json-stream`'s parsing functions can take any iterable object that produces encoded JSON as\n`byte` objects.\n\n```python\nimport json_stream\n\ndef some_iterator():\n    yield b'{\"some\":'\n    yield b' \"JSON\"}'\n\ndata = json_stream.load(some_iterator())\nassert data['some'] == \"JSON\"\n```\n\nThis is actually how the [`requests`](#requests) and [`httpx`](#httpx) extensions work, as\nboth libraries provide methods to iterate over the response content.\n\n### \u003ca id=\"encoding-json-stream-objects\"\u003e\u003c/a\u003e Encoding json-stream objects\n\nYou can re-output (encode) _persistent_ json-stream `dict`-like and `list`-like object back to JSON using the built-in\n`json.dump()` or `json.dumps()` functions, but with a little additional work:\n\n```python\nimport json\n\nimport json_stream\nfrom json_stream.dump import JSONStreamEncoder, default\n\ndata = json_stream.load(f, persistent=True)\n\n# Option 1: supply json_stream.encoding.default as the default argument\nprint(json.dumps(data, default=default))\n\n# Option 2: supply json_stream.encoding.JSONStreamEncoder as the cls argument\n# This allows you to create your own subclass to further customise encoding\nprint(json.dumps(data, cls=JSONStreamEncoder))\n```\n\nIf you are using a library that internally takes data you pass it and encodes\nit using `json.dump()`. You can also use JSONStreamEncoder() as a context manager.\n\nIt works by monkey-patching the built-in `JSONEncoder.default` method during the\nscope of the `with` statement.\n\n```python \n# library code\ndef some_library_function_out_of_your_control(arg):\n    json.dumps(arg)\n\n# your code\nwith JSONStreamEncoder():\n    some_library_function_out_of_your_control(data)\n```\n\n### Converting to standard Python types\n\nTo convert a json-stream `dict`-like or `list`-like object and all its\ndescendants to a standard `list` and `dict`, you can use the\n`json_stream.to_standard_types` utility:\n\n```python\n# JSON: {\"round\": 1, \"results\": [1, 2, 3]}\ndata = json_stream.load(f)\nresults = data[\"results\"]\nprint(results)  # prints \u003cTransientStreamingJSONList: TRANSIENT, STREAMING\u003e\nconverted = json_stream.to_standard_types(results)\nprint(converted)  # prints [1, 2, 3]\n```\n\n#### Thread safety (experimental)\n\nThere is also a thread-safe version of the `json.dump` context manager:\n\n```python\nfrom json_stream.dump.threading import ThreadSafeJSONStreamEncoder\n\n# your code\nwith ThreadSafeJSONStreamEncoder():\n   some_library_function_out_of_your_control(data)\n```\n\nThe thread-safe implementation will ensure that concurrent uses of the \ncontext manager will only apply the patch for the first thread entering\nthe patched section(s) and will only remove the patch when the last\nthread exits the patched sections(s)\n\nAdditionally, if the patch is somehow called by a thread that is _not_\ncurrently in a patched section (i.e. some other thread calling \n`json.dump`) then that thread will block until the patch has been\nremoved. While such an un-patched thread is active, any thread attempting\nto apply the patch is blocked.\n\n### \u003ca id=\"rust-tokenizer\"\u003e\u003c/a\u003e Rust tokenizer speedups\n\nBy default `json-stream` uses the \n[`json-stream-rs-tokenizer`](https://pypi.org/project/json-stream-rs-tokenizer/)\nnative extension.\n\nThis is a 3rd party Rust-based tokenizer implementations that provides\nsignificant parsing speedup compared to pure python implementation.\n\n`json-stream` will fallback to its pure python tokenizer implementation\nif `json-stream-rs-tokenizer` is not available.\n\n### Custom tokenizer\n\nYou can supply an alternative JSON tokenizer implementation. Simply pass \na tokenizer to the `load()` or `visit()` methods.\n\n```python\njson_stream.load(f, tokenizer=some_tokenizer)\n```\n\nThe requests methods also accept a customer tokenizer parameter.\n\n\n# Writing\n\nThe standard library's `json.dump()` function can only accept standard\npython types such as `dict`, `list`, `str`.\n\n`json-stream` allows you to write streaming JSON output based on python\ngenerators instead.\n\nFor actually encoding and writing to the stream, `json-stream` \nstill uses the standard library's `json.dump()` function, but provides\nwrappers that adapt python generators into `dict`/`list` subclasses \nthat `json.dump()` can use.\n\nThe means that you do not have to generate all of your data upfront\nbefore calling `json.dump()`.\n\n## Usage\n\nTo use `json-stream` to generate JSON data iteratively, you must first \nwrite python generators (or use any other iterable).\n\nTo output JSON objects, the iterable must yield key/value pairs.\n\nTo output JSON lists, the iterable must yield individual items.\n\nThe values yielded can be either be standard python types or another other\n`Streamable` object, allowing lists and object to be arbitrarily nested.\n\n`streamable_list`/`streamable_dict` can be used to wrap an existing\niterable:\n```python\nimport sys\nimport json\n\nfrom json_stream import streamable_list\n\n# wrap existing iterable\ndata = streamable_list(range(10))\n\n# consume iterable with standard json.dump()\njson.dump(data, sys.stdout)\n```\n\nOr they can be used as decorators on generator functions:\n```python\nimport json\nimport sys\n\nfrom json_stream import streamable_dict\n\n# declare a new streamable dict generator function\n@streamable_dict\ndef generate_dict_of_squares(n):\n    for i in range(n):\n        # this could be some memory intensive operation\n        # or just a really large value of n\n        yield i, i ** 2\n\n# data is will already be Streamable because\n# of the decorator\ndata = generate_dict_of_squares(10)\njson.dump(data, sys.stdout)\n```\n\n## Example\n\nThe following example generates a JSON object with a nested JSON list.\nIt uses `time.sleep()` to slow down the generation and show that the\noutput is indeed written as the data is created.\n\n```python\nimport sys\nimport json\nimport time\n\nfrom json_stream.writer import streamable_dict, streamable_list\n\n\n# define a list data generator that (slowly) yields \n# items that will be written as a JSON list\n@streamable_list\ndef generate_list(n):\n    # output n numbers and their squares\n    for i in range(n):  # range is itself a generator\n        yield i\n        time.sleep(1)\n\n\n# define a dictionary data generator that (slowly) yields \n# key/value pairs that will be written as a JSON dict\n@streamable_dict\ndef generate_dict(n):\n    # output n numbers and their squares\n    for i in range(n):  # range is itself a generator\n        yield i, i ** 2\n        time.sleep(1)\n\n    # yield another dictionary item key, with the value\n    # being a streamed nested list\n    yield \"a list\", generate_list(n)\n\n\n# get a streamable generator\ndata = generate_dict(5)\n\n# use json.dump() to write dict generator to stdout\njson.dump(data, sys.stdout, indent=2)\n\n# if you already have an iterable object, you can just\n# call streamable_* on it to make it writable\ndata = streamable_list(range(10))\njson.dump(data, sys.stdout)\n\n```\n\nOutput:\n```json\n{\n  \"0\": 0,\n  \"1\": 1,\n  \"2\": 4,\n  \"3\": 9,\n  \"4\": 16,\n  \"a list\": [\n    0,\n    1,\n    2,\n    3,\n    4\n  ]\n}\n```\n\n# \u003ca id=\"standard-json-problems\"\u003e\u003c/a\u003e What are the problems with the standard `json` package?\n\n## Reading with `json.load()`\nThe problem with the `json.load()` stem from the fact that it must read\nthe whole JSON document into memory before parsing it.\n\n### Memory usage\n\n`json.load()` first reads the whole document into memory as a string. It\nthen starts parsing that string and converting the whole document into python\ntypes again stored in memory. For a very large document, this could be more\nmemory than you have available to your system.\n\n`json_stream.load()` does not read the whole document into memory, it only\nbuffers enough from the stream to produce the next item of data.\n\nAdditionally, in the default transient mode (see below) `json-stream` doesn't store \nup all of the parsed data in memory.\n\n### Latency\n\n`json.load()` produces all the data after parsing the whole document. If you\nonly care about the first 10 items in a list of 2 million items, then you\nhave wait until all 2 million items have been parsed first.\n\n`json_stream.load()` produces data as soon as it is available in the stream.\n\n## \u003ca id=\"writing\"\u003e\u003c/a\u003e Writing\n\n### Memory usage\n\nWhile `json.dump()` does iteratively write JSON data to the given\nfile-like object, you must first produce the entire document to be \nwritten as standard python types (`dict`, `list`, etc). For a very\nlarge document, this could be more memory than you have available \nto your system.\n\n`json-stream` allows you iteratively generate your data one item at\na time, and thus consumes only the memory required to generate that\none item.\n\n### Latency\n\n`json.dump()` can only start writing to the output file once all the\ndata has been generated up front at standard python types.\n\nThe iterative generation of JSON items provided by `json-stream`\nallows the data to be written as it is produced.\n\n# Future improvements\n\n* Allow long strings in the JSON to be read as streams themselves\n* Allow transient mode on seekable streams to seek to data earlier in\nthe stream instead of raising a `TransientAccessException`\n* A more efficient tokenizer?\n\n# Alternatives\n\n## NAYA\n\n[NAYA](https://github.com/danielyule/naya) is a pure python JSON parser for\nparsing a simple JSON list as a stream.\n\n### Why not NAYA?\n\n* It can only stream JSON containing a top-level list \n* It does not provide a pythonic `dict`/`list`-like interface \n\n## Yajl-Py\n\n[Yajl-Py](https://pykler.github.io/yajl-py/) is a wrapper around the C YAJL JSON library that can be used to \ngenerate SAX style events while parsing JSON.\n\n### Why not Yajl-Py?\n\n* No pure python implementation\n* It does not provide a pythonic `dict`/`list`-like interface \n\n## jsonslicer\n\n[jsonslicer](https://github.com/AMDmi3/jsonslicer) is another wrapper around the YAJL C library with a\npath lookup based interface.\n\n### Why not jsonslicer?\n\n* No pure python implementation\n* It does not provide a pythonic `dict`/`list`-like interface\n* Must know all data paths lookup in advance (or make multiple passes)\n\n# Contributing\n\nSee the project [contribution guide](https://github.com/daggaz/json-stream/blob/master/CONTRIBUTING.md).\n\n# Donations\n\n[![PayPal](https://www.paypalobjects.com/webstatic/mktg/Logo/pp-logo-100px.png)](https://paypal.me/JCockburn307?country.x=GB\u0026locale.x=en_GB)\n\nOR\n\n[![\"Buy Me A Coffee\"](https://www.buymeacoffee.com/assets/img/custom_images/orange_img.png)](https://www.buymeacoffee.com/daggaz)\n\n# Acknowledgements\n\nThe JSON tokenizer used in the project was taken from the\n[NAYA](https://github.com/danielyule/naya) project.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdaggaz%2Fjson-stream","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fdaggaz%2Fjson-stream","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdaggaz%2Fjson-stream/lists"}