{"id":15576360,"url":"https://github.com/tzoiker/pysimdjson-schemaful","last_synced_at":"2025-06-14T21:35:24.337Z","repository":{"id":205118270,"uuid":"707093311","full_name":"tzoiker/pysimdjson-schemaful","owner":"tzoiker","description":"Schema-aware pysimdjson loader for efficient parsing of large excessive inputs","archived":false,"fork":false,"pushed_at":"2024-03-24T18:41:19.000Z","size":61,"stargazers_count":4,"open_issues_count":0,"forks_count":1,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-04-01T11:47:21.344Z","etag":null,"topics":["json","json-parser","python","simdjson"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/tzoiker.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2023-10-19T07:58:45.000Z","updated_at":"2024-03-24T17:35:10.000Z","dependencies_parsed_at":"2023-11-07T03:56:48.686Z","dependency_job_id":"e37bf451-6759-4f2d-ac85-5add6270b071","html_url":"https://github.com/tzoiker/pysimdjson-schemaful","commit_stats":null,"previous_names":["tzoiker/pysimdjson-schemaful"],"tags_count":4,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/tzoiker%2Fpysimdjson-schemaful","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/tzoiker%2Fpysimdjson-schemaful/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/tzoiker%2Fpysimdjson-schemaful/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/tzoiker%2Fpysimdjson-schemaful/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/tzoiker","download_url":"https://codeload.github.com/tzoiker/pysimdjson-schemaful/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248083022,"owners_count":21045021,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["json","json-parser","python","simdjson"],"created_at":"2024-10-02T18:47:32.892Z","updated_at":"2025-04-09T17:51:04.527Z","avatar_url":"https://github.com/tzoiker.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# pysimdjson-schemaful\n\nSchema-aware [pysimdjson](https://github.com/TkTech/pysimdjson) loader for\nefficient parsing of large excessive JSON inputs.\n\nWhen working with external APIs you have zero influence on, you may face the\nfollowing unfortunate edge-case (as we did):\n\n* Particular endpoint responds with a relatively massive JSON-body, say, ≥ 1 MB.\n* The amount of data you really need is several magnitudes smaller, e.g., 1 KB.\n* There is no server-side filtering available.\n\nIn such a case it may be very excessive in terms of memory, cpu time and delay to\ndeserialize and, subsequently, validate the whole response, even when using\nfast JSON-deseralization libraries, such as\n[orjson](https://github.com/ijl/orjson\u003e).\n\nIn our particular case we needed less than 0.1% of ~5 MB responses, which we\nvalidated with [pydantic](https://github.com/pydantic/pydantic\u003e).\nFirst, we compared several combinations of deserializers and validators:\n\n* `json` + `pydantic v1` (`Model.parse_raw(json.loads(data))`)\n* `orjson` + `pydantic v1` (`Model.parse_raw(orjson.loads(data))`)\n* `pysimdjson` + `pydantic v1` (`Model.parse_raw(simdjson.loads(data))`)\n* `pydantic v2` (`Model.model_validate_json(data)`)\n\nTo our surprise internal `pydantic v2` parser appeared to be ~2-3 times slower\nthan `json` + `pydantic v1`. The fastest was `orjson` + `pydantic v1`\n(~2-3 times faster than `json` and a bit faster than full `simdjson` parsing).\nSuch a speed-up, however, still comes with excessive memory spending\n(as a complete python dict object is created and populated on deserialization).\n\nThus, we ended up using `pysimdjson` with its fast lazy parsing and manually\niterated over nested JSON objects/arrays and extracted only required keys. It is\nugly, tedious and hard to maintain of course. However, it showed to be several\ntimes faster than `orjson` and decreased memory consumption.\n\n\n## Table of Contents\n\n* [The crux](#crux)\n* [When to use?](#when_use)\n* [Installation](#installation)\n* [Usage](#usage)\n  * [Basic](#usage_basic)\n  * [Reusing parser](#usage_reusing_parser)\n  * [Pydantic v1](#usage_pydantic_v1)\n  * [Pydantic v2](#usage_pydantic_v2)\n* [Benchmarks (TBD)](#benchmarks)\n\n## \u003ca name=\"crux\"/\u003e The crux\nThis package aims to automate the manual labour of lazy loading with pysimdjson.\n\nSimply feed the JSON-schema in and the input data will be traversed\nand loaded with pysimdjson accordingly.\n\nSupports\n* `pydantic\u003e=1,\u003c3`\n* `python\u003e=3.8,\u003c3.12`\n* `simdjson\u003e=2,\u003c6` (with caveats)\n\nDoes not support complex schemas (it may be not very reasonable from the\npractical standpoint anyway), e.g.,\n* `anyOf` (`Union[Model1, Model2]`)\n* ...\n\nIn such cases it will fully (not lazily) load the underlying objects.\n\n## \u003ca name=\"when_use\"/\u003e  When to use?\n\n* [ ] Input JSON data is large relatively to what is needed in there, i.e.,\nselectivity is small.\n* [ ] Other deserialization methods appear to be slower and/or more memory\nconsuming.\n\nIf you can check all the boxes, then, this package may prove useful to you.\n**Never** use it as a default deserialization method: run some benchmarks for\nyour particular case first, otherwise, it may and will disappoint you.\n\n## \u003ca name=\"installation\"/\u003e Installation\n\n```bash\npip install pysimdjson-schemaful\n```\n\nIf you need pydantic support\n```bash\npip install \"pysimdjson-schemaful[pydantic]\"\n```\n\n## \u003ca name=\"Usage\"/\u003e Usage\n\n### \u003ca name=\"usage_basic\"/\u003e Basic\n\n\u003c!--  name: test_basic --\u003e\n```python\nimport json\nfrom simdjson_schemaful import loads\n\nschema = {\n  \"type\": \"array\",\n  \"items\": {\n    \"$ref\": \"#/definitions/Model\"\n  },\n  \"definitions\": {\n    \"Model\": {\n      \"type\": \"object\",\n      \"properties\": {\n        \"key\": {\"type\": \"integer\"},\n      }\n    }\n  }\n}\n\ndata = json.dumps([\n    {\"key\": 0, \"other\": 1},\n    {\"missing\": 2},\n])\n\nparsed = loads(data, schema=schema)\n\nassert parsed == [\n    {\"key\": 0},\n    {},\n]\n```\n\nExample with `additionalProperties`:\n\n\u003c!--  name: test_basic --\u003e\n```python\nschema = {\n  \"type\": \"object\",\n  \"additionalProperties\": {\n    \"$ref\": \"#/definitions/Model\",\n  },\n  \"definitions\": {\n    \"Model\": {\n      \"type\": \"object\",\n      \"properties\": {\n        \"key\": {\"type\": \"integer\"},\n      }\n    }\n  }\n}\n\ndata = json.dumps({\n    \"some\": {\"key\": 0, \"other\": 1},\n    \"other\": {\"missing\": 2},\n})\n\nparsed = loads(data, schema=schema)\n\nassert parsed == {\n    \"some\": {\"key\": 0},\n    \"other\": {},\n}\n```\n\n### \u003ca name=\"usage_reusing_parser\"/\u003e Reusing parser\n\nWith re-used simdjson parser **(recommended when used in a single thread,\notherwise better consult pysimdjson project on thread-safety)**:\n\n\u003c!--  name: test_basic --\u003e\n```python\nfrom simdjson import Parser\n\nparser = Parser()\nparsed = loads(data, schema=schema, parser=parser)\n\nassert parsed == {\n    \"some\": {\"key\": 0},\n    \"other\": {},\n}\n```\n\n### \u003ca name=\"usage_pydantic_v1\"/\u003e Pydantic v1\n\nWith model (call `BaseModel.parse_raw_simdjson`):\n\n\u003c!--  name: test_pydantic_v1_model --\u003e\n```python\nimport json\nfrom simdjson_schemaful.pydantic.v1 import BaseModel\n\nclass Model(BaseModel):\n  key: int\n\ndata = json.dumps({\"key\": 0, \"other\": 1})\n\nobj = Model.parse_raw_simdjson(data)\n```\n\nWith type (call `parse_raw_as_simdjson`):\n\n\u003c!--  name: test_pydantic_v1_type --\u003e\n```python\nimport json\nfrom typing import List\nfrom simdjson_schemaful.pydantic.v1 import BaseModel, parse_raw_simdjson_as\n\nclass Model(BaseModel):\n  key: int\n\nType = List[Model]\n\ndata = json.dumps([\n  {\"key\": 0, \"other\": 1},\n  {\"key\": 1, \"another\": 2},\n])\n\nobj1, obj2 = parse_raw_simdjson_as(Type, data)\n```\n\n### \u003ca name=\"usage_pydantic_v2\"/\u003e Pydantic v2\n\nWith model (call `BaseModel.model_validate_simdjson`):\n\n\u003c!--  name: test_pydantic_v2_model --\u003e\n```python\nimport json\nfrom simdjson_schemaful.pydantic.v2 import BaseModel\n\nclass Model(BaseModel):\n  key: int\n\ndata = json.dumps({\"key\": 0, \"other\": 1})\n\nobj = Model.model_validate_simdjson(data)\n```\n\nWith type adapter (call `TypeAdapter.validate_simdjson`)\n\n\u003c!--  name: test_pydantic_v2_type_adapter --\u003e\n```python\nimport json\nfrom typing import List\nfrom simdjson_schemaful.pydantic.v2 import BaseModel, TypeAdapter\n\nclass Model(BaseModel):\n  key: int\n\nadapter = TypeAdapter(List[Model])\n\ndata = json.dumps([\n  {\"key\": 0, \"other\": 1},\n  {\"key\": 1, \"another\": 2},\n])\n\nobj1, obj2 = adapter.validate_simdjson(data)\n```\n\n## \u003ca name=\"benchmarks\"/\u003e Benchmarks\n\nTBD\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ftzoiker%2Fpysimdjson-schemaful","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Ftzoiker%2Fpysimdjson-schemaful","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ftzoiker%2Fpysimdjson-schemaful/lists"}