{"id":21657607,"url":"https://github.com/sourcemeta-research/jsonschema-benchmark","last_synced_at":"2025-05-08T00:22:07.340Z","repository":{"id":255035278,"uuid":"848339906","full_name":"sourcemeta-research/jsonschema-benchmark","owner":"sourcemeta-research","description":null,"archived":false,"fork":false,"pushed_at":"2025-03-26T14:37:38.000Z","size":11937,"stargazers_count":1,"open_issues_count":7,"forks_count":2,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-03-26T15:39:19.624Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/sourcemeta-research.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-08-27T15:26:30.000Z","updated_at":"2025-03-26T14:37:41.000Z","dependencies_parsed_at":"2024-08-27T17:12:05.106Z","dependency_job_id":"db0cc4dc-48ff-46cd-bc04-f2d0a69d2793","html_url":"https://github.com/sourcemeta-research/jsonschema-benchmark","commit_stats":null,"previous_names":["sourcemeta-research/jsonschema-benchmark"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sourcemeta-research%2Fjsonschema-benchmark","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sourcemeta-research%2Fjsonschema-benchmark/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sourcemeta-research%2Fjsonschema-benchmark/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sourcemeta-research%2Fjsonschema-benchmark/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/sourcemeta-research","download_url":"https://codeload.github.com/sourcemeta-research/jsonschema-benchmark/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":252974480,"owners_count":21834261,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-25T09:27:14.789Z","updated_at":"2025-05-08T00:22:07.331Z","avatar_url":"https://github.com/sourcemeta-research.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# JSON Schema Benchmark\n\nThe goal of this benchmark is measure the performance of various JSON Schema validators.\nEach validator is run with multiple schemas and a collection of those documents expected to be valid for that schema.\nWhile implementations that do not produce the correct result on validation will be flagged, our goal is not to test for correctness.\nFor correctness tests of validators, please see [Bowtie](https://bowtie.report/).\n\n## Setup\n\nThe benchmark requires Python, make, and Docker.\n[uv](https://github.com/astral-sh/uv) is used for dependency management in the Python scripts.\nTo run all the benchmarks, a report can be produced via `make dist/report.csv`.\nThe `Makefile` accepts parameters `IMPLEMENTATIONS` to specify which implementations to run and `RUNS` for the number of runs per implementation.\nFor example, `make IMPLEMENTATIONS='blaze jsoncons' RUNS=5` will run the Blaze and jsoncons implementations 5 times each.\nTo plot the runtime for each implementation, run `make plots`.\n\n## Implementations\n\nEach implementation is run via Docker.\nFirst, a Docker container is built with all the necessary dependencies.\nThen, at runtime, a folder containing the schema and the necessary dependencies is mounted and the time to validate all documents is measured.\nAll implementations can be found in the `implementations/` subdirectory.\nA summary of these implementations is given below.\n\n- [Ajv](https://ajv.js.org/) (JS)\n- [Blaze](https://github.com/sourcemeta/blaze) (C++)\n- [@exodus/schemasafe](https://github.com/ExodusMovement/schemasafe) (JS)\n- [boon](https://github.com/santhosh-tekuri/boon) (Rust)\n- [Corvus.JsonSchema](https://github.com/corvus-dotnet/Corvus.JsonSchema) (C#)\n- [Hyperjump](https://github.com/hyperjump-io/json-schema) (JS)\n- [Opis](https://opis.io/json-schema) (PHP)\n- [jsoncons](https://github.com/danielaparker/jsoncons) (C++)\n- [json_schemer](https://github.com/davishmcclurg/json_schemer) (Ruby)\n- [json-schema-validator](https://github.com/networknt/json-schema-validator) (Java)\n- [json-schema-validator](https://github.com/OptimumCode/json-schema-validator) (Kotlin)\n- [jsonschema](https://github.com/santhosh-tekuri/jsonschema/) (Go)\n- [jsonschema](https://python-jsonschema.readthedocs.io/en/stable/) (Python)\n- [JsonSchema.NET](https://github.com/json-everything/json-everything) (C#)\n- [JSV](https://github.com/lud/jsv) (Elixir)\n\nNote that some implementations are currently ignored by default to limit the runtime of the benchmark.\nThese implementations are identified by a `.benchmark-ignore` file in the implementation subdirectory.\nTo run one of these implementations, it must be explicitly specified as described above.\n\n## Adding a new implementation\n\nFirst, each implementation must have a `Dockerfile` that copies in any necessary scripts and installs dependencies.\nThere is also a `version.sh` script that must output the version of the implementation (often extracted from whatever dependency management tool is used).\nFinally, appropriate targets must be added to the `Makefile` to build the Docker container and run the benchmark.\nWe will gladly accept pull requests to add new implementations.\n\n## Results\n\nThe most recent results can be seen (via GitHub Actions)[https://github.com/sourcemeta-research/jsonschema-benchmark/actions/workflows/ci.yml].\nNote that while there is noise in the results across runs due to the use of shared infrastructure, the relative ranking of different implementations is generally consistent.\nIt also worth noting that some implementations compile schemas ahead of time into a more efficient representation, while others interpret the entire schema at runtime.\nCurrently we operate under the assumption that a schema changes infrequently enough that the compilation process is unlikely to be a performance bottleneck.\nAs such, we currently only measure the time for validation and exclude any compilation time.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsourcemeta-research%2Fjsonschema-benchmark","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fsourcemeta-research%2Fjsonschema-benchmark","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsourcemeta-research%2Fjsonschema-benchmark/lists"}