{"id":17299875,"url":"https://github.com/twmb/chkjson","last_synced_at":"2025-04-14T12:32:42.612Z","repository":{"id":57482133,"uuid":"128599095","full_name":"twmb/chkjson","owner":"twmb","description":"Fast, in place, and allocation free JSON validating, compacting, and escaping for Go.","archived":false,"fork":false,"pushed_at":"2021-09-21T18:06:59.000Z","size":771,"stargazers_count":7,"open_issues_count":0,"forks_count":1,"subscribers_count":1,"default_branch":"master","last_synced_at":"2025-03-28T01:50:03.419Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/twmb.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2018-04-08T04:51:58.000Z","updated_at":"2025-03-04T10:15:58.000Z","dependencies_parsed_at":"2022-09-02T04:20:22.188Z","dependency_job_id":null,"html_url":"https://github.com/twmb/chkjson","commit_stats":null,"previous_names":[],"tags_count":2,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/twmb%2Fchkjson","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/twmb%2Fchkjson/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/twmb%2Fchkjson/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/twmb%2Fchkjson/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/twmb","download_url":"https://codeload.github.com/twmb/chkjson/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248881830,"owners_count":21176926,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-10-15T11:24:17.965Z","updated_at":"2025-04-14T12:32:42.560Z","avatar_url":"https://github.com/twmb.png","language":"Go","funding_links":[],"categories":[],"sub_categories":[],"readme":"chkjson\n=======\n\n[![GoDoc](https://godoc.org/github.com/twmb/chkjson?status.svg)](https://godoc.org/github.com/twmb/chkjson)\n\nThis repo / package provides alternatives to Go's\n[encoding/json](https://golang.org/pkg/encoding/json/) package for validating\nJSON and compacting it to a slice. A great appeal for this package is the\nin-place `AppendCompact` function and the potentially in-place\n`AppendConcatJSONP` function (and string variants).\n\nA minor appeal is the quick and easy string and slice escaping.\n\n## Why would I need this?\n\nThis package was designed for streaming untrusted JSON input to endpoints that\nspecifically take JSON. Because the input is untrusted, it should be validated\nbefore being sent off. This is even more important if the JSON is being batched\nbefore sending—you do not want to mix good JSON with bad and have the whole\nbatch rejected.\n\n## Why this package?\n\nIf you are streaming _lots_ of JSON _all of the time_, the CPU and memory\nsavings this package provides add up.\n\nThis package outperforms any other JSON validating or compacting\nimplementations. The code, while _supremely_ ugly and repetitive,\nis written for the compiler.\nThe implementation was guided by profiling sections of code individually with\nvarious implementations.\n\nThis is the type of code that truly is write once, maintain rarely. It\nimplements validating (and compacting) to spec. The current tests are\ncomprehensive, the parsers have been fuzzed, and the implementation is nearly\nas good as it gets without dropping into assembly. Validating passes all\nrequired `y_` and `n_` files in [JSONTestSuite](https://github.com/nst/JSONTestSuite),\nand compacting followed by unmarshalling for all of these files matches stdlib\nunmarshalling.\n\n\n## Why not this package?\n\nThe parsing implementation is recursive. This is much faster than a stack based\nparsing implementation but does consume more memory. The implementation only\nrecurses when necessary and this consumes only about 2.2x the memory a simple\n(not extremely memory optimized) stack based implementation would. For nearly\nall cases, this extra memory consumption is not an issue, especially so since\nthe CPU is freed up more for actual important work.\n\nThe implementation is extremely repetitive and ugly, making it difficult to\nmaintain. This tradeoff was made due to ideally not ever _needing_ changes.\n\nIf you are only validating or compacting a few (countable) amount of times,\nthis package is overkill.\n\n## Documentation\n\nFull documentation can be found on [`godoc`](https://godoc.org/github.com/twmb/chkjson).\n\n## Benchmarks\n\nWhat follows is `benchstat` output for JSON files taken from [valyala/fastjson](https://github.com/valyala/fastjson)\ncomparing stdlib against my code.\n\nMemory allocation savings are elided for brevity (normally 72 to 312 bytes).\n\n```\nname                  old time/op    new time/op    delta\nExtCompact/canada-4     16.8ms ± 1%     4.0ms ± 0%   -76.06%  (p=0.000 n=8+9)\nExtCompact/citm-4       13.9ms ± 1%     2.6ms ± 0%   -81.50%  (p=0.000 n=10+9)\nExtCompact/large-4       234µs ± 0%      70µs ± 0%   -70.11%  (p=0.000 n=10+9)\nExtCompact/medium-4     21.0µs ± 0%     5.2µs ± 1%   -75.11%  (p=0.000 n=8+10)\nExtCompact/small-4      1.80µs ± 0%    0.45µs ± 0%   -74.86%  (p=0.000 n=10+10)\nExtCompact/twitter-4    5.34ms ± 0%    1.43ms ± 0%   -73.24%  (p=0.000 n=9+9)\nExtValid/canada-4       10.9ms ± 0%     3.0ms ± 0%   -72.12%  (p=0.000 n=9+10)\nExtValid/citm-4         8.36ms ± 0%    2.31ms ± 0%   -72.36%  (p=0.000 n=9+10)\nExtValid/large-4         153µs ± 0%      48µs ± 0%   -68.83%  (p=0.000 n=9+10)\nExtValid/medium-4       12.4µs ± 0%     3.8µs ± 0%   -69.70%  (p=0.000 n=10+10)\nExtValid/small-4        1.09µs ± 0%    0.31µs ± 0%   -71.79%  (p=0.000 n=10+10)\nExtValid/twitter-4      3.30ms ± 0%    1.06ms ± 0%   -67.76%  (p=0.000 n=10+9)\n\nname                  old speed      new speed      delta\nExtCompact/canada-4    134MB/s ± 1%   561MB/s ± 0%  +317.73%  (p=0.000 n=8+9)\nExtCompact/citm-4      124MB/s ± 1%   672MB/s ± 0%  +440.60%  (p=0.000 n=10+9)\nExtCompact/large-4     120MB/s ± 0%   402MB/s ± 0%  +234.53%  (p=0.000 n=10+9)\nExtCompact/medium-4    111MB/s ± 0%   445MB/s ± 1%  +301.72%  (p=0.000 n=8+10)\nExtCompact/small-4     106MB/s ± 0%   419MB/s ± 0%  +297.49%  (p=0.000 n=10+10)\nExtCompact/twitter-4   118MB/s ± 0%   442MB/s ± 0%  +273.75%  (p=0.000 n=9+9)\nExtValid/canada-4      206MB/s ± 0%   739MB/s ± 0%  +258.67%  (p=0.000 n=9+10)\nExtValid/citm-4        207MB/s ± 0%   747MB/s ± 0%  +261.84%  (p=0.000 n=9+10)\nExtValid/large-4       184MB/s ± 0%   590MB/s ± 0%  +220.77%  (p=0.000 n=9+10)\nExtValid/medium-4      187MB/s ± 0%   618MB/s ± 0%  +230.03%  (p=0.000 n=10+10)\nExtValid/small-4       175MB/s ± 0%   618MB/s ± 0%  +254.06%  (p=0.000 n=10+10)\nExtValid/twitter-4     191MB/s ± 0%   593MB/s ± 0%  +210.19%  (p=0.000 n=10+9)\n```\n\nIn place compacting directly using the `Compact` function is even faster.\n\n```\nname                         old time/op    new time/op    delta\nExtCompactInplace/canada-4     16.8ms ± 0%     3.5ms ± 1%    -78.88%  (p=0.000 n=8+10)\nExtCompactInplace/citm-4       13.9ms ± 1%     0.9ms ± 1%    -93.36%  (p=0.000 n=10+10)\nExtCompactInplace/large-4       235µs ± 1%      56µs ± 1%    -76.35%  (p=0.000 n=10+10)\nExtCompactInplace/medium-4     21.1µs ± 1%     3.2µs ± 1%    -84.73%  (p=0.000 n=10+10)\nExtCompactInplace/small-4      1.85µs ± 3%    0.28µs ± 2%    -84.83%  (p=0.000 n=10+10)\nExtCompactInplace/twitter-4    5.35ms ± 0%    0.88ms ± 1%    -83.58%  (p=0.000 n=10+10)\n\nname                         old speed      new speed      delta\nExtCompactInplace/canada-4    134MB/s ± 0%   636MB/s ± 1%   +373.47%  (p=0.000 n=8+10)\nExtCompactInplace/citm-4      124MB/s ± 1%  1872MB/s ± 1%  +1406.08%  (p=0.000 n=10+10)\nExtCompactInplace/large-4     120MB/s ± 1%   506MB/s ± 1%   +322.89%  (p=0.000 n=10+10)\nExtCompactInplace/medium-4    110MB/s ± 1%   723MB/s ± 0%   +554.90%  (p=0.000 n=10+10)\nExtCompactInplace/small-4     103MB/s ± 3%   677MB/s ± 2%   +558.42%  (p=0.000 n=10+10)\nExtCompactInplace/twitter-4   118MB/s ± 0%   719MB/s ± 1%   +508.94%  (p=0.000 n=10+10)\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ftwmb%2Fchkjson","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Ftwmb%2Fchkjson","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ftwmb%2Fchkjson/lists"}