{"id":22748519,"url":"https://github.com/pitr/jsontokenizer","last_synced_at":"2026-05-03T06:38:05.984Z","repository":{"id":57597998,"uuid":"378762277","full_name":"pitr/jsontokenizer","owner":"pitr","description":"Zero-allocation JSON tokenizer","archived":false,"fork":false,"pushed_at":"2023-12-31T16:00:42.000Z","size":71,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":2,"default_branch":"main","last_synced_at":"2025-03-30T05:45:04.380Z","etag":null,"topics":["go","golang","json","tokenizer"],"latest_commit_sha":null,"homepage":"","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/pitr.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2021-06-21T00:01:05.000Z","updated_at":"2024-08-05T21:31:17.000Z","dependencies_parsed_at":"2024-12-11T03:33:12.510Z","dependency_job_id":"b56a6cea-0b7e-4ed5-bfe5-f1a16f940104","html_url":"https://github.com/pitr/jsontokenizer","commit_stats":null,"previous_names":[],"tags_count":5,"template":false,"template_full_name":null,"purl":"pkg:github/pitr/jsontokenizer","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/pitr%2Fjsontokenizer","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/pitr%2Fjsontokenizer/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/pitr%2Fjsontokenizer/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/pitr%2Fjsontokenizer/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/pitr","download_url":"https://codeload.github.com/pitr/jsontokenizer/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/pitr%2Fjsontokenizer/sbom","scorecard":{"id":735624,"data":{"date":"2025-08-18","repo":{"name":"github.com/pitr/jsontokenizer","commit":"62b519fac1259bbf9e3832da2fe393e99d7f5549"},"scorecard":{"version":"v5.2.1-41-g40576783","commit":"40576783fda6698350fcbbeaea760ff827433034"},"score":3,"checks":[{"name":"Binary-Artifacts","score":10,"reason":"no binaries found in the repo","details":null,"documentation":{"short":"Determines if the project has generated executable (binary) artifacts in the source repository.","url":"https://github.com/ossf/scorecard/blob/40576783fda6698350fcbbeaea760ff827433034/docs/checks.md#binary-artifacts"}},{"name":"Token-Permissions","score":-1,"reason":"No tokens found","details":null,"documentation":{"short":"Determines if the project's workflows follow the principle of least privilege.","url":"https://github.com/ossf/scorecard/blob/40576783fda6698350fcbbeaea760ff827433034/docs/checks.md#token-permissions"}},{"name":"Code-Review","score":0,"reason":"Found 0/10 approved changesets -- score normalized to 0","details":null,"documentation":{"short":"Determines if the project requires human code review before pull requests (aka merge requests) are merged.","url":"https://github.com/ossf/scorecard/blob/40576783fda6698350fcbbeaea760ff827433034/docs/checks.md#code-review"}},{"name":"Maintained","score":0,"reason":"0 commit(s) and 0 issue activity found in the last 90 days -- score normalized to 0","details":null,"documentation":{"short":"Determines if the project is \"actively maintained\".","url":"https://github.com/ossf/scorecard/blob/40576783fda6698350fcbbeaea760ff827433034/docs/checks.md#maintained"}},{"name":"Packaging","score":-1,"reason":"packaging workflow not detected","details":["Warn: no GitHub/GitLab publishing workflow detected."],"documentation":{"short":"Determines if the project is published as a package that others can easily download, install, easily update, and uninstall.","url":"https://github.com/ossf/scorecard/blob/40576783fda6698350fcbbeaea760ff827433034/docs/checks.md#packaging"}},{"name":"Pinned-Dependencies","score":-1,"reason":"no dependencies found","details":null,"documentation":{"short":"Determines if the project has declared and pinned the dependencies of its build process.","url":"https://github.com/ossf/scorecard/blob/40576783fda6698350fcbbeaea760ff827433034/docs/checks.md#pinned-dependencies"}},{"name":"SAST","score":0,"reason":"no SAST tool detected","details":["Warn: no pull requests merged into dev branch"],"documentation":{"short":"Determines if the project uses static code analysis.","url":"https://github.com/ossf/scorecard/blob/40576783fda6698350fcbbeaea760ff827433034/docs/checks.md#sast"}},{"name":"Dangerous-Workflow","score":-1,"reason":"no workflows found","details":null,"documentation":{"short":"Determines if the project's GitHub Action workflows avoid dangerous patterns.","url":"https://github.com/ossf/scorecard/blob/40576783fda6698350fcbbeaea760ff827433034/docs/checks.md#dangerous-workflow"}},{"name":"CII-Best-Practices","score":0,"reason":"no effort to earn an OpenSSF best practices badge detected","details":null,"documentation":{"short":"Determines if the project has an OpenSSF (formerly CII) Best Practices Badge.","url":"https://github.com/ossf/scorecard/blob/40576783fda6698350fcbbeaea760ff827433034/docs/checks.md#cii-best-practices"}},{"name":"Security-Policy","score":0,"reason":"security policy file not detected","details":["Warn: no security policy file detected","Warn: no security file to analyze","Warn: no security file to analyze","Warn: no security file to analyze"],"documentation":{"short":"Determines if the project has published a security policy.","url":"https://github.com/ossf/scorecard/blob/40576783fda6698350fcbbeaea760ff827433034/docs/checks.md#security-policy"}},{"name":"Fuzzing","score":0,"reason":"project is not fuzzed","details":["Warn: no fuzzer integrations found"],"documentation":{"short":"Determines if the project uses fuzzing.","url":"https://github.com/ossf/scorecard/blob/40576783fda6698350fcbbeaea760ff827433034/docs/checks.md#fuzzing"}},{"name":"License","score":10,"reason":"license file detected","details":["Info: project has a license file: LICENSE:0","Info: FSF or OSI recognized license: MIT License: LICENSE:0"],"documentation":{"short":"Determines if the project has defined a license.","url":"https://github.com/ossf/scorecard/blob/40576783fda6698350fcbbeaea760ff827433034/docs/checks.md#license"}},{"name":"Signed-Releases","score":-1,"reason":"no releases found","details":null,"documentation":{"short":"Determines if the project cryptographically signs release artifacts.","url":"https://github.com/ossf/scorecard/blob/40576783fda6698350fcbbeaea760ff827433034/docs/checks.md#signed-releases"}},{"name":"Branch-Protection","score":0,"reason":"branch protection not enabled on development/release branches","details":["Warn: branch protection not enabled for branch 'main'"],"documentation":{"short":"Determines if the default and release branches are protected with GitHub's branch protection settings.","url":"https://github.com/ossf/scorecard/blob/40576783fda6698350fcbbeaea760ff827433034/docs/checks.md#branch-protection"}},{"name":"Vulnerabilities","score":10,"reason":"0 existing vulnerabilities detected","details":null,"documentation":{"short":"Determines if the project has open, known unfixed vulnerabilities.","url":"https://github.com/ossf/scorecard/blob/40576783fda6698350fcbbeaea760ff827433034/docs/checks.md#vulnerabilities"}}]},"last_synced_at":"2025-08-22T15:43:12.079Z","repository_id":57597998,"created_at":"2025-08-22T15:43:12.079Z","updated_at":"2025-08-22T15:43:12.079Z"},"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":32560906,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-03T06:36:36.687Z","status":"ssl_error","status_checked_at":"2026-05-03T06:36:09.306Z","response_time":103,"last_error":"SSL_read: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["go","golang","json","tokenizer"],"created_at":"2024-12-11T03:33:00.767Z","updated_at":"2026-05-03T06:38:05.962Z","avatar_url":"https://github.com/pitr.png","language":"Go","funding_links":[],"categories":[],"sub_categories":[],"readme":"[![Go Reference](https://pkg.go.dev/badge/pitr.ca/jsontokenizer.svg)](https://pkg.go.dev/pitr.ca/jsontokenizer)\n\n# JSON Tokenizer\n\nZero-allocation JSON tokenizer.\n\n## Features\n\n- Fast. ~15x faster than `encoding/json.Decoder`. See benchmarks below.\n- Similar API to `encoding/json.Decoder`.\n- No reflection.\n- No allocations, beyond small buffer for reading.\n- Can be reused with a call to `Reset`.\n\n## Anti-Features\n\n- Does **NOT** parse JSON. Will not verify semantic correctness. `[}` will produce 2 tokens without errors.\n- Needs an `io.Writer` to write numbers and strings into. Based on the use case, can be `os.Stdout`, `bytes.Buffer`, [ByteBuffer](https://github.com/valyala/bytebufferpool), etc.\n- Does not escape strings. `\"he is 5'11\\\\\".\"` will be exactly that.\n- Does not parse numbers into floats/ints. Use `strconv.Atoi()` if needed.\n- Not thread safe. Use with `sync.Mutex` or the like to prevent simultaneous calls.\n\n## Quick Start\n\n```go\nimport (\n\t\"io\"\n\n\tjson \"pitr.ca/jsontokenizer\"\n)\n\nfunc example(in io.Reader) error {\n\ttk := json.New(in)\n\n\tfor {\n\t\ttok, err := tk.Token()\n\t\tif err == io.EOF {\n\t\t\treturn nil\n\t\t}\n\t\tif err != nil {\n\t\t\treturn err\n\t\t}\n\t\tswitch tok {\n\t\tcase json.TokNull:\n\t\t\tprintln(\"got null\")\n\t\tcase json.TokTrue, json.TokFalse:\n\t\t\tprintln(\"got bool\")\n\t\tcase json.TokArrayOpen, json.TokArrayClose, json.TokObjectOpen, json.TokObjectClose, json.TokObjectColon, json.TokComma:\n\t\t\tprintln(\"got delimiter\")\n\t\tcase json.TokNumber:\n\t\t\tprintln(\"got number\")\n\t\t\t_, err := tk.ReadNumber(io.Discard)\n\t\t\tif err != nil {\n\t\t\t\treturn err\n\t\t\t}\n\t\tcase json.TokString:\n\t\t\tprintln(\"got string\")\n\t\t\t_, err := tk.ReadString(io.Discard)\n\t\t\tif err != nil {\n\t\t\t\treturn err\n\t\t\t}\n\t\t}\n\t}\n}\n```\n\n## Benchmarks\n\nSizes are buffer sizes, which can be specified with `NewWithSize`. Default is 64. Tokenizer is re-used between benchmark iterations, but this doesn't impact performance.\n\n`BenchmarkBuiltinDecoder` is `encoding/json.Decoder`.\n\n```\nBenchmarkTokenizer/size=8-8         \t    1419\t    788208 ns/op\t       0 B/op\t       0 allocs/op\nBenchmarkTokenizer/size=16-8         \t    1668\t    688656 ns/op\t       0 B/op\t       0 allocs/op\nBenchmarkTokenizer/size=32-8         \t    1792\t    628601 ns/op\t       0 B/op\t       0 allocs/op\nBenchmarkTokenizer/size=64-8         \t    2040\t    571411 ns/op\t       0 B/op\t       0 allocs/op\nBenchmarkTokenizer/size=128-8        \t    2228\t    520646 ns/op\t       0 B/op\t       0 allocs/op\nBenchmarkTokenizer/size=256-8        \t    2392\t    482151 ns/op\t       0 B/op\t       0 allocs/op\nBenchmarkTokenizer/size=512-8        \t    2516\t    460283 ns/op\t       0 B/op\t       0 allocs/op\nBenchmarkTokenizer/size=1024-8       \t    2553\t    458148 ns/op\t       0 B/op\t       0 allocs/op\nBenchmarkTokenizer/size=2048-8       \t    2618\t    451937 ns/op\t       0 B/op\t       0 allocs/op\nBenchmarkTokenizer/size=4096-8       \t    2499\t    451601 ns/op\t       0 B/op\t       0 allocs/op\nBenchmarkTokenizer/size=8192-8       \t    2610\t    443493 ns/op\t       0 B/op\t       0 allocs/op\n\nBenchmarkBuiltinDecoder-8            \t     157\t   7607729 ns/op\t 1755495 B/op\t  107836 allocs/op\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fpitr%2Fjsontokenizer","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fpitr%2Fjsontokenizer","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fpitr%2Fjsontokenizer/lists"}