{"id":19317988,"url":"https://github.com/prymitive/current","last_synced_at":"2025-04-22T17:30:55.011Z","repository":{"id":59043428,"uuid":"527621697","full_name":"prymitive/current","owner":"prymitive","description":null,"archived":false,"fork":false,"pushed_at":"2024-12-31T15:56:34.000Z","size":507,"stargazers_count":5,"open_issues_count":0,"forks_count":0,"subscribers_count":2,"default_branch":"main","last_synced_at":"2025-04-02T02:05:06.680Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Go","has_issues":false,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/prymitive.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":".github/CODEOWNERS","security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2022-08-22T15:23:39.000Z","updated_at":"2024-12-31T15:56:38.000Z","dependencies_parsed_at":"2023-07-17T20:33:05.917Z","dependency_job_id":"e80c34be-c62f-4a9c-86d2-1fa607ab17d2","html_url":"https://github.com/prymitive/current","commit_stats":{"total_commits":29,"total_committers":3,"mean_commits":9.666666666666666,"dds":"0.24137931034482762","last_synced_commit":"263467a6a7ca2d32a03d1e07e859cc2d10890c59"},"previous_names":["prymitive/jstream"],"tags_count":2,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/prymitive%2Fcurrent","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/prymitive%2Fcurrent/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/prymitive%2Fcurrent/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/prymitive%2Fcurrent/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/prymitive","download_url":"https://codeload.github.com/prymitive/current/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":250287424,"owners_count":21405611,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-10T01:16:51.421Z","updated_at":"2025-04-22T17:30:54.610Z","avatar_url":"https://github.com/prymitive.png","language":"Go","funding_links":[],"categories":[],"sub_categories":[],"readme":"# current - JSON streaming wrapper for encoding/json\n\n## Unmarshalling JSON in Go \n\nGo has many ways of dealing with JSON using standard library.\n[go.dev/blog/json](https://go.dev/blog/json) covers it pretty well.\n\nThe most common is to unmarshal JSON blob into a struct:\n\n```Go\ntype User struct {\n    Name      string `json:\"name\"`\n    Nicknames []string `json:\"nicknames\"`\n    Age       int `json:\"age\"`\n}\n\nvar user User\nvar b []byte\nb = get(...)\njson.Unmarshal(b, \u0026user)\n```\n\nThis handles all of the complexity like dealing with different types etc.\n\nWhen dealing with HTTP responses there is a Decoder we can use to read our response\nbody, so we don't have to:\n\n```Go\nr, err := http.Get(url)\ndefer r.Body.Close()\nvar user User\njson.NewDecoder(r.Body).Decode(user)\n```\n\nIn most cases this is enough, but what if you're dealing with a JSON content that\nis a list of users instead just decoding a single user onto a single struct,\nespecially if that list of users is very long?\n\nCalling `Unmarshal` on the entire list would need the whole response to be read\ninto memory and decoded in a single call, so it would require a lot of memory.\n\nThis is what `json.NewDecoder(r.Body).Decode(user)` would do - it will read\nthe whole response body and then call `Unmarshal` on it.\n\nIf that's just one user then it's perfectly fine, but if we had a huge list\nof users to unmarshal this way and we have limited memory available, then\nideally we would want to avoid having to read the entire response into memory.\n\nLet's assume this is our JSON response to parse:\n\n```JSON\n[\n    {\"name\": \"bob\", \"nicknames\": [\"b\", \"bobby\"], \"age\": 5},\n    {\"name\": \"...\", \"nicknames\": [\"...\", \"...\", \"...\"], \"age\": ...},\n    {\"name\": \"...\", \"nicknames\": [\"...\", \"...\", \"...\"], \"age\": ...},\n    {...}\n]\n```\n\nThe more users there are the bigger the response will be and the more memory will\nbe needed to parse it.\n\n[go.dev/blog/json](https://go.dev/blog/json) mentions streaming, but the example\nprovided is for reading JSON objects pushed into a long lived connection, rather\nthat a single structure of a HTTP response, what it expects is:\n\n```JSON\n{\"name\": \"bob\", \"nicknames\": [\"b\", \"bobby\"], \"age\": 5},\n{\"name\": \"...\", \"nicknames\": [\"...\", \"...\", \"...\"], \"age\": ...},\n{\"name\": \"...\", \"nicknames\": [\"...\", \"...\", \"...\"], \"age\": ...},\n{...}\n```\n\nIf our HTTP response looked like that we could simply call `Decode` in a loop\nuntil there is no more objects to decode:\n\n```Go\ndec := json.NewDecoder(r.Body)\nfor dec.More() {\n    var user User\n    err := dec.Decode(\u0026user)\n}\n```\n\nEach `Decode` call will handle next JSON object:\n\n```JSON\n{\"name\": \"bob\", \"nicknames\": [\"b\", \"bobby\"], \"age\": 5},          \u003c-- Decode()\n{\"name\": \"...\", \"nicknames\": [\"...\", \"...\", \"...\"], \"age\": ...}, \u003c-- Decode()\n{\"name\": \"...\", \"nicknames\": [\"...\", \"...\", \"...\"], \"age\": ...}, \u003c-- Decode()\n{...}\n```\n\n`More()` will peek at what would be the next token and when it finds `]` it will return\n`false`.\n\nThe problem is that we have our array tokens wrapping all objects (`[...]`).\nLuckily we can move ourselves in JSON stream by calling `Token` instead of `Decode`.\n`Decode` call tries to decode next JSON token in a stream onto provided struct.\n`Token` on the other hand simply reads the next token and returns it, it's then our\njob to do something with it.\nWe can use `Token` to navigate in the stream until we're in the right place and then\nstart decoding users.\n\nSo with our users list we would want to read first `[` using `Token` call so that\nwe're in front of our first user, then we hand over decoding to our loop:\n\n```JSON\n[\n    We want to be here \u003c---\n    {\"name\": \"bob\", \"nicknames\": [\"b\", \"bobby\"], \"age\": 5},\n    {\"name\": \"...\", \"nicknames\": [\"...\", \"...\", \"...\"], \"age\": ...},\n    {\"name\": \"...\", \"nicknames\": [\"...\", \"...\", \"...\"], \"age\": ...},\n    {...}\n]\n```\n\n```Go\nfor dec.More() {\n    var user User\n    err := dec.Decode(\u0026user)\n}\n```\n\nSo our final code would be:\n\n```Go\ndec := json.NewDecoder(r.Body)\n\n// first token should be our array opening\nt, err := dec.Token()\nif t != json.Delim('[') {\n    panic(\"Expected [, got %s\", t)\n}\n\nvar users []user\nfor dec.More() {\n    var user User\n    err := dec.Decode(\u0026user)\n    // append user to a slice so we can do something with it\n    users = append(users, user)\n}\n\n// we're done with last user, so we should get array end token next\nt, err := dec.Token()\nif t != json.Delim(']') {\n    panic(\"Expected ], got %s\", t)\n}\n```\n\nGreat success! This code is fairly simple to write and, because we don't load\nthe whole body at once into memory, we limit how much memory is needed to parse\neven a very big file.\n\nThe only problem is that navigating JSON streams can be very error prone, especially\nfor deeply nested JSON blobs. Just imagine that our response is a lot less flat:\n\n```JSON\n{\n    \"status\": \"ok\",\n    \"response\": {\n        \"data\": {\n            \"users\": [\n                {\"name\": \"bob\", \"nicknames\": [\"b\", \"bobby\"], \"age\": 5},\n                {\"name\": \"...\", \"nicknames\": [\"...\", \"...\", \"...\"], \"age\": ...},\n                {\"name\": \"...\", \"nicknames\": [\"...\", \"...\", \"...\"], \"age\": ...},\n                {...}\n            ]\n        }\n    }\n}\n```\n\nWe still need to navigate just before the first user object before we can start\nour `Decode` loop, but now we need to issue multiple `Token` calls and keep track\nwhere we are in the stream.\n\n## github.com/prymitive/current\n\n`current` is helper package that allows you to easily navigate in a JSON stream, so\nyou don't have to manually write all those `Token` calls and keep track of your\nposition. It also uses generics for decoding simple fields, like `\"status\": \"ok\"`\nin the example above.\n\nThanks to current you can parse large JSON responses using a streaming decoder\nand keep your memory usage low.\n\n## Benchmarks\n\nMy initial experience with streaming JSON responses was when I tried to parse\na huge (170MB) JSON response from [Prometheus targets endpoint](https://prometheus.io/docs/prometheus/latest/querying/api/#targets).\nDoing so required ~800MB of heap and caused my service to run out of memory\nand crash. Writing manual streaming code allowed me to make it work with\nvery little memory.\n\nHere is a benchmark using a similar sized targets response.\nIt's broken down by used parser, you can find complete code in the [benchmarks] folder.\n\n```go\nimport (\n\t\"encoding/json\"\n\t\"fmt\"\n\t\"io\"\n\t\"os\"\n\t\"testing\"\n\n\titer \"github.com/json-iterator/go\"\n\t\"github.com/prymitive/current\"\n\t\"github.com/stretchr/testify/require\"\n)\n\ntype Response struct {\n\tStatus string `json:\"status\"`\n\tData   struct {\n\t\tActiveTargets []ActiveTarget `json:\"activeTargets\"`\n\t} `json:\"data\"`\n}\n\ntype ActiveTarget struct {\n\tLabels     map[string]string `json:\"labels\"`\n\tScrapePool string            `json:\"scrapePool\"`\n\tScrapeURL  string            `json:\"scrapeUrl\"`\n}\n\ntype parserFN func(io.Reader) ([]ActiveTarget, error)\n```\n\n### encoding/json\n\n```go\nfunc parseTargetsVanilla(r io.Reader) (targets []ActiveTarget, err error) {\n\tvar resp Response\n\terr = json.NewDecoder(r).Decode(\u0026resp)\n\treturn resp.Data.ActiveTargets, err\n}\n```\n\n### encoding/json with manual streaming code\n\n```go\nfunc parseTargetsStream(r io.Reader) ([]ActiveTarget, error) {\n\tmapStart := json.Delim('{')\n\n\tdec := json.NewDecoder(r)\n\n\t// Expect { as the first token.\n\tt, err := dec.Token()\n\tif err != nil {\n\t\treturn nil, err\n\t}\n\tif t != mapStart {\n\t\treturn nil, fmt.Errorf(\"expected {, got %v\", t)\n\t}\n\n\tresults := []ActiveTarget{}\n\tvar inData, inActiveTargets bool\n\tvar aTarget ActiveTarget\n\tvar key string\n\t// now we need to iterate over input until we reach data-\u003eactiveTargets\n\tfor dec.More() {\n\t\t// we reached the list of active targets so let's decode next target\n\t\t// and append it to results\n\t\tif inData \u0026\u0026 inActiveTargets {\n\t\t\tif err = dec.Decode(\u0026aTarget); err != nil {\n\t\t\t\treturn nil, err\n\t\t\t}\n\t\t\tresults = append(results, aTarget)\n\t\t\taTarget.Labels = map[string]string{}\n\t\t\tcontinue\n\t\t}\n\n\t\t// we didn't hit a target yet, decode current token to see what it is\n\t\tt, err = dec.Token()\n\t\tif err != nil {\n\t\t\treturn nil, err\n\t\t}\n\n\t\t// if this is { then we're almost there (\"data\": {)\n\t\tif t == mapStart {\n\t\t\tcontinue\n\t\t}\n\n\t\tkey = t.(string)\n\t\tswitch key {\n\t\tcase \"data\":\n\t\t\tinData = true\n\t\tcase \"activeTargets\":\n\t\t\tinActiveTargets = true\n\t\t\tif _, err = dec.Token(); err != nil {\n\t\t\t\treturn nil, err\n\t\t\t}\n\t\t}\n\t}\n\n\treturn results, nil\n}\n```\n\n### prymitive/current (this package)\n\n```go\nfunc parseTargetsCurrent(r io.Reader) (targets []ActiveTarget, err error) {\n\ttargets = []ActiveTarget{}\n\tvar target ActiveTarget\n\tdecoder := current.Object(\n\t\tfunc() {},\n\t\tcurrent.Key(\"activeTargets\", current.Array(\u0026target, func() {\n\t\t\ttargets = append(targets, target)\n\t\t\ttarget.Labels = map[string]string{}\n\t\t})),\n\t)\n\n\tdec := json.NewDecoder(r)\n\tif err = decoder.Stream(dec); err != nil {\n\t\treturn nil, err\n\t}\n\n\treturn targets, nil\n}\n```\n\n### github.com/json-iterator/go\n\n```go\nfunc parseTargetsGoIter(r io.Reader) (targets []ActiveTarget, err error) {\n\tvar resp Response\n\terr = iter.NewDecoder(r).Decode(\u0026resp)\n\treturn resp.Data.ActiveTargets, err\n}\n```\n\n### Benchmark code\n\n```go\nfunc BenchmarkTargets(b *testing.B) {\n\tb.ReportAllocs()\n\n\tfor _, tc := range []struct {\n\t\tname string\n\t\tfn   parserFN\n\t}{\n\t\t{name: \"vanilla\", fn: parseTargetsVanilla},\n\t\t{name: \"goiter\", fn: parseTargetsGoIter},\n\t\t{name: \"stream\", fn: parseTargetsStream},\n\t\t{name: \"current\", fn: parseTargetsCurrent},\n\t} {\n\t\tb.Run(tc.name, func(b *testing.B) {\n\t\t\tfor n := 0; n \u003c b.N; n++ {\n\t\t\t\tb.StopTimer()\n\t\t\t\tr, err := os.Open(\"targets.json\")\n\t\t\t\trequire.NoError(b, err)\n\t\t\t\trequire.NoError(b, err)\n\t\t\t\tb.StartTimer()\n\t\t\t\ttargets, err := tc.fn(r)\n\t\t\t\tb.StopTimer()\n\t\t\t\trequire.NoError(b, err)\n\t\t\t\tb.ReportMetric(float64(len(targets)), \"targets\")\n\t\t\t}\n\t\t})\n\t}\n}\n```\n\n### Benchmark results\n\nRunning this benchmark against a real-world response from Prometheus with 157466 targets (165MB JSON file):\n\n```\nBenchmarkTargets/vanilla-8         \t       1\t2062807784 ns/op\t    157466 targets\t830797392 B/op\t 7181439 allocs/op\nBenchmarkTargets/goiter-8          \t       1\t1467535850 ns/op\t    157466 targets\t432962160 B/op\t14387618 allocs/op\nBenchmarkTargets/stream-8          \t       1\t2158223629 ns/op\t    157466 targets\t304209216 B/op\t 7181324 allocs/op\nBenchmarkTargets/current-8         \t       1\t2099241602 ns/op\t    157466 targets\t304203704 B/op\t 7181534 allocs/op\n```\n\nPassed through [benchstat](https://pkg.go.dev/golang.org/x/perf/cmd/benchstat) to make\nit more human readable:\n\n```\nname               time/op\nTargets/vanilla-8  2.13s ±10%\nTargets/goiter-8   1.34s ± 1%\nTargets/stream-8   1.96s ± 1%\nTargets/current-8  2.21s ±19%\n\nname               targets\nTargets/vanilla-8   157k ± 0%\nTargets/goiter-8    157k ± 0%\nTargets/stream-8    157k ± 0%\nTargets/current-8   157k ± 0%\n\nname               alloc/op\nTargets/vanilla-8  831MB ± 0%\nTargets/goiter-8   433MB ± 0%\nTargets/stream-8   304MB ± 0%\nTargets/current-8  304MB ± 0%\n\nname               allocs/op\nTargets/vanilla-8  7.18M ± 0%\nTargets/goiter-8   14.4M ± 0%\nTargets/stream-8   7.18M ± 0%\nTargets/current-8  7.18M ± 0%\n```\n\nRunning same benchmark against mock JSON file generated using\n[examples/benchmarks/mock.go](examples/benchmarks/mock.go) shows similar results:\n\n```\nBenchmarkTargets/vanilla-8         \t       1\t2202727471 ns/op\t    200000 targets\t753842168 B/op\t 7200237 allocs/op\nBenchmarkTargets/goiter-8          \t       1\t1381775190 ns/op\t    200000 targets\t351971712 B/op\t14794136 allocs/op\nBenchmarkTargets/stream-8          \t       1\t2385830895 ns/op\t    200000 targets\t236299544 B/op\t 7200079 allocs/op\nBenchmarkTargets/current-8         \t       1\t2342428154 ns/op\t    200000 targets\t236299848 B/op\t 7200085 allocs/op\n```\n\nSteps to reproduce:\n\n```SHELL\ncd benchmarks\ngo run mock.go\ngo test -run=none -bench=. -benchmem .\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fprymitive%2Fcurrent","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fprymitive%2Fcurrent","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fprymitive%2Fcurrent/lists"}