{"id":13413281,"url":"https://github.com/iOliverNguyen/ujson","last_synced_at":"2025-03-14T19:31:51.249Z","repository":{"id":57561001,"uuid":"172915995","full_name":"iOliverNguyen/ujson","owner":"iOliverNguyen","description":"µjson - A fast and minimal JSON parser and transformer that works on unstructured JSON","archived":false,"fork":false,"pushed_at":"2024-05-26T21:24:36.000Z","size":40,"stargazers_count":76,"open_issues_count":1,"forks_count":10,"subscribers_count":2,"default_branch":"main","last_synced_at":"2024-05-28T23:25:28.250Z","etag":null,"topics":["go","json","minimal","parser","performance"],"latest_commit_sha":null,"homepage":"","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/iOliverNguyen.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2019-02-27T12:58:07.000Z","updated_at":"2024-06-07T08:46:10.699Z","dependencies_parsed_at":"2024-06-07T08:57:37.040Z","dependency_job_id":null,"html_url":"https://github.com/iOliverNguyen/ujson","commit_stats":{"total_commits":16,"total_committers":1,"mean_commits":16.0,"dds":0.0,"last_synced_commit":"510dfd2dfec12696a0bfd824c90aca5f86756862"},"previous_names":["iolivern/ujson","ng-vu/ujson","olvrng/ujson"],"tags_count":4,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/iOliverNguyen%2Fujson","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/iOliverNguyen%2Fujson/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/iOliverNguyen%2Fujson/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/iOliverNguyen%2Fujson/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/iOliverNguyen","download_url":"https://codeload.github.com/iOliverNguyen/ujson/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":221498728,"owners_count":16833055,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["go","json","minimal","parser","performance"],"created_at":"2024-07-30T20:01:36.754Z","updated_at":"2025-03-14T19:31:51.241Z","avatar_url":"https://github.com/iOliverNguyen.png","language":"Go","funding_links":[],"categories":["JSON"],"sub_categories":["Search and Analytic Databases"],"readme":"# µjson\n\n[![GoDoc](http://img.shields.io/badge/go-documentation-blue.svg?style=flat-square)](https://pkg.go.dev/github.com/olvrng/ujson)\n[![License](http://img.shields.io/badge/license-mit-blue.svg?style=flat-square)](https://raw.githubusercontent.com/olvrng/ujson/master/LICENSE)\n[![Go Report Card](https://goreportcard.com/badge/github.com/olvrng/ujson?style=flat-square)](https://goreportcard.com/report/github.com/olvrng/ujson)\n[![Github code coverage](https://img.shields.io/badge/code%20coverage-97%25-brightgreen?style=flat-square)](https://gocover.io/github.com/olvrng/ujson)\n\n---\n\n👉 _Checkout the new library for iterating unstructured JSON with [Go1.23 iterator](https://pkg.go.dev/iter#hdr-Iterators) syntax_ 👈\n\n## [ezpkg.io/iter.json](https://ezpkg.io/iter.json) \u0026middot; A Powerful and Efficient Way to Iterate and Manipulate Unstructured JSON in Go\n\n\u003ca href=\"https://pkg.go.dev/ezpkg.io/iter.json\" target=\"_blank\"\u003e \u003cimg alt=\"godoc\" src=\"https://pkg.go.dev/badge/ezpkg.io/iter.json\" style=\"height: 28px\"\u003e\u003c/a\u003e\n\u003ca href=\"https://github.com/ezpkg/iter.json\" target=\"_blank\"\u003e \u003cimg alt=\"source\" src=\"https://img.shields.io/badge/GitHub-100000?style=for-the-badge\u0026amp;logo=github\u0026amp;logoColor=white\"\u003e \u003c/a\u003e\n\nRead more about [iter.json](https://ezpkg.io/iter.json) on [the introduction article](https://olivernguyen.io/w/iter.json/).\n\nExample usage:\n\n```go\npackage main\n\nimport \"ezpkg.io/iter.json\"\n\nfunc main() {\n    data := `{\"id\": 12345, \"name\": \"foo\", \"numbers\": [\"one\", \"two\"]}`\n    // 👇 iterate over the JSON with Go1.23 iterator syntax\n    for item, err := range iterjson.Parse([]byte(data)) {\n        if err != nil { panic(err) }\n        println(item.GetPathString(), item.Key, item.Token, item.Level)\n    }\n}\n```\n\n---\n\n## µjson\n\n_(is now deprecated and replaced by [iter.json](https://ezpkg.io/iter.json))_\n\nA fast and minimal JSON parser and transformer that works on unstructured JSON.\nIt works by parsing input and calling the given callback function when\nencountering each item.\n\nRead more on [the introduction article](https://hackernoon.com/json-a-minimal-json-parser-and-transformer-in-go-8eo34lp).\n\n## Motivation\n\nSometimes we just want to make some minimal changes to a JSON document, or do\nsome generic transformations without fully unmarshalling it. For example,\nremoving [blacklist fields](https://pkg.go.dev/github.com/olvrng/ujson#example-Walk-RemoveBlacklistFields2)\nfrom response JSON. Why spend all the cost on unmarshalling into a `map[string]interface{}`\njust to immediate marshal it again.\n\nThe following code is taken from [StackOverflow](https://stackoverflow.com/questions/35441254/making-minimal-modification-to-json-data-without-a-structure-in-golang?ref=hackernoon.com):\n\n```json\n{\n  \"responseHeader\": {\n    \"status\": 0,\n    \"QTime\": 0,\n    \"params\": {\n      \"q\": \"solo\",\n      \"wt\": \"json\"\n    }\n  },\n  \"response\": {\n    \"numFound\": 2,\n    \"start\": 0,\n    \"docs\": [\n      { \"name\": \"foo\" },\n      { \"name\": \"bar\" }\n    ]\n  }\n}\n```\n\nWith µjson, we can quickly write [a simple\ntransformation](https://pkg.go.dev/github.com/olvrng/ujson#example-Walk-RemoveBlacklistFields2)\nto remove \"responseHeader\" completely from all responses, once and forever:\n\n```go\nfunc removeBlacklistFields(input []byte) []byte {\nblacklistFields := [][]byte{\n[]byte(`\"responseHeader\"`), // note the quotes\n}\nb := make([]byte, 0, 1024)\nerr := ujson.Walk(input, func(_ int, key, value []byte) bool {\nif len(key) != 0 {\nfor _, blacklist := range blacklistFields {\nif bytes.Equal(key, blacklist) {\n// remove the key and value from the output\nreturn false\n}\n}\n}\n// write to output\nif len(b) != 0 \u0026\u0026 ujson.ShouldAddComma(value, b[len(b)-1]) {\nb = append(b, ',')\n}\nif len(key) \u003e 0 {\nb = append(b, key...)\nb = append(b, ':')\n}\nb = append(b, value...)\nreturn true\n})\nif err != nil {\npanic(err)\n}\nreturn b\n}\n```\n\nThe original scenario that leads me to write the package is because of\n***int64***. When working in Go and PostgreSQL, I use ***int64*** (instead of\n***string***) for ***ids*** because it’s more effective and has enormous space\nfor randomly generated ids. It’s not as big as UUID, 128 bits, but still big\nenough for production use. In PostgreSQL, those ids can be stored as\n***bigint*** and being effectively indexed. But for JavaScript, it can only\nprocess integer up to 53 bits (JavaScript has BigInt but that’s a different\nstory, and using it will make things even more complicated).\n\nSo we need to wrap those int64s into strings before sending them to JavaScript.\nIn Go and PostgreSQL, the JSON is `{\"order_id\": 12345678}` but JavaScript will\nsee it as `{\"order_id\": \"12345678\"}` (note that the value is quoted). In Go, we\ncan define a custom type and implement the\n[`json.Marshaler`](https://golang.org/pkg/encoding/json/#Marshaler) interface. But\nin PostgreSQL, that’s just not possible or too complicated. I wrote a service\nthat receives JSON from PostgreSQL and converts it to be consumable by\nJavaScript. The service also removes some blacklisted keys or does some other\ntransformations (for example, change `orderId` to `order_id`).\n\n### Example use cases:\n\n1. Walk through unstructured JSON:\n    - [Print all keys and values](https://pkg.go.dev/github.com/olvrng/ujson#example-Walk)\n    - Extract some values\n\n2. Transform unstructured JSON:\n    - [Remove all spaces](https://pkg.go.dev/github.com/olvrng/ujson#example-Walk-Reconstruct)\n    - [Reformat](https://pkg.go.dev/github.com/olvrng/ujson#example-Walk-Reformat)\n    - [Remove blacklist fields](https://pkg.go.dev/github.com/olvrng/ujson#example-Walk-RemoveBlacklistFields2)\n    - [Wrap int64 in string for processing by JavaScript](https://pkg.go.dev/github.com/olvrng/ujson#example-Walk-WrapInt64InString)\n\nwithout fully unmarshalling it into a `map[string]interface{}`.\n\nSee usage and examples on\n[pkg.go.dev](https://pkg.go.dev/github.com/olvrng/ujson).\n\n**Important**: *Behaviour is undefined on invalid JSON. Use on trusted input\nonly. For untrusted input, you may want to run it through\n[`json.Valid()`](https://golang.org/pkg/encoding/json/#Valid) first.*\n\n## Usage\n\nThe single most important function is [`Walk(input,\ncallback)`](https://pkg.go.dev/github.com/olvrng/ujson#Walk), which parses the\n`input` JSON and call `callback` function for each key/value pair processed.\n\nThe callback function is called when an object key/value or an array key is\nencountered. It receives 3 params in order: `level`, `key` and `value`.\n\n- `level` is the indentation level of the JSON, if you format it properly. It\n  starts from 0. It increases after entering an object or array and decreases\n  after leaving.\n\n- `key` is the raw key of the current object or empty otherwise. It can be a\n  double-quoted string or empty.\n\n- `value` is the raw value of the current item or a bracket. It can be a string,\n  number, boolean, null, or one of the following brackets: `{ } [ ]`.\n\nIt’s important to note that `key` and `value` are provided as raw. Strings are\nalways double-quoted. It’s there for keeping the library fast and ignoring\nunnecessary operations. For example, when you only want to reformat the output\nJSON properly; you don’t want to unquote those strings and then immediately\nquote them again; you just need to output them unmodified. And there are\n[`ujson.Unquote()`](https://pkg.go.dev/github.com/olvrng/ujson#Unquote) and\n[`ujson.AppendQuote()`](https://pkg.go.dev/github.com/olvrng/ujson#AppendQuote)\nwhen you need to get the original strings.\n\nFor valid JSON, values will never be empty. We can test the first byte of value\n(`value[0]`) to get its type:\n\n- `n`: Null (`null`)\n- `f`, `t`: Boolean (`false`, `true`)\n- `0`-`9`, `-`: Number\n- `\"`: String, see [`Unquote()`](https://pkg.go.dev/github.com/olvrng/ujson#Unquote)\n- `[`, `]`: Array\n- `{`, `}`: Object\n\nWhen processing arrays and objects, first the open bracket (`[`, `{`) will be\nprovided as `value`, followed by its children, and finally the close bracket\n(`]`, `}`). When encountering open brackets, You can make the callback function\nreturn `false` to skip the array/object entirely.\n\n## Examples\n\n### 1. Print all keys and values in order\n\nThis example gives a quick idea about how *µjson* works.\n\n```go \n input := []byte(`{  \n     \"id\": 12345,    \n     \"name\": \"foo\",  \n     \"numbers\": [\"one\", \"two\"],  \n     \"tags\": {\"color\": \"red\", \"priority\": \"high\"},   \n     \"active\": true  \n }`)\nujson.Walk(input, func(level int, key, value []byte) bool {\nfmt.Printf(\"%2v% 12s : %s\\n\", level, key, value)\nreturn true\n})\n```\n\n```json\n{\n  \"id\": 12345,\n  \"name\": \"foo\",\n  \"numbers\": [\"one\", \"two\"],\n  \"tags\": {\"color\": \"red\", \"priority\": \"high\"},\n  \"active\": true\n}\n```\n\nCalling `Walk()` with the above input will produce:\n\n| level | key        | value   |\n|:-----:|:----------:|:-------:|\n|`0`    |            |`{`      |\n|`1`    |`\"id\"`      |`12345`  |\n|`1`    |`\"name\"`    |`\"foo\"`  |\n|`1`    |`\"numbers\"` |`[`      |\n|`2`    |            |`\"one\"`  |\n|`2`    |            |`\"two\"`  |\n|`1`    |            |`]`      |\n|`1`    |`\"tags\"`    |`{`      |\n|`2`    |`\"color\"`   |`\"red\"`  |\n|`2`    |`\"priority\"`|`\"high\"` |\n|`1`    |            |`}`      |\n|`1`    |`\"active\"`  |`true`   |\n|`0`    |            |`}`      |\n\n### 0. The simplest examples\n\nTo easily get an idea on `level`, `key` and `value`, here are the simplest\nexamples:\n\n```go\n input0 := []byte(`true`)\nujson.Walk(input0, func(level int, key, value []byte) bool {\nfmt.Printf(\"level=%v key=%s value=%s\\n\", level, key, value)\nreturn true\n})\n// output:\n//   level=0 key= value=true\n\ninput1 := []byte(`{ \"key\": 42 }`)\nujson.Walk(input1, func(level int, key, value []byte) bool {\nfmt.Printf(\"level=%v key=%s value=%s\\n\", level, key, value)\nreturn true\n})\n// output:\n//   level=0 key= value={\n//   level=1 key=\"key\" value=42\n//   level=0 key= value=}\n\ninput2 := []byte(`[ true ]`)\nujson.Walk(input2, func(level int, key, value []byte) bool {\nfmt.Printf(\"level=%v key=%s value=%s\\n\", level, key, value)\nreturn true\n})\n// output:\n//   level=0 key= value=[\n//   level=1 key= value=true\n//   level=0 key= value=]\n```\n\nIn the first example, there is only a single boolean value. The callback\nfunction is called once with `level=0`, `key` is empty and `value=true`.\n\nIn the second example, the callback function is called 3 times. Two times for\nopen and close brackets with `level=0`, `key` is empty and `value` is the\nbracketed character. The other time for the only key with `level=1`, `key` is\n`\"key\"` and `value=42`. Note that the key is quoted and you need to call\n[`ujson.Unquote()`](https://pkg.go.dev/github.com/olvrng/ujson#Unquote) to\nretrieve the unquoted string.\n\nThe last example is like the second, but with an array instead. Keys are always\nempty inside arrays.\n\n### 2. Reformat input\n\nIn this example, the input JSON is formatted with correct indentation. As\nprocessing the input key by key, the callback function reconstructs the JSON. It\noutputs each key/value pair in its own line, prefixed with spaces equal to the\nparam level. There is a catch, though. Valid JSON requires commas between values\nin objects and arrays. So there is\n[`ujson.ShouldAddComma()`](https://pkg.go.dev/github.com/olvrng/ujson#ShouldAddComma)\nfor checking whether a comma should be inserted.\n\n```go\n input := []byte(`{\"id\":12345,\"name\":\"foo\",\"numbers\":[\"one\",\"two\"],\"tags\":{\"color\":\"red\",\"priority\":\"high\"},\"active\":true}`)\n\nb := make([]byte, 0, 1024)\nerr := ujson.Walk(input, func(level int, key, value []byte) bool {\nif len(b) != 0 \u0026\u0026 ujson.ShouldAddComma(value, b[len(b)-1]) {\nb = append(b, ',')\n}\nb = append(b, '\\n')\nfor i := 0; i \u003c level; i++ {\nb = append(b, '\\t')\n}\nif len(key) \u003e 0 {\nb = append(b, key...)\nb = append(b, `: `...)\n}\nb = append(b, value...)\nreturn true\n})\nif err != nil {\npanic(err)\n}\nfmt.Printf(\"%s\\n\", b)\n```\n\n#### Output:\n\n```json\n{\n  \"id\": 12345,\n  \"name\": \"foo\",\n  \"numbers\": [\n    \"one\",\n    \"two\"\n  ],\n  \"tags\": {\n    \"color\": \"red\",\n    \"priority\": \"high\"\n  },\n  \"active\": true\n}\n```\n\nThere is a built-in method\n[`ujson.Reconstruct()`](https://pkg.go.dev/github.com/olvrng/ujson?ref=hackernoon.com#Reconstruct)\nwhen you want to remove all the whitespaces.\n\n### 3. Remove blacklisted keys\n\nThis example demonstrates removing some keys from the input JSON. The key param\nis compared with a pre-defined list. If there is a match, the blacklisted key\nand its value are dropped. The callback function returns false for skipping the\nentire value (which may be an object or array). Note that the list is quoted,\ni.e. \"numbers\" and \"active\" instead of number and active. For more advanced\nchecking, you may want to run [`ujson.Unquote()`](https://pkg.go.dev/github.com/olvrng/ujson#Unquote) on the key.\n\n```go\n input := []byte(`{\n     \"id\": 12345,\n     \"name\": \"foo\",\n     \"numbers\": [\"one\", \"two\"],\n     \"tags\": {\"color\": \"red\", \"priority\": \"high\"},\n     \"active\": true\n }`)\n\nblacklistFields := [][]byte{\n[]byte(`\"numbers\"`), // note the quotes\n[]byte(`\"active\"`),\n}\nb := make([]byte, 0, 1024)\nerr := ujson.Walk(input, func(_ int, key, value []byte) bool {\nfor _, blacklist := range blacklistFields {\nif bytes.Equal(key, blacklist) {\n// remove the key and value from the output\nreturn false\n}\n}\n\n// write to output\nif len(b) != 0 \u0026\u0026 ujson.ShouldAddComma(value, b[len(b)-1]) {\nb = append(b, ',')\n}\nif len(key) \u003e 0 {\nb = append(b, key...)\nb = append(b, ':')\n}\nb = append(b, value...)\nreturn true\n})\nif err != nil {\npanic(err)\n}\nfmt.Printf(\"%s\\n\", b)\n```\n\n#### Output:\n\n```json\n{\"id\":12345,\"name\":\"foo\",\"tags\":{\"color\":\"red\",\"priority\":\"high\"}}\n```\n\n### 4. Wrap int64 in string\n\nThis is the original motivation behind µjson. The following example finds keys\nending with `_id\"` (`\"order_id\"`, `\"item_id\"`, etc.) and converts their values\nfrom numbers to strings, by simply wrapping them in double-quotes.\n\nFor valid JSON, values will never be empty. We can test the first byte of value\n(`value[0]`) to get its type:\n\n- `n`: Null (`null`)\n- `f`, `t`: Boolean (`false`, `true`)\n- `0`-`9`, `-`: Number\n- `\"`: String, see [`Unquote()`](https://pkg.go.dev/github.com/olvrng/ujson#Unquote)\n- `[`, `]`: Array\n- `{`, `}`: Object\n\nIn this case, we check `value[0]` within `0`…`9` to see whether it’s a number,\nthen insert double-quotes.\n\n```go\n input := []byte(`{\"order_id\": 12345678901234, \"number\": 12, \"item_id\": 12345678905678, \"counting\": [1,\"2\",3]}`)\n\nsuffix := []byte(`_id\"`) // note the ending quote \"\nb := make([]byte, 0, 256)\nerr := ujson.Walk(input, func(_ int, key, value []byte) bool {\n// Test for keys with suffix _id\" and value is an int64 number. For valid json,\n// values will never be empty, so we can safely test only the first byte.\nshouldWrap := bytes.HasSuffix(key, suffix) \u0026\u0026 value[0] \u003e '0' \u0026\u0026 value[0] \u003c= '9'\n\n// transform the input, wrap values in double quotes\nif len(b) != 0 \u0026\u0026 ujson.ShouldAddComma(value, b[len(b)-1]) {\nb = append(b, ',')\n}\nif len(key) \u003e 0 {\nb = append(b, key...)\nb = append(b, ':')\n}\nif shouldWrap {\nb = append(b, '\"')\n}\nb = append(b, value...)\nif shouldWrap {\nb = append(b, '\"')\n}\nreturn true\n})\nif err != nil {\npanic(err)\n}\nfmt.Printf(\"%s\\n\", b)\n```\n\n#### Output:\n\n```json\n{\"order_id\":\"12345678901234\",\"number\":12,\"item_id\":\"12345678905678\",\"counting\":[1,\"2\",3]}\n```\n\nAfter processing, the numbers in `\"order_id\"` and `\"item_id\"` are quoted as strings. And JavaScript should be happy now! 🎉 🎉\n\n## Author\n\n\u003ca href=\"https://olivernguyen.io\"\u003e\u003cimg alt=\"olivernguyen.io\" src=\"https://olivernguyen.io/_/badge.png\" height=\"28px\"\u003e\u003c/a\u003e\n\n## License\n\nMIT\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FiOliverNguyen%2Fujson","html_url":"https://awesome.ecosyste.ms/projects/github.com%2FiOliverNguyen%2Fujson","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FiOliverNguyen%2Fujson/lists"}