{"id":13481297,"url":"https://github.com/goccy/go-json","last_synced_at":"2025-05-13T15:01:54.804Z","repository":{"id":38224004,"uuid":"256961429","full_name":"goccy/go-json","owner":"goccy","description":"Fast JSON encoder/decoder compatible with encoding/json for Go","archived":false,"fork":false,"pushed_at":"2025-01-28T01:49:03.000Z","size":4819,"stargazers_count":3349,"open_issues_count":115,"forks_count":165,"subscribers_count":22,"default_branch":"master","last_synced_at":"2025-05-05T22:14:15.586Z","etag":null,"topics":["go","golang","golang-library","json"],"latest_commit_sha":null,"homepage":null,"language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/goccy.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":null,"funding":".github/FUNDING.yml","license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null},"funding":{"github":["goccy"]}},"created_at":"2020-04-19T09:32:36.000Z","updated_at":"2025-05-05T14:16:44.000Z","dependencies_parsed_at":"2024-05-21T11:00:25.344Z","dependency_job_id":"908b5f10-2da0-4bcd-850b-ab66487b9b16","html_url":"https://github.com/goccy/go-json","commit_stats":{"total_commits":830,"total_committers":24,"mean_commits":"34.583333333333336","dds":"0.13734939759036147","last_synced_commit":"5e2ae3f23c1db71990844a230a4d11559efe128e"},"previous_names":[],"tags_count":77,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/goccy%2Fgo-json","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/goccy%2Fgo-json/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/goccy%2Fgo-json/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/goccy%2Fgo-json/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/goccy","download_url":"https://codeload.github.com/goccy/go-json/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":253967861,"owners_count":21992251,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["go","golang","golang-library","json"],"created_at":"2024-07-31T17:00:50.583Z","updated_at":"2025-05-13T15:01:54.758Z","avatar_url":"https://github.com/goccy.png","language":"Go","readme":"# go-json\n\n![Go](https://github.com/goccy/go-json/workflows/Go/badge.svg)\n[![GoDoc](https://godoc.org/github.com/goccy/go-json?status.svg)](https://pkg.go.dev/github.com/goccy/go-json?tab=doc)\n[![codecov](https://codecov.io/gh/goccy/go-json/branch/master/graph/badge.svg)](https://codecov.io/gh/goccy/go-json)\n\nFast JSON encoder/decoder compatible with encoding/json for Go\n\n\u003cimg width=\"400px\" src=\"https://user-images.githubusercontent.com/209884/92572337-42b42900-f2bf-11ea-973a-c74a359553a5.png\"\u003e\u003c/img\u003e\n\n# Roadmap\n\n```\n* version ( expected release date )\n\n* v0.9.0\n |\n | while maintaining compatibility with encoding/json, we will add convenient APIs\n |\n v\n* v1.0.0\n```\n\nWe are accepting requests for features that will be implemented between v0.9.0 and v.1.0.0.\nIf you have the API you need, please submit your issue [here](https://github.com/goccy/go-json/issues).\n\n# Features\n\n- Drop-in replacement of `encoding/json`\n- Fast ( See [Benchmark section](https://github.com/goccy/go-json#benchmarks) )\n- Flexible customization with options\n- Coloring the encoded string\n- Can propagate context.Context to `MarshalJSON` or `UnmarshalJSON`\n- Can dynamically filter the fields of the structure type-safely\n\n# Installation\n\n```\ngo get github.com/goccy/go-json\n```\n\n# How to use\n\nReplace import statement from `encoding/json` to `github.com/goccy/go-json`\n\n```\n-import \"encoding/json\"\n+import \"github.com/goccy/go-json\"\n```\n\n# JSON library comparison\n\n|  name  |  encoder | decoder | compatible with `encoding/json` |\n| :----: | :------: | :-----: | :-----------------------------: |\n| encoding/json |  yes | yes | N/A |\n| [json-iterator/go](https://github.com/json-iterator/go) | yes | yes | partial |\n| [easyjson](https://github.com/mailru/easyjson) | yes | yes |  no |\n| [gojay](https://github.com/francoispqt/gojay) | yes | yes |  no |\n| [segmentio/encoding/json](https://github.com/segmentio/encoding/tree/master/json) | yes | yes | partial |\n| [jettison](https://github.com/wI2L/jettison) | yes | no | no |\n| [simdjson-go](https://github.com/minio/simdjson-go) | no | yes | no |\n| goccy/go-json | yes | yes | yes |\n\n- `json-iterator/go` isn't compatible with `encoding/json` in many ways (e.g. https://github.com/json-iterator/go/issues/229 ), but it hasn't been supported for a long time.\n- `segmentio/encoding/json` is well supported for encoders, but some are not supported for decoder APIs such as `Token` ( streaming decode )\n\n## Other libraries\n\n- [jingo](https://github.com/bet365/jingo)\n\nI tried the benchmark but it didn't work.\nAlso, it seems to panic when it receives an unexpected value because there is no error handling...\n\n- [ffjson](https://github.com/pquerna/ffjson)\n\nBenchmarking gave very slow results.\nIt seems that it is assumed that the user will use the buffer pool properly.\nAlso, development seems to have already stopped\n\n# Benchmarks\n\n```\n$ cd benchmarks\n$ go test -bench .\n```\n\n## Encode\n\n\u003cimg width=\"700px\" src=\"https://user-images.githubusercontent.com/209884/107126758-0845cb00-68f5-11eb-8db7-086fcf9bcfaa.png\"\u003e\u003c/img\u003e\n\u003cimg width=\"700px\" src=\"https://user-images.githubusercontent.com/209884/107126757-07ad3480-68f5-11eb-87aa-858cc5eacfcb.png\"\u003e\u003c/img\u003e\n\n## Decode\n\n\u003cimg width=\"700\" alt=\"\" src=\"https://user-images.githubusercontent.com/209884/107979944-bd1d6d80-7002-11eb-944b-9d17b6674e3f.png\"\u003e\n\u003cimg width=\"700\" alt=\"\" src=\"https://user-images.githubusercontent.com/209884/107979931-b989e680-7002-11eb-87a0-66fc22d90dd4.png\"\u003e\n\u003cimg width=\"700\" alt=\"\" src=\"https://user-images.githubusercontent.com/209884/107979940-bc84d700-7002-11eb-9647-869bbc25c9d9.png\"\u003e\n\n\n# Fuzzing\n\n[go-json-fuzz](https://github.com/goccy/go-json-fuzz) is the repository for fuzzing tests.\nIf you run the test in this repository and find a bug, please commit to corpus to go-json-fuzz and report the issue to [go-json](https://github.com/goccy/go-json/issues).\n\n# How it works\n\n`go-json` is very fast in both encoding and decoding compared to other libraries.\nIt's easier to implement by using automatic code generation for performance or by using a dedicated interface, but `go-json` dares to stick to compatibility with `encoding/json` and is the simple interface. Despite this, we are developing with the aim of being the fastest library.\n\nHere, we explain the various speed-up techniques implemented by `go-json`.\n\n## Basic technique\n\nThe techniques listed here are the ones used by most of the libraries listed above.\n\n### Buffer reuse\n\nSince the only value required for the result of `json.Marshal(interface{}) ([]byte, error)` is `[]byte`, the only value that must be allocated during encoding is the return value `[]byte` .\n\nAlso, as the number of allocations increases, the performance will be affected, so the number of allocations should be kept as low as possible when creating `[]byte`.\n\nTherefore, there is a technique to reduce the number of times a new buffer must be allocated by reusing the buffer used for the previous encoding by using `sync.Pool`.\n\nFinally, you allocate a buffer that is as long as the resulting buffer and copy the contents into it, you only need to allocate the buffer once in theory.\n\n```go\ntype buffer struct {\n    data []byte\n}\n\nvar bufPool = sync.Pool{\n    New: func() interface{} {\n        return \u0026buffer{data: make([]byte, 0, 1024)}\n    },\n}\n\nbuf := bufPool.Get().(*buffer)\ndata := encode(buf.data) // reuse buf.data\n\nnewBuf := make([]byte, len(data))\ncopy(newBuf, buf)\n\nbuf.data = data\nbufPool.Put(buf)\n```\n\n### Elimination of reflection\n\nAs you know, the reflection operation is very slow.\n\nTherefore, using the fact that the address position where the type information is stored is fixed for each binary ( we call this `typeptr` ),\nwe can use the address in the type information to call a pre-built optimized process.\n\nFor example, you can get the address to the type information from `interface{}` as follows and you can use that information to call a process that does not have reflection.\n\nTo process without reflection, pass a pointer (`unsafe.Pointer`) to the value is stored.\n\n```go\n\ntype emptyInterface struct {\n    typ unsafe.Pointer\n    ptr unsafe.Pointer\n}\n\nvar typeToEncoder = map[uintptr]func(unsafe.Pointer)([]byte, error){}\n\nfunc Marshal(v interface{}) ([]byte, error) {\n    iface := (*emptyInterface)(unsafe.Pointer(\u0026v)\n    typeptr := uintptr(iface.typ)\n    if enc, exists := typeToEncoder[typeptr]; exists {\n        return enc(iface.ptr)\n    }\n    ...\n}\n```\n\n※ In reality, `typeToEncoder` can be referenced by multiple goroutines, so exclusive control is required.\n\n## Unique speed-up technique\n\n## Encoder\n\n### Do not escape arguments of `Marshal`\n\n`json.Marshal` and `json.Unmarshal` receive `interface{}` value and they perform type determination dynamically to process.\nIn normal case, you need to use the `reflect` library to determine the type dynamically, but since `reflect.Type` is defined as `interface`, when you call the method of `reflect.Type`, The reflect's argument is escaped.\n\nTherefore, the arguments for `Marshal` and `Unmarshal` are always escaped to the heap.\nHowever, `go-json` can use the feature of `reflect.Type` while avoiding escaping.\n\n`reflect.Type` is defined as `interface`, but in reality `reflect.Type` is implemented only by the structure `rtype` defined in the `reflect` package.\nFor this reason, to date `reflect.Type` is the same as `*reflect.rtype`.\n\nTherefore, by directly handling `*reflect.rtype`, which is an implementation of `reflect.Type`, it is possible to avoid escaping because it changes from `interface` to using `struct`.\n\nThe technique for working with `*reflect.rtype` directly from `go-json` is implemented at [rtype.go](https://github.com/goccy/go-json/blob/master/internal/runtime/rtype.go)\n\nAlso, the same technique is cut out as a library ( https://github.com/goccy/go-reflect )\n\nInitially this feature was the default behavior of `go-json`.\nBut after careful testing, I found that I passed a large value to `json.Marshal()` and if the argument could not be assigned to the stack, it could not be properly escaped to the heap (a bug in the Go compiler).\n\nTherefore, this feature will be provided as an **optional** until this issue is resolved.\n\nTo use it, add `NoEscape` like `MarshalNoEscape()`\n\n### Encoding using opcode sequence\n\nI explained that you can use `typeptr` to call a pre-built process from type information.\n\nIn other libraries, this dedicated process is processed by making it an function calling like anonymous function, but function calls are inherently slow processes and should be avoided as much as possible.\n\nTherefore, `go-json` adopted the Instruction-based execution processing system, which is also used to implement virtual machines for programming language.\n\nIf it is the first type to encode, create the opcode ( instruction ) sequence required for encoding.\nFrom the second time onward, use `typeptr` to get the cached pre-built opcode sequence and encode it based on it. An example of the opcode sequence is shown below.\n\n```go\njson.Marshal(struct{\n    X int `json:\"x\"`\n    Y string `json:\"y\"`\n}{X: 1, Y: \"hello\"})\n```\n\nWhen encoding a structure like the one above, create a sequence of opcodes like this:\n\n```\n- opStructFieldHead ( `{` )\n- opStructFieldInt ( `\"x\": 1,` )\n- opStructFieldString ( `\"y\": \"hello\"` )\n- opStructEnd ( `}` )\n- opEnd\n```\n\n※ When processing each operation, write the letters on the right.\n\nIn addition, each opcode is managed by the following structure ( \nPseudo code ).\n\n```go\ntype opType int\nconst (\n    opStructFieldHead opType = iota\n    opStructFieldInt\n    opStructFieldStirng\n    opStructEnd\n    opEnd\n)\ntype opcode struct {\n    op opType\n    key []byte\n    next *opcode\n}\n```\n\nThe process of encoding using the opcode sequence is roughly implemented as follows.\n\n```go\nfunc encode(code *opcode, b []byte, p unsafe.Pointer) ([]byte, error) {\n    for {\n        switch code.op {\n        case opStructFieldHead:\n            b = append(b, '{')\n            code = code.next\n        case opStructFieldInt:\n            b = append(b, code.key...)\n            b = appendInt((*int)(unsafe.Pointer(uintptr(p)+code.offset)))\n            code = code.next\n        case opStructFieldString:\n            b = append(b, code.key...)\n            b = appendString((*string)(unsafe.Pointer(uintptr(p)+code.offset)))\n            code = code.next\n        case opStructEnd:\n            b = append(b, '}')\n            code = code.next\n        case opEnd:\n            goto END\n        }\n    }\nEND:\n    return b, nil\n}\n```\n\nIn this way, the huge `switch-case` is used to encode by manipulating the linked list opcodes to avoid unnecessary function calls.\n\n### Opcode sequence optimization\n\nOne of the advantages of encoding using the opcode sequence is the ease of optimization.\nThe opcode sequence mentioned above is actually converted into the following optimized operations and used.\n\n```\n- opStructFieldHeadInt ( `{\"x\": 1,` )\n- opStructEndString ( `\"y\": \"hello\"}` )\n- opEnd\n```\n\nIt has been reduced from 5 opcodes to 3 opcodes !\nReducing the number of opcodees means reducing the number of branches with `switch-case`.\nIn other words, the closer the number of operations is to 1, the faster the processing can be performed.\n\nIn `go-json`, optimization to reduce the number of opcodes itself like the above and it speeds up by preparing opcodes with optimized paths.\n\n### Change recursive call from CALL to JMP\n\nRecursive processing is required during encoding if the type is defined recursively as follows:\n\n```go\ntype T struct {\n    X int\n    U *U\n}\n\ntype U struct {\n    T *T\n}\n\nb, err := json.Marshal(\u0026T{\n    X: 1,\n    U: \u0026U{\n        T: \u0026T{\n            X: 2,\n        },\n    },\n})\nfmt.Println(string(b)) // {\"X\":1,\"U\":{\"T\":{\"X\":2,\"U\":null}}}\n```\n\nIn `go-json`, recursive processing is processed by the operation type of ` opStructFieldRecursive`.\n\nIn this operation, after acquiring the opcode sequence used for recursive processing, the function is **not** called recursively as it is, but the necessary values ​​are saved by itself and implemented by moving to the next operation.\n\nThe technique of implementing recursive processing with the `JMP` operation while avoiding the `CALL` operation is a famous technique for implementing a high-speed virtual machine.\n\nFor more details, please refer to [the article](https://engineering.mercari.com/blog/entry/1599563768-081104c850) ( but Japanese only ).\n\n### Dispatch by typeptr from map to slice\n\nWhen retrieving the data cached from the type information by `typeptr`, we usually use map.\nMap requires exclusive control, so use `sync.Map` for a naive implementation.\n\nHowever, this is slow, so it's a good idea to use the `atomic` package for exclusive control as implemented by `segmentio/encoding/json` ( https://github.com/segmentio/encoding/blob/master/json/codec.go#L41-L55 ).\n\nThis implementation slows down the set instead of speeding up the get, but it works well because of the nature of the library, it encodes much more for the same type.\n\nHowever, as a result of profiling, I noticed that `runtime.mapaccess2` accounts for a significant percentage of the execution time. So I thought if I could change the lookup from map to slice.\n\nThere is an API named `typelinks` defined in the `runtime` package that the `reflect` package uses internally.\nThis allows you to get all the type information defined in the binary at runtime.\n\nThe fact that all type information can be acquired means that by constructing slices in advance with the acquired total number of type information, it is possible to look up with the value of `typeptr` without worrying about out-of-range access.\n\nHowever, if there is too much type information, it will use a lot of memory, so by default we will only use this optimization if the slice size fits within **2Mib** .\n\nIf this approach is not available, it will fall back to the `atomic` based process described above.\n\nIf you want to know more, please refer to the implementation [here](https://github.com/goccy/go-json/blob/master/internal/runtime/type.go#L36-L100)\n\n## Decoder\n\n### Dispatch by typeptr from map to slice\n\nLike the encoder, the decoder also uses typeptr to call the dedicated process.\n\n### Faster termination character inspection using NUL character\n\nIn order to decode, you have to traverse the input buffer character by position.\nAt that time, if you check whether the buffer has reached the end, it will be very slow.\n\n`buf` : `[]byte` type variable. holds the string passed to the decoder\n`cursor` : `int64` type variable. holds the current read position\n\n```go\nbuflen := len(buf)\nfor ; cursor \u003c buflen; cursor++ { // compare cursor and buflen at all times, it is so slow.\n    switch buf[cursor] {\n    case ' ', '\\n', '\\r', '\\t':\n    }\n}\n```\n\nTherefore, by adding the `NUL` (`\\000`) character to the end of the read buffer as shown below, it is possible to check the termination character at the same time as other characters.\n\n```go\nfor {\n    switch buf[cursor] {\n    case ' ', '\\n', '\\r', '\\t':\n    case '\\000':\n        return nil\n    }\n    cursor++\n}\n```\n\n### Use Boundary Check Elimination\n\nDue to the `NUL` character optimization, the Go compiler does a boundary check every time, even though `buf[cursor]` does not cause out-of-range access.\n\nTherefore, `go-json` eliminates boundary check by fetching characters for hotspot by pointer operation. For example, the following code.\n\n```go\nfunc char(ptr unsafe.Pointer, offset int64) byte {\n\treturn *(*byte)(unsafe.Pointer(uintptr(ptr) + uintptr(offset)))\n}\n\np := (*sliceHeader)(\u0026unsafe.Pointer(buf)).data\nfor {\n    switch char(p, cursor) {\n    case ' ', '\\n', '\\r', '\\t':\n    case '\\000':\n        return nil\n    }\n    cursor++\n}\n```\n\n### Checking the existence of fields of struct using Bitmaps\n\nI found by the profiling result, in the struct decode, lookup process for field was taking a long time.\n\nFor example, consider decoding a string like `{\"a\":1,\"b\":2,\"c\":3}` into the following structure:\n\n```go\ntype T struct {\n    A int `json:\"a\"`\n    B int `json:\"b\"`\n    C int `json:\"c\"`\n}\n```\n\nAt this time, it was found that it takes a lot of time to acquire the decoding process corresponding to the field from the field name as shown below during the decoding process.\n\n```go\nfieldName := decodeKey(buf, cursor) // \"a\" or \"b\" or \"c\"\ndecoder, exists := fieldToDecoderMap[fieldName] // so slow\nif exists {\n    decoder(buf, cursor)\n} else {\n    skipValue(buf, cursor)\n}\n```\n\nTo improve this process, `json-iterator/go` is optimized so that it can be branched by switch-case when the number of fields in the structure is 10 or less (switch-case is faster than map). However, there is a risk of hash collision because the value hashed by the FNV algorithm is used for conditional branching. Also, `gojay` processes this part at high speed by letting the library user yourself write `switch-case`.\n\n\n`go-json` considers and implements a new approach that is different from these. I call this **bitmap field optimization**.\n\nThe range of values ​​per character can be represented by `[256]byte`. Also, if the number of fields in the structure is 8 or less, `int8` type can represent the state of each field.\nIn other words, it has the following structure.\n\n- Base ( 8bit ): `00000000`\n- Key \"a\": `00000001` ( assign key \"a\" to the first bit )\n- Key \"b\": `00000010` ( assign key \"b\" to the second bit )\n- Key \"c\": `00000100` ( assign key \"c\" to the third bit )\n\nBitmap structure is the following\n\n```\n        | key index(0) |\n------------------------\n 0      | 00000000     |\n 1      | 00000000     |\n~~      |              |\n97 (a)  | 00000001     |\n98 (b)  | 00000010     |\n99 (c)  | 00000100     |\n~~      |              |\n255     | 00000000     |\n```\n\nYou can think of this as a Bitmap with a height of `256` and a width of the maximum string length in the field name.\nIn other words, it can be represented by the following type .\n\n```go\n[maxFieldKeyLength][256]int8\n```\n\nWhen decoding a field character, check whether the corresponding character exists by referring to the pre-built bitmap like the following.\n\n```go\nvar curBit int8 = math.MaxInt8 // 11111111\n\nc := char(buf, cursor)\nbit := bitmap[keyIdx][c]\ncurBit \u0026= bit\nif curBit == 0 {\n    // not found field\n}\n```\n\nIf `curBit` is not `0` until the end of the field string, then the string is\nYou may have hit one of the fields.\nBut the possibility is that if the decoded string is shorter than the field string, you will get a false hit.\n\n- input: `{\"a\":1}`\n```go\ntype T struct {\n    X int `json:\"abc\"`\n}\n```\n※ Since `a` is shorter than `abc`, it can decode to the end of the field character without `curBit` being 0.\n\nRest assured. In this case, it doesn't matter because you can tell if you hit by comparing the string length of `a` with the string length of `abc`.\n\nFinally, calculate the position of the bit where `1` is set and get the corresponding value, and you're done.\n\nUsing this technique, field lookups are possible with only bitwise operations and access to slices.\n\n`go-json` uses a similar technique for fields with 9 or more and 16 or less fields. At this time, Bitmap is constructed as `[maxKeyLen][256]int16` type.\n\nCurrently, this optimization is not performed when the maximum length of the field name is long (specifically, 64 bytes or more) in addition to the limitation of the number of fields from the viewpoint of saving memory usage.\n\n### Others\n\nI have done a lot of other optimizations. I will find time to write about them. If you have any questions about what's written here or other optimizations, please visit the `#go-json` channel on `gophers.slack.com` .\n\n## Reference\n\nRegarding the story of go-json, there are the following articles in Japanese only.\n\n- https://speakerdeck.com/goccy/zui-su-falsejsonraiburariwoqiu-mete\n- https://engineering.mercari.com/blog/entry/1599563768-081104c850/\n\n# Looking for Sponsors\n\nI'm looking for sponsors this library. This library is being developed as a personal project in my spare time. If you want a quick response or problem resolution when using this library in your project, please register as a [sponsor](https://github.com/sponsors/goccy). I will cooperate as much as possible. Of course, this library is developed as an MIT license, so you can use it freely for free.\n\n# License\n\nMIT\n","funding_links":["https://github.com/sponsors/goccy"],"categories":["开源类库","Go","JSON parsers \u0026 validators","Open source library","Repositories","JSON 处理"],"sub_categories":["JSON"],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fgoccy%2Fgo-json","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fgoccy%2Fgo-json","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fgoccy%2Fgo-json/lists"}