Data

Tools

Indexes

Applications

Experiments

Awesome

An open API service indexing awesome lists of open source software.

https://github.com/johanlindvall/lightning

A fast Go JSON unmarshaler
https://github.com/johanlindvall/lightning

go json simd

Last synced: 16 days ago
JSON representation

A fast Go JSON unmarshaler

Awesome Lists containing this project

README 

          
# Lightning ⚡



A small Go code generator that emits fast, allocation-light

`json.Unmarshaler` implementations from your struct definitions.



Instead of decoding JSON with reflection at run time (like `encoding/json`),

lightning reads a struct definition at build time and writes a hand-written

style `UnmarshalJSON` method plus the recursive decoders it needs. The decoders

share a single set of scanning primitives in [`pkg/support`](pkg/support), so the

generated files stay small.



## Installation



Run it straight from the module path (no clone needed):



```sh

go run github.com/JohanLindvall/lightning@latest path/to/data.go

```



`@latest` can be any version, branch, or commit (`@v1.0.0`, `@main`, `@`).

Or install the binary once:



```sh

go install github.com/JohanLindvall/lightning@latest

lightning path/to/data.go

```



The generated code imports `github.com/JohanLindvall/lightning/pkg/support`, so

the module you generate into must depend on lightning:



```sh

go get github.com/JohanLindvall/lightning

```



A `go:generate` directive in the file that holds your structs works well:



```go

//go:generate go run github.com/JohanLindvall/lightning@latest $GOFILE

```



## How it works



Point the generator at a Go file containing one or more struct types. From

inside this repo:



```sh

go run . path/to/data.go

```



(`go run .` only works in this repo; elsewhere use the module path shown above.)



For each input file `FOO.go` it writes `FOO_unmarshal.go` next to it, containing

an `UnmarshalJSON` method for every top-level struct type. The generated code

imports `github.com/JohanLindvall/lightning/pkg/support` for the shared scanner.



Given:



```go

package cloudflare



type Log struct {

    RayID              string `json:"RayID"`

    EdgeResponseStatus int64  `json:"EdgeResponseStatus"`

    // ...

}

```



you get a `func (v *Log) UnmarshalJSON(data []byte) error` that parses the JSON

with an index-based scanner, no reflection, and no allocation on the common

paths (unescaped strings, integers, object keys).



## Supported types



`string`, `bool`, every sized `int`/`uint` kind, `float32`/`float64`,

`json.RawMessage` (and `RawValue`), `time.Time` (RFC 3339, like `encoding/json`;

the [`lax`](#the-lax-tag-option) option also accepts a space separator and Unix

timestamps), nested named and anonymous structs, slices, fixed-size arrays

(`[N]T`), maps with string keys, pointers, and `interface{}`/`any` (decoded into

the usual Go representation of an arbitrary JSON value). Unknown object keys are

skipped.



A fixed-size array follows `encoding/json`: the leading elements are filled, a

shorter JSON array leaves the remaining elements zero, and a longer one's extras

are discarded.



Embedded struct fields are promoted like `encoding/json`: an embedded struct's

exported fields decode as if they were the outer struct's own (an embedded

pointer is allocated on demand), a name present on both the outer struct and an

embed is resolved by Go's shallower-wins rule, an equal-depth clash is dropped

unless a single field is tagged, and an embedded field with its own JSON tag name

is a plain named field rather than promoted. Embedding a type from another

package, whose fields aren't visible to the generator, is the one gap — it is

decoded as a single named field instead of being flattened.



## The `nocopy` tag option



By default, string and `json.RawMessage` fields copy their bytes out of the

input, matching `encoding/json` semantics. Add `nocopy` to the json tag to make

a field alias the input buffer instead — zero-copy, but the caller must keep the

input `[]byte` unchanged while the result is in use:



```go

type Log struct {

    RayID string          `json:"RayID,nocopy"` // aliases the input

    Body  json.RawMessage `json:"Body,nocopy"`  // aliases the input

}

```



`nocopy` propagates through slices, maps, and pointers, but stops at struct

boundaries (each struct's own field tags govern). Strings containing escape

sequences still allocate, since they can't be a slice of the raw input.



## Alternate field names



A json tag may list several pipe-separated names. Any of them appearing in the

input fills the field, which is handy when an upstream source renamed a key and

you want to accept both spellings:



```go

type Log struct {

    EdgeResponseStatus int64 `json:"EdgeResponseStatus|AnotherField"`

}

```



Comma-separated options still follow the name as usual, so names and `nocopy`

combine freely — `json:"Name|Title,nocopy"` accepts both `Name` and `Title`,

zero-copy.



## The `lax` tag option



By default a value of the wrong type fails the whole decode: a string where a

number is expected returns an error. Add `lax` to the json tag to make such a

mismatch a no-op instead — the offending value is skipped and the field left at

its zero value, while the rest of the object decodes normally:



```go

type Log struct {

    Status int64 `json:"Status,lax"` // a non-number Status is ignored, leaving 0

}

```



Only type mismatches are tolerated; genuinely malformed JSON (a syntax error in

the value) still fails, since a well-formed value of the wrong type can be

skipped but a broken one cannot. `lax` works for every field type, including

nested structs, slices, and maps, where a decode error anywhere in the value

leaves the whole field unset. It combines with the other options and with

alternate names — `json:"Name|Title,nocopy,lax"`.



On a `time.Time` field, `lax` additionally widens what counts as a valid

timestamp. Besides strict RFC 3339, it accepts a space in place of the `T`

date/time separator and a Unix timestamp given as a JSON number or numeric

string, inferring seconds, milliseconds, or microseconds from the magnitude; the

result is normalized to UTC. An unrecognized timestamp is skipped and the field

left unset, like any other lax mismatch. As with `nocopy`, the lenient parser

propagates through slices, maps, and pointers (e.g. `[]time.Time`) but stops at

struct boundaries.



## The `unwrap` tag option



Some payloads carry a nested document as a *string* — JSON embedded in JSON,

sometimes base64-encoded. Add `unwrap` to a field's json tag to decode through

that wrapper: the field's value is read as a JSON string, its body unescaped,

and the result decoded as JSON into the field.



```go

type Envelope struct {

    Name    string  `json:"name"`

    Payload Message `json:"payload,unwrap"` // value is a string holding JSON

}

```



Both forms are accepted automatically. If the unescaped string is itself JSON

(its first non-whitespace byte starts a JSON value) it is decoded directly;

otherwise it is base64-decoded first (standard alphabet, with or without

padding) and the decoded bytes are the JSON. So a `"payload"` of

`"{\"id\":7}"` and of `"eyJpZCI6N30="` both fill `Payload`. A `null` or empty

string leaves the field at its zero value.



The field decodes with its normal rules, so `unwrap` composes with the field's

type (struct, slice, map, scalar…) and with `nocopy` — a `nocopy` string inside

the embedded document aliases the decoded buffer, which is retained for as long

as the result is in use. The embedded document is parsed as a fresh input, so

its own whitespace, escaping, and structure are independent of the outer JSON.



## Comment directives



Some behavior is selected with a `//lightning:` comment on the struct type

(or its declaration), separate from the per-field json tags above.



### `//lightning:compact`



By default a decoder calls `SkipWS` around every token so it accepts JSON with

any whitespace. Mark a type `//lightning:compact` to assert the input has no

whitespace *between* tokens — the form `encoding/json`'s `Marshal` and most wire

protocols emit — and the generator drops those inter-token `SkipWS` calls,

decoding tokens back-to-back:



```go

//lightning:compact

type Log struct {

    RayID  string `json:"RayID"`

    Status int64  `json:"Status"`

}

```



This runs a few percent faster on object-heavy payloads (the `cloudflare-compact`

benchmark beats `cloudflare-nocopy`, its non-compact equivalent, by ~4%).

Whitespace surrounding the whole document is still tolerated — a

trailing newline is fine — so only *inter-token* whitespace is assumed absent.



The directive is an assertion you make about the input: a compact decoder fed

input that does contain inter-token whitespace (for example the same document

pretty-printed) returns an error instead of parsing it. Use it only for sources

that are guaranteed compact. The directive applies to the whole type graph it

roots, including nested structs, slices, and maps.



## Generated function names



The `UnmarshalJSON` methods keep their exact name (the `json.Unmarshaler`

interface requires it). The unexported decoder helpers they call are named

`lightningdecode…` — a prefix derived from the package's import

path and the top-level type — so generating decoders for several types into one

package never produces colliding helper names. No annotation is needed; the

prefix is automatic.



## Key lookups



When you only need a few values out of a document and don't want to generate (or

decode into) a struct, the [`pkg/json`](pkg/json) package exposes the scanner's

key-lookup primitives. They walk the input with the same allocation-free

`Skip`/`ReadKey` machinery the generated decoders use, and every value they

return aliases the input `[]byte` (keep it unchanged while the result is in

use). A returned value follows the same conventions throughout: a string keeps

its surrounding quotes with escapes intact, an object or array spans the whole

`{`…`}` or `[`…`]`, and a scalar is the literal token.



- `Get(data []byte, keys ...string) ([]byte, int, error)` — walks the object-key

  path `keys` one level per key and returns the value's raw bytes (and the offset

  it starts at), without reporting a value type. With no keys it returns the whole

  root value; a missing key returns `ErrKeyNotFound`.

- `GetMany(data []byte, keys []string, out [][]byte) ([][]byte, error)` — looks up

  several *top-level* keys in a **single pass** over the object, where N separate

  `Get` calls would rescan it N times. Results are written into `out[:0]` (pass a

  slice to reuse across calls, allocation-free; a `nil` reuses nothing) and

  returned with `len == len(keys)`: `out[n]` is the value for `keys[n]`, or `nil`

  if that key is absent. A missing key is reported by the `nil` slot, not an

  error (a present key whose value is JSON `null` yields the bytes `"null"`,

  distinct from absent); a non-object root or malformed JSON returns an error.

- `ObjectEach(data []byte, fn func(key string, value []byte) error, keys ...string) error`

  — calls `fn` for every member of the object reached by the path `keys` (the

  root object with no keys). If `fn` returns an error, iteration stops and

  returns it.



```go

// Pull a few fields out of a log record in one pass, reusing a scratch slice.

keys := []string{"ClientIP", "EdgeResponseStatus", "RayID"}

vals, err := json.GetMany(data, keys, scratch[:0])

// vals[0] == []byte(`"203.0.113.23"`), vals[1] == []byte("599"), …

```



Each function has a **compact counterpart** — `GetCompact`, `GetManyCompact`,

`ObjectEachCompact` — with the identical signature and result. Like the

[`//lightning:compact`](#lightningcompact) directive, they assume the input has

no whitespace *between* tokens (the form `encoding/json`'s `Marshal` and most

wire protocols emit) and skip the inter-token `SkipWS` scans, running about 10%

faster; whitespace surrounding the whole document is still tolerated. Feed one

input that does contain inter-token whitespace and it may return an error, so use

them only for sources guaranteed compact.



## String escaping and unescaping



The [`pkg/json`](pkg/json) package exposes the scanner's string codec on its

own, for when you have a JSON string body (the bytes between the quotes) and

just want to decode or encode it.



**Unescaping** (escaped body → decoded value):



- `UnescapeString(in []byte) (string, error)` — decodes the escapes in `in`. If

  `in` contains no escapes the result aliases `in` with no copy; otherwise a new

  string is allocated. `in` is left unchanged.

- `UnescapeStringInto(in, out []byte) (string, error)` — same, but writes the

  decoded bytes into `out` instead of allocating.

  With no escapes the result aliases `in`; otherwise it aliases `out` and

  allocates nothing when `cap(out) >= len(in)`, since unescaping never lengthens

  a string. Pass `out == in` (e.g. `in[:0]`) to decode truly in place,

  overwriting `in`.



Both return a string that aliases a buffer, so keep that buffer unchanged while

the result is in use.



**Escaping** (raw value → escaped body, escaping `"`, `\`, and control bytes;

`/` is left as-is and `\b`/`\f` are written in `\u00XX` form):



- `EscapeString(s []byte, out *strings.Builder)` — writes the escaped form of

  `s` to `out`. The common no-escape case is detected with a vectorized scan and

  written straight to the builder, with no scratch buffer or copy.

- `EscapeStringInto(s, out []byte) []byte` — appends the escaped form of `s` to

  `out` and returns the extended slice, allocating nothing when `out` has room

  (escaping can grow a string up to 6× for control bytes). Pass `out[:0]` to

  reuse a buffer across calls.



Clean runs are skipped eight bytes at a time (SWAR), so strings that need little

or no escaping cost roughly one `memcpy`.



## Number parsing



The [`pkg/json`](pkg/json) package also exposes the scanner's float parser:



- `ParseFloat(b []byte) (float64, error)` — parses the JSON number in `b` as a

  `float64`. It takes the scanner's Clinger fast path first — when the mantissa is

  exact and the decimal exponent is small, the result is a single multiply or

  divide by a power of ten. Numbers that miss it (a mantissa ≥ 2^53 or a larger

  exponent, e.g. high-precision coordinates) take an Eisel-Lemire pass that

  converts the extracted mantissa and exponent with a 128-bit multiply, the same

  fast path `strconv.ParseFloat` uses internally but without re-scanning the

  digits; only the rare ambiguous or >19-digit cases fall back to

  `strconv.ParseFloat`. The Eisel-Lemire result is bit-for-bit identical to

  `strconv` (verified by a differential fuzz test). `b` must be exactly one number

  with no surrounding whitespace; trailing bytes or an empty input return an

  error. Nothing is retained or copied.



## Stripping default fields



The [`pkg/json`](pkg/json) package can also prune a JSON document in a single

pass, dropping object members whose value is a "default" — useful for shrinking

verbose, mostly-default records (Cloudflare HTTP logs, say) before storing or

forwarding them:



- `StripDefaults(input, output []byte, defaults, keep [][]byte, ws WhitespaceMode) []byte`

  — copies `input` to `output`, dropping every object member whose value is a

  default and then dropping any object or array that this leaves empty. A value

  is a default when it is byte-equal to one of `defaults`, compared against the

  bare token — the unquoted contents for a string, the literal token for a number

  or keyword (e.g. `[]byte("none")`, `[]byte("false")`). Empty values are *not*

  special-cased: to drop empty strings (and other empty tokens) include an empty

  entry `[]byte("")` in `defaults`. A member is kept despite a default value when

  its unquoted key is byte-equal to one of `keep` (e.g. `[]byte("WallTimeMs")`).

  String values keep their surrounding quotes and escapes; scalars keep their

  literal token. (Empty `{}`/`[]` are always dropped, independent of `defaults`.)



`output` is filled from the front and the populated prefix is returned; `input`

is never modified. StripDefaults never lengthens the document, so `output` is grown

(allocated) only when `cap(output) < len(input)` — pass `input[:0]` to strip in

place, or a reused buffer to run allocation-free. It is best effort and copies

malformed input through unchanged.



`ws` chooses how inter-token whitespace is handled:

- `RemoveWhitespace` (the zero value) — tolerate any whitespace; output is compact.

- `AssumeCompact` — assert the input has no inter-token whitespace and skip the

  `SkipWS` scans (faster, as [`GetCompact`](#key-lookups) does); misreads spaced input.

- `PreserveWhitespace` — keep the input's whitespace around surviving content, so a

  pretty-printed document stays pretty-printed; only dropped members are removed.



```go

// "" opts empty strings in; "0"/"none"/"false"/"unknown" are the non-empty defaults.

defaults := [][]byte{[]byte(""), []byte("0"), []byte("none"), []byte("false"), []byte("unknown")}

keep := [][]byte{[]byte("WallTimeMs")} // retained even when their value is a default

var scratch []byte

scratch = json.StripDefaults(record, scratch[:0], defaults, keep, json.AssumeCompact)

// {"a":0,"b":"x","e":"","WallTimeMs":0}  ->  {"b":"x","WallTimeMs":0}

```



Matching is length-pre-filtered so a value or key longer than any candidate

skips the scan, and a kept member is moved with a single `copy` when its

`"key":value` span is contiguous in the input.



## SIMD scanning



Two hot scan loops use a single vectorized pass instead of byte-at-a-time work,

with kernels in `pkg/support/index_amd64.s` and `pkg/support/index_arm64.s`

(arm64 uses NEON/ASIMD, 16 bytes/pass, for both):



- **next `"` or `\` in a string** — replaces two `bytes.IndexByte` scans; speeds

  up string-heavy payloads. On amd64 it uses SSE2 (16-byte vectors, two compares

  per 32-byte step), which avoids the `VZEROUPPER` an AVX2 routine must run on

  every call — pure overhead for the short keys and values typical of JSON.

- **next structural byte (`{`, `}`, `[`, `]`, `"`)** — AVX2 on amd64, 32

  bytes/pass, lets `skipObject` / `skipArray` jump over inert content (numbers,

  keys, whitespace) when skipping unknown values. Skipping a large ignored

  array/object is dramatically faster (the `skip-heavy` benchmark decodes at

  >50 GB/s, ~230× `encoding/json`).



Feature detection is at run time (`golang.org/x/sys/cpu`); other platforms, CPUs

without the feature, and inputs shorter than the vector width fall back to scalar

(`bytes.IndexByte`-based) code. Behavior is identical across paths — verified by

fuzzing each primitive against a reference and by `SkipValue`/decode round-trips

vs `encoding/json`, on amd64 and on arm64 under qemu.



## Benchmarks



The [`bench/`](bench) directory is a separate module (so its benchmark-only

dependencies on [easyjson](https://github.com/mailru/easyjson) and

[sonic](https://github.com/bytedance/sonic) stay out of the main module). It

benchmarks the same payload decoded four ways — lightning, `encoding/json`,

easyjson, and bytedance/sonic — across each `bench//` folder.



**See the per-architecture results for the full, current numbers:

[`bench/results_amd64.md`](bench/results_amd64.md) and

[`bench/results_arm64.md`](bench/results_arm64.md).**



Run them yourself with:



```sh

./bench/run_bench.sh

```



which (re)generates each case's deserializers and writes `bench/results.txt`

and an architecture-specific `bench/results_.md` (so runs on different

CPUs do not overwrite each other's committed results).



Representative numbers for a 1.8 KB Cloudflare log (Go 1.26, amd64):



| Decoder | ns/op | B/op | allocs/op | vs stdlib |

|---|--:|--:|--:|--:|

| lightning (`nocopy`) | ~660 | 0 | 0 | ~13× |

| lightning (default)  | ~800 | 144 | 10 | ~10× |

| easyjson             | ~1600–1770 | 24–144 | 1–10 | ~5× |

| sonic                | ~4600 | 3380 | 40 | ~1.9× |

| `encoding/json`      | ~8250 | 920 | 17 | 1.0× |



## Layout



| Path | Description |

|---|---|

| [`main.go`](main.go) | the generator (`package main`) |

| [`pkg/support`](pkg/support) | shared JSON scanning primitives used by generated code |

| [`pkg/json`](pkg/json) | small public API over the scanner (`Get`/`GetMany`/`ObjectEach`, `UnescapeString`, `ParseFloat`, `StripDefaults`) |

| [`bench/`](bench) | benchmark module: hand-written `data.go` + `input.json` per case, plus the generated decoders, harness, and results |



Generated files (`*_unmarshal.go`, `bench/*/bench_test.go`, `bench/*/ej/`, and

the `bench/results.*` outputs) are reproducible and excluded from version

control via [`.gitignore`](.gitignore).