https://github.com/bserdar/jsonstream
Go library to deal with streaming JSON documents with line-delimited JSON (NDJSON), record separator-delimited JSON, concatenated JSON, and length-prefixed JSON
https://github.com/bserdar/jsonstream
go json jsonstream ndjson
Last synced: 9 months ago
JSON representation
Go library to deal with streaming JSON documents with line-delimited JSON (NDJSON), record separator-delimited JSON, concatenated JSON, and length-prefixed JSON
- Host: GitHub
- URL: https://github.com/bserdar/jsonstream
- Owner: bserdar
- License: apache-2.0
- Created: 2019-03-30T18:22:15.000Z (almost 7 years ago)
- Default Branch: master
- Last Pushed: 2019-04-28T03:24:08.000Z (over 6 years ago)
- Last Synced: 2025-03-26T20:51:23.337Z (10 months ago)
- Topics: go, json, jsonstream, ndjson
- Language: Go
- Size: 14.6 KB
- Stars: 8
- Watchers: 3
- Forks: 0
- Open Issues: 0
-
Metadata Files:
- Readme: readme.md
- License: LICENSE
Awesome Lists containing this project
README
# JSON Streams
[](https://godoc.org/github.com/bserdar/jsonstream)
This library supports streaming JSON streaming conventions described
in this [Wikipedia page](https://en.wikipedia.org/wiki/JSON_streaming).
This is for the concatenated JSON streams, where each JSON document is
concatenated one after the other:
```
rd:=jsonstream.NewConcatReader(reader)
var entry MyEntry
for {
err:=rd.Unmarshal(&entry)
if err==io.EOF {
break
}
if err!=nil {
return err
}
processEntry(entry)
}
```
This is for JSON streams with a separate JSON document in each line
(NDJSON):
```
ndLinesReader:=jsonstream.NewLineReader(reader)
ndLinesWriter:=jsonstream.NewLineWriter(writer)
```
This is for JSON streams separated with record separator delimiter:
```
seqReader:=jsonstream.NewSeqReader(reader) // 0x1e record separator
seqWriter:=jsonstream.NewSeqWriter(writer)
```
```
seqReader:=jsonstream.NewSeqReaderWithSep(reader,'\n') // Line-separated JSON
seqWriter:=jsonstream.NewSeqWriterWithSep(writer,'\n')
```
This is for JSON streams with length prefixed JSON stream, where each
JSON document is prefixed by its byte length:
```
lpReader:=jsonstream.NewLenPrefixedReader(reader)
lpWriter:=jsonstream.NewLenPrefixesWriter(writer)
```
## APIs
All four stream readers/writers use the same APIs.
### Readers
```
data, err:=reader.ReadRaw()
```
ReadRaw reads the next JSON document. Only the ConcatReader validates
that the JSON document is a valid document, the remaining readers
simply read until the next delimiter. The returned byte array is a
newly allocated copy of the underlying read buffer. Some of the
readers use buffered input, so the state of the underlying reader is
unknown.
```
var data myStruct
err:=reader.Unmarshal(&data)
```
Unmarshals the next entry from the input. For concatenated JSON,
errors invalidate the rest of the stream. For others stream processing
can continue.
### Writers
```
err:=writer.WriteRaw(data)
```
WriteRaw simply writes the []byte data to the output, with the correct
delimiter. For NDJSON, WriteRaw removes the newline characters from
data.
```
err:=writer.Marshal(data)
```
Marshal first encodes data to JSON, and then writes it to the output.