https://github.com/aybabtme/parajson

Decode streams of \n separated objects in parallel.
https://github.com/aybabtme/parajson

Last synced: about 1 month ago
JSON representation

Decode streams of \n separated objects in parallel.

• Host: GitHub
• URL: https://github.com/aybabtme/parajson
• Owner: aybabtme
• License: mit
• Created: 2014-06-24T05:51:34.000Z (over 10 years ago)
• Default Branch: master
• Last Pushed: 2014-06-24T05:57:00.000Z (over 10 years ago)
• Last Synced: 2024-06-20T17:48:34.396Z (5 months ago)
• Language: Go
• Homepage:
• Size: 121 KB
• Stars: 1
• Watchers: 3
• Forks: 0
• Open Issues: 0
• Metadata Files:
  • Readme: README.md
  • License: LICENSE

Awesome Lists containing this project

README

        
# parajson

Decode streams of `\n` separated objects in parallel.

# Performance

The gain in speed from using a normal, single threaded decoding is:

| Technique | Throughput  |

|-----------|-------------|

| normal    | 27.30 MB/s  |

| parajson  | 115.07 MB/s |

Using a 386MB file of 1'302'811 `s3.Key` in JSON, on my 8 cores

MBPr 2012.

# Usage

This package implements a simple pipeline to decode JSON objects in

parallel.  Use it like this:

```go

r := getReader()

n := runtime.NumCPU()

protofact := func() interface{} {

    return &s3.Key{}

}

defer r.Close()

keys, errc := parajson.Decode(r, n, protofact)

for proto := range keys {

    key := proto.(*s3.Key)

    // use key

}

for err := range errc {

    log.Fatal(err)

}

```

If you want to use your own decoder (instead of `encoding/json`), pass

your own decoding func:

```go

parajson.SetUnmarshal(func(data []byte, v interface{}) error {

    return ownDecoder(data, v)

})

// do the decoding

```

# License

MIT.