https://github.com/mre/hyperjson
🐍 A hyper-fast Python module for reading/writing JSON data using Rust's serde-json.
https://github.com/mre/hyperjson
decode encode extension hacktoberfest json module python python-json rust serde
Last synced: 3 months ago
JSON representation
🐍 A hyper-fast Python module for reading/writing JSON data using Rust's serde-json.
- Host: GitHub
- URL: https://github.com/mre/hyperjson
- Owner: mre
- License: apache-2.0
- Archived: true
- Created: 2018-04-18T09:01:40.000Z (about 7 years ago)
- Default Branch: master
- Last Pushed: 2023-01-04T04:09:29.000Z (over 2 years ago)
- Last Synced: 2024-09-17T22:52:42.615Z (9 months ago)
- Topics: decode, encode, extension, hacktoberfest, json, module, python, python-json, rust, serde
- Language: Python
- Homepage:
- Size: 4.67 MB
- Stars: 504
- Watchers: 10
- Forks: 40
- Open Issues: 15
-
Metadata Files:
- Readme: README.md
- Funding: .github/FUNDING.yml
- License: LICENSE-APACHE
Awesome Lists containing this project
README
A hyper-fast, safe Python module to read and write JSON data. Works as a
drop-in replacement for Python's built-in
[json](https://docs.python.org/3/library/json.html) module.
This is alpha software and there will be bugs, so maybe don't deploy to production _just_ yet. :wink:
## :warning: NOTE
This project is not actively maintained. [orjson](https://github.com/ijl/orjson) is likely the better alternative.
## Installation
```
pip install hyperjson
```
## Usage
hyperjson is meant as a drop-in replacement for Python's [json
module](https://docs.python.org/3/library/json.html):
```python
>>> import hyperjson
>>> hyperjson.dumps([{"key": "value"}, 81, True])
'[{"key":"value"},81,true]'
>>> hyperjson.loads("""[{"key": "value"}, 81, true]""")
[{u'key': u'value'}, 81, True]
```
## Motivation
Parsing JSON is a solved problem; so, no need to reinvent the wheel, right?
Well, unless you care about **performance and safety**.
Turns out, parsing JSON _correctly_ is a [hard problem](http://seriot.ch/parsing_json.php). Thanks to Rust however, we can minimize the risk of running into [stack overflows or segmentation faults](https://github.com/esnme/ultrajson/issues) however.
hyperjson is a thin wrapper around Rust's [serde-json](https://github.com/serde-rs/json) and [pyo3](https://github.com/PyO3/pyo3). It is compatible with Python 3 (and 2 on a best-effort basis).
For a more in-depth discussion, [watch the talk about this project recorded at the Rust Cologne Meetup in August 2018.](https://media.ccc.de/v/rustcologne.2018.08.hyperjson)
## Goals
- **Compatibility**: Support the full feature-set of Python's `json` module.
- **Safety**: No segfaults, panics, or overflows.
- **Performance**: Significantly faster than `json` and as fast as `ujson` (both written in C).
## Non-goals
- **Support ujson and simplejson extensions**:
Custom extensions like `encode()`, `__json__()`, or `toDict()` are not
supported. The reason is, that they go against PEP8 (e.g. `dunder` methods
are restricted to the standard library, camelCase is not Pythonic) and are not
available in Python's `json` module.
- **Whitespace preservation**: Whitespace in JSON strings is not preserved.
Mainly because JSON is a whitespace-agnostic format and `serde-json` strips
them out by default. In practice this should not be a problem, since your
application must not depend on whitespace padding, but it's something to be
aware of.
## Benchmark
We are _not_ fast yet. That said, we haven't done any big optimizations.
In the long-term we might explore features of newer CPUs like multi-core and SIMD.
That's one area other (C-based) JSON extensions haven't touched yet, because it might
make code harder to debug and prone to race-conditions. In Rust, this is feasible due to crates like
[faster](https://github.com/AdamNiederer/faster) or
[rayon](https://github.com/nikomatsakis/rayon).
So there's a chance that the following measurements might improve soon.
If you want to help, check the instructions in the _Development Environment_ section below.
**Test machine:**
MacBook Pro 15 inch, Mid 2015 (2,2 GHz Intel Core i7, 16 GB RAM) Darwin 17.6.18
## Contributions welcome!
If you would like to hack on hyperjson, here's what needs to be done:
- [x] Implement [`loads()`](https://docs.python.org/3/library/json.html#json.loads)
- [x] Implement [`load()`](https://docs.python.org/3/library/json.html#json.load)
- [x] Implement [`dumps()`](https://docs.python.org/3/library/json.html#json.dumps)
- [x] Implement [`dump()`](https://docs.python.org/3/library/json.html#json.dump)
- [x] Benchmark against [json](https://docs.python.org/3/library/json.html) and
[ujson](https://github.com/esnme/ultrajson/) (see [#1](https://github.com/mre/hyperjson/issues/1))
- [x] Add a CI/CD pipeline for easier testing (see [#2](https://github.com/mre/hyperjson/issues/2))
- [x] Create a proper pip package from it, to make installing easier (see [#3](https://github.com/mre/hyperjson/issues/3)).
- [ ] Profile and optimize performance (see [#16](https://github.com/mre/hyperjson/issues/16))
- [ ] Add remaining [keyword-only arguments](https://docs.python.org/3/library/json.html#basic-usage) to methods
Just pick one of the open tickets. We can provide mentorship if you like. :smiley:
## Developer guide
This project uses [poetry](https://python-poetry.org/docs/) for managing the development environment. If you don't have it installed, run
```
curl -sSL https://raw.githubusercontent.com/python-poetry/poetry/master/get-poetry.py | python
export PATH="$HOME/.poetry/bin:$PATH"
```
The project requires the `nightly` version of Rust.
Install it via `rustup`:
```
rustup install nightly
```
If you have already installed the `nightly` version, make sure it is up-to-date:
```
rustup update nightly
```
After that, you can compile the current version of hyperjson and execute all tests and benchmarks with the following commands:
```
make install
make test
make bench
```
🤫 Pssst!... run `make help` to learn more.
## Drawing pretty diagrams
In order to recreate the benchmark histograms, you first need a few additional prerequisites:
- [Matplotlib](https://matplotlib.org/)
- [Numpy](http://www.numpy.org/)
On macOS, please also add the following to your `~/.matplotlib/matplotlibrc` ([reference](https://markhneedham.com/blog/2018/05/04/python-runtime-error-osx-matplotlib-not-installed-as-framework-mac/)):
```
backend: TkAgg
```
After that, run the following:
```
make plot
```
## License
hyperjson is licensed under either of
- Apache License, Version 2.0, (LICENSE-APACHE or
http://www.apache.org/licenses/LICENSE-2.0)
- MIT license (LICENSE-MIT or http://opensource.org/licenses/MIT)
at your option.
## Contribution
Unless you explicitly state otherwise, any contribution intentionally submitted
for inclusion in hyperjson by you, as defined in the Apache-2.0 license, shall
be dual licensed as above, without any additional terms or conditions.