https://github.com/fandreuz/python-serialization-benchmark
Low-effort benchmark of several serialization libraries for scientific applications
https://github.com/fandreuz/python-serialization-benchmark
avro json numpy protobuf python serialization
Last synced: about 1 month ago
JSON representation
Low-effort benchmark of several serialization libraries for scientific applications
- Host: GitHub
- URL: https://github.com/fandreuz/python-serialization-benchmark
- Owner: fandreuz
- License: mit
- Created: 2024-02-13T21:22:55.000Z (almost 2 years ago)
- Default Branch: master
- Last Pushed: 2024-02-16T21:45:55.000Z (almost 2 years ago)
- Last Synced: 2025-01-21T14:45:12.243Z (about 1 year ago)
- Topics: avro, json, numpy, protobuf, python, serialization
- Language: Python
- Homepage:
- Size: 36.1 KB
- Stars: 0
- Watchers: 1
- Forks: 0
- Open Issues: 1
-
Metadata Files:
- Readme: README.md
- License: LICENSE
Awesome Lists containing this project
README
# Python serialization benchmark
A simple serialization benchmark for textual data and NumPy arrays. Results
are obtained using `time.time_ns()`, and they are printed in nanoseconds.
## Libraries
- Avro
- `json` (standard library)
- orjson
- pickle4
- pickle5
- Protocol Buffers
- RapidJSON
## Running the benchmark
```bash
podman run --interactive --tty --rm \
-v .:/home \
-w /home \
--env BENCHMARK_DATA_SIZE=$SIZE \
--env BENCHMARK_ITERATIONS=$ITERATIONS \
python:$PY_VERSION \
sh -c "./run_benchmark.sh"
```
## Data type
`--env BENCHMARK_TYPE=...`
- `numeric` : See `benchmark/run/numeric_benchmark.py`
- `textual` : See `benchmark/run/textual_benchmark.py`
## Output
### CSV
`--env BENCHMARK_PP=csv` (default)
```
Serialization
Avro,Json,Orjson,Pickle4,Pickle5,Protobuf,RapidJSON
290244.85,26887717.19,1806935.09,48814.31,34544.63,78751.63,28539102.41
Deserialization
Avro,Json,Orjson,Pickle4,Pickle5,Protobuf,RapidJSON
105083.1,11868387.1,2163872.31,19100.97,22108.22,33005.58,12793808.94
```
### PGFPlots
`--env BENCHMARK_PP=pgfplot_histogram`
```
Serialization
(AvroBackend,284676.57) (JsonBackend,28447828.45) (OrjsonBackend,2088943.27) (Pickle4Backend,50752.19) (Pickle5Backend,39036.18) (ProtobufBackend,82686.77) (RapidjsonBackend,31793210.02)
Deserialization
(AvroBackend,113478.34) (JsonBackend,12809206.11) (OrjsonBackend,2546817.87) (Pickle4Backend,20552.24) (Pickle5Backend,24479.14) (ProtobufBackend,34602.56) (RapidjsonBackend,14472453.05)
```
## Results aggregation
`--env BENCHMARK_AGGREGATION=...`
- `mean` : Mean
- `min`/`max`: Min/max
- `percentile99`: 99th percentile