https://github.com/ohsayan/sky-benches
Attempts at benchmarking Skytable with the others to see where we stand
https://github.com/ohsayan/sky-benches
benchmark database key-value key-value-store nosql redis skytable
Last synced: 7 months ago
JSON representation
Attempts at benchmarking Skytable with the others to see where we stand
- Host: GitHub
- URL: https://github.com/ohsayan/sky-benches
- Owner: ohsayan
- License: mit
- Created: 2021-08-06T05:52:49.000Z (almost 5 years ago)
- Default Branch: next
- Last Pushed: 2023-12-05T20:56:41.000Z (over 2 years ago)
- Last Synced: 2023-12-05T21:40:53.229Z (over 2 years ago)
- Topics: benchmark, database, key-value, key-value-store, nosql, redis, skytable
- Homepage:
- Size: 424 KB
- Stars: 16
- Watchers: 2
- Forks: 0
- Open Issues: 1
-
Metadata Files:
- Readme: README.md
- License: LICENSE
Awesome Lists containing this project
README
# Skytable Benchmarks
The goal here is to see how [Skytable](https://github.com/skytable/skytable), which is primarily in-memory but instead of simple commands uses an actual query language (BlueQL) stacks up against in-memory key/value stores like Redis (which is inherently single threaded except for I/O threads) and KeyDB (which is multi-threaded).
> ### IMPORTANT NOTE: Skytable has an actual query language BlueQL while others don't. This means Skytable does way more work than the key/value stores which use commands!
## Results
### Results conclusion
**Even with a complete query language, Skytable is as fast, or faster** than popular key/value stores (which use simple commands and do not have a query language).
**Legend**:
- Redis is red
- KeyDB is orange
- Skytable is blue
## Notes
> **Future planned benchmarks**:
> - Dragonfly
> - Aerospike
> - MongoDB
> - ArangoDB
>
> **Future planned implementation**:
> - Automated benches and report generation
> - Benchmark matrix so that you can benchmark one database against another, as you choose! Compatible commands/queries would be automatically generated
>
> **Future planned VM benches**:
> - Bench against multiple AWS+GCP+Azure VM types
> - Bench across x86_64 and ARM64
## Benchmark setup
- **Here's what we'll do:**
- We'll start up each server with the ability to use multi-threads (or io-threads) and set this to the number of available logical CPU cores (for example a processor might have 6 cores but with HT can run 12 threads so we go with 12)
- For each CRD operation, we'll run 1,000,000 queries for a total of 4M queries.
> **Why not `UPDATE`s?** Unfortunately, there's no SQL equivalent of an `UPDATE` in Redis/KeyDB. So, we skip it for now.
- We'll run a CRUD sequence:
- For simple key/value stores:
- `SET <7 digit key> <1 byte value>`
- `GET <7 digit key>`
- `DEL <7 digit key>`
- For Skytable:
- `INSERT INTO mymodel(<7 digit key>, <1 byte value>)`
- `SELECT pw FROM mymodel WHERE un = <7 digit key>`
- `DELETE FROM mymodel WHERE un = <7 digit key>`
- **Benchmark machine**:
- The benchmarking machine has:
- a 6C/12T configuration
- 32GB RAM
- Base clock: 2.7Ghz, Turbo: 4.5 Ghz
- Arch: Intel x86_64 (manufacturer is also Intel)
- Tiger Lake CPU (11th Generation)
- **OS**:
- Ubuntu 22.04
- Kernel: 5.19.0-42-generic
- Governor: `performance` (checking `cat /sys/devices/system/cpu/cpu0/cpufreq/scaling_governor`)
- **Database versions**:
- Skytable v0.8.0-beta
- Redis `master` (991aff1)
- KeyDB `master` (674d9fb)
## How the benchmarks were run
- Skytable:
- Server started with `skyd `
- Benchmark tool run with varying values for ``: `sky-bench --password --connections `
- Redis and KeyDB:
- Server started with `redis-server --io-threads=12` and `keydb-server --io-threads=12`
- Benchmark tool run for different tests (`-t` each for `get` and `set`) like this: `redis-benchmark --threads 12 -n 1000000 -d 7 -t -c `
## License
This project is distributed under the [MIT License](/LICENSE).