https://github.com/threedotslabs/watermill-benchmark
Benchmarking tools for Watermill.
https://github.com/threedotslabs/watermill-benchmark
benchmark go google-cloud-pubsub kafka mysql nats rabbitmq watermill
Last synced: 3 months ago
JSON representation
Benchmarking tools for Watermill.
- Host: GitHub
- URL: https://github.com/threedotslabs/watermill-benchmark
- Owner: ThreeDotsLabs
- License: mit
- Created: 2019-09-12T17:58:27.000Z (about 6 years ago)
- Default Branch: master
- Last Pushed: 2024-11-19T15:38:52.000Z (11 months ago)
- Last Synced: 2025-06-06T03:54:07.685Z (4 months ago)
- Topics: benchmark, go, google-cloud-pubsub, kafka, mysql, nats, rabbitmq, watermill
- Language: Go
- Homepage: https://watermill.io
- Size: 129 KB
- Stars: 22
- Watchers: 3
- Forks: 8
- Open Issues: 2
-
Metadata Files:
- Readme: README.md
- License: LICENSE
Awesome Lists containing this project
README
# Watermill Benchmark
This is a set of tools for benchmarking [watermill](https://github.com/ThreeDotsLabs/watermill).
**Warning:** This tool is meant to provide a rough estimate on how fast each Pub/Sub can process messages.
It uses very simplified infrastructure to set things up and default configurations.Keep in mind that final performance depends on multiple factors.
**It's not meant to be a definitive answer on which Pub/Sub is the fastest.**
It should give you an idea of the ballpark performance you can expect.## How it works
* All tests are run on a single 16 CPU GCloud compute instance (`n1-highcpu-16`).
* Docker Compose is used to run Pub/Sub infrastructure and benchmark code (except for Google Cloud Pub/Sub).
* The tool will first produce a big number of messages on a generated topic.
* Then it will subscribe to the topic and consume all of the messages.
* Multiple message sizes can be chosen (by default: 16, 64 and 256 bytes).## Results (as of 18 November 2024)
### Kafka (one node)
| Message size (bytes) | Publish (messages / s) | Subscribe (messages / s) |
|----------------------|------------------------|--------------------------|
| 16 | 41,492 | 101,669 |
| 64 | 40,189 | 106,264 |
| 256 | 40,044 | 107,278 |### NATS Jetstream (16 Subscribers)
| Message size (bytes) | Publish (messages / s) | Subscribe (messages / s) | Subscribe (messages / s - async ack) |
|----------------------|------------------------|--------------------------|--------------------------------------|
| 16 | 50,668 | 34,713 | 59,728 |
| 64 | 49,204 | 34,561 | 59,743 |
| 256 | 48,242 | 34,097 | 59,385 |### NATS Jetstream (48 Subscribers)
| Message size (bytes) | Publish (messages / s) | Subscribe (messages /s ) | Subscribe (messages / s - async ack) |
|----------------------|------------------------|--------------------------|--------------------------------------|
| 16 | 50,680 | 46,377 | 86,348 |
| 64 | 49,341 | 46,307 | 86,078 |
| 256 | 48,744 | 46,035 | 86,499 |### Redis
| Message size (bytes) | Publish (messages / s) | Subscribe (messages / s) |
|----------------------|------------------------|--------------------------|
| 16 | 59,158 | 12,134 |
| 64 | 58,988 | 12,392 |
| 256 | 58,038 | 12,133 |### SQL (MySQL)
| Message size (bytes) | Publish (messages / s) | Subscribe (messages / s - batch size = 1) | Subscribe (messages / s - batch size = 100) |
|----------------------|------------------------|-------------------------------------------|---------------------------------------------|
| 16 | 6,371 | 283 | 2,794 |
| 64 | 9,887 | 281 | 2,637 |
| 256 | 9,596 | 271 | 2,766 |### SQL (PostgreSQL)
| Message size (bytes) | Publish (messages / s) | Subscribe (messages / s - batch size = 1) | Subscribe (messages / s - batch size = 100) |
|----------------------|------------------------|-------------------------------------------|---------------------------------------------|
| 16 | 2,552 | 122 | 9,460 |
| 64 | 2,831 | 118 | 9,045 |
| 256 | 2,744 | 104 | 7,843 |### SQL (PostgreSQL Queue)
| Subscribe Batch Size | Message size (bytes) | Publish (messages / s) | Subscribe (messages / s - batch size = 1) | Subscribe (messages / s - batch size = 100) |
|----------------------|----------------------|------------------------|--------------------------------------------|---------------------------------------------|
| 100 | 16 | 2,825 | 146 | 10,466 |
| 100 | 64 | 2,842 | 147 | 9,626 |
| 100 | 256 | 2,845 | 138 | 8,276 || Message size (bytes) | Publish (messages / s) | Subscribe (messages / s) |
|----------------------|------------------------|--------------------------|
| 16 | 2,825 | 0,146 |
| 64 | 2,842 | 0,147 |
| 256 | 2,845 | 0,138 |### Google Cloud Pub/Sub (16 subscribers)
| Message size (bytes) | Publish (messages / s) | Subscribe (messages / s) |
|----------------------|------------------------|--------------------------|
| 16 | 3,027 | 28,589 |
| 64 | 3,020 | 31,057 |
| 256 | 2,918 | 32,722 |### AMQP (RabbitMQ, 16 subscribers)
| Message size (bytes) | Publish (messages / s) | Subscribe (messages / s) |
|----------------------|------------------------|--------------------------|
| 16 | 2,770 | 14,604 |
| 64 | 2,752 | 12,128 |
| 256 | 2,750 | 8,550 |### GoChannel
| Message size (bytes) | Publish (messages / s) | Subscribe (messages / s) |
|----------------------|------------------------|--------------------------|
| 16 | 315,776 | 138,743 |
| 64 | 325,341 | 163,034 |
| 256 | 341,223 | 145,718 |## VM Setup
The project includes [Terraform](https://www.terraform.io/) definition for setting up an instance on Google Cloud Platform.
It will spin up a fresh Ubuntu 19.04 instance, install docker with dependencies and clone this repository.
Set environment variables:
```bash
# project name on GCP
TF_VAR_project=
# public part of the key that you will use to access SSH
TF_VAR_pub_key_path=
```Create the VM:
```bash
cd setup
terraform apply
```The command will output the public IP address of the server. Use ssh with user `benchmark` to access it.
After running all benchmarks, destroy the VM:
```bash
terraform destroy
```## Configuration
### Google Pub/Sub
Set environment variables in `compose/.env`:
```bash
# path to json file within the project with GCP credentials
GOOGLE_APPLICATION_CREDENTIALS=compose/key.json
# project name on GCP
GOOGLE_CLOUD_PROJECT=
```## Running
Run benchmarks with:
```bash
./run.sh
```