https://github.com/mulbc/gosbench
Distributed S3 benchmarking tool - Replacement of Cosbench
https://github.com/mulbc/gosbench
benchmark benchmarking ceph cosbench distributed-storage ocs prometheus-exporter s3
Last synced: about 1 year ago
JSON representation
Distributed S3 benchmarking tool - Replacement of Cosbench
- Host: GitHub
- URL: https://github.com/mulbc/gosbench
- Owner: mulbc
- License: gpl-3.0
- Created: 2019-06-30T16:18:29.000Z (about 7 years ago)
- Default Branch: master
- Last Pushed: 2024-12-03T10:38:05.000Z (over 1 year ago)
- Last Synced: 2025-04-12T09:56:16.358Z (about 1 year ago)
- Topics: benchmark, benchmarking, ceph, cosbench, distributed-storage, ocs, prometheus-exporter, s3
- Language: Go
- Homepage:
- Size: 731 KB
- Stars: 48
- Watchers: 5
- Forks: 16
- Open Issues: 1
-
Metadata Files:
- Readme: Readme.md
- License: LICENSE
Awesome Lists containing this project
README
# Gosbench
Gosbench is the Golang reimplementation of [Cosbench](https://github.com/intel-cloud/cosbench).
It is a distributed S3 performance benchmark tool with [Prometheus exporter](https://opencensus.io/exporters/supported-exporters/go/prometheus/) leveraging the official [Golang AWS SDK](https://aws.amazon.com/sdk-for-go/)
## Usage
Gosbench consists of two parts:
* Server: Coordinates Workers and general test queue
* Workers: Actually connect to S3 and perform reading, writing, deleting and listing of objects
INFO: `-d` activates debug logging, `-t` activates trace logging
### Running a test
1. Build the server: `go install github.com/mulbc/gosbench/server`
1. Run the server, specifying a config file: `server -c path/to/config.yaml` - you can find an example config [in the example folder](examples/example_config.yaml)
1. The server will open port 2000 for workers to connect to - make sure this port is not blocked by your firewall!
1. Build the worker: `go install github.com/mulbc/gosbench/worker`
1. Run the worker, specifying the server connection details: `worker -s 192.168.1.1:2000`
1. The worker will immediately connect to the server and will start to get to work.
The worker opens port 8888 for the Prometheus exporter. Please make sure this port is allowed in your firewall and that you added the worker to the Prometheus config.
#### Prometheus configuration
Make sure your prometheus configuration looks similar to this:
```yaml
global:
scrape_interval: 15s # Set the scrape interval to every 15 seconds. Default is every 1 minute.
evaluation_interval: 15s # Evaluate rules every 15 seconds. The default is every 1 minute.
scrape_configs:
- job_name: 'prometheus'
static_configs:
- targets:
- localhost:9090
- job_name: 'gosbench'
scrape_interval: 1s
static_configs:
- targets:
- WORKER1.example.com:8888
- WORKER2.example.com:8888
```
To reload the configuration, you can either send a SIGHUP to your prometheus server or just restart it ;)
Afterwards ensure that you have your Gosbench workers listed at http://your.prometheus.server.example.com:9090/targets
It is expected that the workers are in state `DOWN` most of the time... they are only scrapeable during a test run.
It is Best Practice to run the [Prometheus Node Exporter](https://github.com/prometheus/node_exporter) on all hosts as well, to gather common system metrics during the tests. This will help you in identifying bottlenecks. Please consult the Node Exporter manuals on how to install and configure it on your platform.
### Evaluating a test
During a test, Prometheus will scrape the performance data continuously from the workers.
You can visualize this data in Grafana. To get an overview of what the provided data looks like, check out [the example scrape](examples/example_prom_exporter.log).
There is also an [example Grafana dashboard](examples/grafana_dashboard.json) that you can import and use. The Dashboard has some basic overview of the most common stats that people are interested in:
### Docker
There are now Docker container images available for easy consumption:
```shell
docker pull quay.io/mulbc/goroom-server
docker pull quay.io/mulbc/goroom-worker
```
In the `k8s` folder you will find example files to deploy Gosbench on Openshift and Kubernetes.
Be sure to modify the ConfigMaps in `gosbench.yaml` to use your S3 endpoint credentials.
### Reading pre-existing files from buckets
Due to popular demand, reading pre-existing files have been added. You activate this special mode by setting `existing_read_weight` to something higher than 0.
There are some important things to consider though ;)
Just like with other operations, the `bucket_prefix` value will be evaluated to determine the bucket name to search for pre-existing objects.
**Example:** This is an excerpt of your config:
```yaml
objects:
size_min: 5
size_max: 100
part_size: 0
# distribution: constant, random, sequential
size_distribution: random
unit: KB
number_min: 10
number_max: 100
# distribution: constant, random, sequential
number_distribution: constant
buckets:
number_min: 2
number_max: 10
# distribution: constant, random, sequential
number_distribution: constant
bucket_prefix: myBucket-
```
Note: Due to the constant distribution, we will only consider the `_min` values.
This will cause each workers to search for pre-existing files in the buckets `myBucket-0` and `myBucket-1` and read 10 objects from these buckets. If there are less than 10 objects in any of these buckets, some objects will be read multiple times. The object size given in your config will be ignored when reading pre-existing files.
## Cosbench vs Gosbench benchmark comparision
When a new tool is presented, it’s essential to compare it to existing tools for accuracy. For this reason, we ran a comparision between Cosbench and Gosbench. Both benchmarks were tasked to do a 100% write test and 100% read test on 4KB, 16KB, 256KB, 1MB, 4MB objects for 60 seconds each. The tests were to run on one RGW using S3 protocol in ceph storage clusteri, also run in the test configuration in parallel. Figure below show writing and reading, respectively. From these charts, it’s apparent that the performance metrics for all objects are similar for both tools.
## Contributing
* Be aware that this repo uses pre-commit hooks - install them via `pre-commit install`
* [More info](https://pre-commit.com/)
* We are using Go modules in this repository - read up on it [here](https://blog.golang.org/using-go-modules)
* Check out the open [TODOs](TODO.md) for hints on what to work on
## Known issues
* Workers will error out when the config's min value is larger than the max value (even for a constant distribution)