Ecosyste.ms: Awesome
An open API service indexing awesome lists of open source software.
https://github.com/processone/rtb
Benchmarking tool to stress real-time protocols
https://github.com/processone/rtb
benchmark benchmarking erlang jabber mqtt real-time realtime websocket xmpp
Last synced: about 1 month ago
JSON representation
Benchmarking tool to stress real-time protocols
- Host: GitHub
- URL: https://github.com/processone/rtb
- Owner: processone
- License: apache-2.0
- Created: 2018-07-06T12:30:55.000Z (over 6 years ago)
- Default Branch: master
- Last Pushed: 2023-12-04T16:03:59.000Z (about 1 year ago)
- Last Synced: 2024-08-03T16:08:55.573Z (5 months ago)
- Topics: benchmark, benchmarking, erlang, jabber, mqtt, real-time, realtime, websocket, xmpp
- Language: Erlang
- Homepage: https://www.process-one.net/
- Size: 408 KB
- Stars: 50
- Watchers: 13
- Forks: 7
- Open Issues: 4
-
Metadata Files:
- Readme: README.md
- Contributing: CONTRIBUTING.md
- License: LICENSE
- Code of conduct: CODE_OF_CONDUCT.md
Awesome Lists containing this project
README
RTB: Benchmarking tool to stress real-time protocols
====================================================
The idea of RTB is to provide a benchmarking tool to stress test
XMPP and MQTT servers with the minimum configuration overhead.
Also, at least in the XMPP world, a "golden standard" benchmark
is highly demanded in order to compare server implementations,
avoiding disambiguations as much as possible. RTB is believed
to be used in such role because it has sane defaults (gathered
from statistics of real world servers) and it is able to
stress test all features defined in the
[XMPP Compliance Suite 2019](https://xmpp.org/extensions/xep-0412.html)
**Table of Contents**:
1. [Status](#status)
2. [System requirements](#system-requirements)
3. [Compiling](#compiling)
4. [Usage](#usage)
5. [Database population](#database-population)
1. [XMPP scenario](#xmpp-scenario)
2. [MQTT scenario](#mqtt-scenario)
6. [Configuration](#configuration)
1. [General parameters](#general-parameters)
2. [Parameters of the XMPP scenario](#parameters-of-the-xmpp-scenario)
3. [Parameters of the MQTT scenario](#parameters-of-the-mqtt-scenario)
4. [Patterns](#patterns)
# Status
RTB is in an early stage of development with the following limitations:
- MQTT support is limited to versions 3.1.1 and 5.0
- For XMPP protocol, support for
[Personal Eventing Protocol](https://xmpp.org/extensions/xep-0163.html)
is lacking.
Also, "sane" defaults and what should be considered a
"golden benchmark" is yet to be discussed within the XMPP and MQTT community.
However, the tool has been already battle-tested:
[ProcessOne](https://www.process-one.net/) is using the tool to
stress test [ejabberd SaaS](https://ejabberd-saas.com/) deployments.
# System requirements
To compile RTB you need:
- Unix-like OS. Windows is not supported. Only Linux is tested.
- GNU Make.
- GCC
- G++
- Libexpat ≥ 1.95
- Libyaml ≥ 0.1.4
- Erlang/OTP ≥ 19.0
- OpenSSL ≥ 1.0.0
- Zlib ≥ 1.2.3
- gnuplot ≥ 4.4
For Debian based distros to install all the dependencies run:
```
# apt install gcc g++ make libexpat1-dev libyaml-dev libssl-dev \
zlib1g-dev gnuplot-nox erlang-nox erlang-dev
```
For Arch Linux:
```
# pacman -S expat libyaml erlang-nox gnuplot
```
For other Linux distros, *BSD and OSX:
**TODO**: Please create an issue/PR if you know the sequence of packages to install.
# Compiling
As usual, the following commands are used to obtain and compile the tool:
```
$ git clone https://github.com/processone/rtb.git
$ cd rtb
$ make
```
# Usage
Once compiled, you will find `rtb.sh`, `rtb.yml.xmpp.example` and
`rtb.yml.mqtt.example` files. Copy either `rtb.yml.mqtt.example` or
`rtb.yml.xmpp.example` to `rtb.yml`, edit it (see section below) and run:
```
$ cp rtb.yml.xmpp.example rtb.yml
$ editor rtb.yml
$ ./rtb.sh
```
Investigate the output of the script for presence of errors. All errors
are supposed to be self-explanatory and most of them have some hints on
what should be done to fix them.
To stop the benchmark press `Ctrl+C+C`. In order to start the
benchmark in background append `-detached` switch:
```
$ ./rtb.sh -detached
```
You will find logs in the `log` directory. To monitor the progress
open the statistics web page: it's located at `http://this.machine.tld:8080`
by default. Edit `www_port` option to change the port if needed.
# Database population
During compilation a special utility is created which aims to help
you populating the server's database. It's located at `priv/bin/rtb_db`.
Run `priv/bin/rtb_db --help` to see available options.
## XMPP scenario
The utility is able to generate files for users and rosters in either
CSV format ([ejabberd](https://www.ejabberd.im/) and [jackal](https://github.com/ortuman/jackal))
or in Lua format ([Metronome](https://metronome.im/)/[Prosody](https://prosody.im/)).
In order to generate files for ejabberd execute something like:
```
$ priv/bin/rtb_db -t ejabberd -c 1000 -u user%@domain.tld -p pass% -r 20
```
The same, but for Metronome will look like:
```
$ priv/bin/rtb_db -t metronome -c 1000 -u user%@domain.tld -p pass% -r 20
```
For Prosody:
```
$ priv/bin/rtb_db -t prosody -c 1000 -u user%@domain.tld -p pass% -r 20
```
For jackal:
```
$ priv/bin/rtb_db -t jackal -c 1000 -u user%@domain.tld -p pass% -r 20
```
Here 1000 is the total amount of users (must match `capacity` parameter
of the configuration file) and 20 is the number of items in rosters.
Don't provide `-r` option or set it to zero (0) if you don't want to
generate rosters.
Follow the hint provided by the utility to load generated files
into the server's spool/database. Note that `--username` and
`--password` arguments must match those defined in the configuration file
(see `jid` and `password` parameters).
## MQTT scenario
The utility is also able to generate `passwd` file for
[Mosquitto](https://mosquitto.org/).
In order to generate the file execute something like:
```
$ priv/bin/rtb_db -t mosquitto -c 1000 -u user% -p pass%
```
Here 1000 is the total amount of users (must match `capacity` parameter
of the configuration file).
Follow the hint provided by the utility to set up `passwd` file in Mosquitto
configuration.
Note that `--username` and `--password` arguments must match those defined
in the configuration file (see `username` and `password` parameters).
# Configuration
All configuration is performed via editing parameters of `rtb.yml` file.
The file has [YAML](http://en.wikipedia.org/wiki/YAML) syntax.
There are mandatory and optional parameters.
The majority of parameters are optional.
## General parameters
This group of parameters are common for all scenarios.
### Mandatory parameters
- **scenario**: `string()`
The benchmarking scenario to use. Available values are `mqtt` and `xmpp`.
- **interval**: `non_neg_integer()`
The option is used to set a timeout to wait before spawning
the next connection. The value is in **milliseconds**.
- **capacity**: `pos_integer()`
The total amount of connections to be spawned, starting from 1.
- **certfile**: `string()`
A path to a certificate file. The file MUST contain both a full certficate
chain and a private key in PEM format.
The option is only mandatory in the case when your scenario is configured
to utilize TLS connections.
- **servers**: `[uri()]`
The list of server URIs to connect. The format of the URI must be
`scheme://hostname:port/path` where `scheme` can be `tcp`, `tls`, `ws` or `wss`;
`hostname` can be any DNS name or IP address and `port` is a port number.
Note that `scheme`, `hostname` and `port` parts of the URI are mandatory, where
`path` part is optional and only meaningful when the scheme is `ws` or `wss`.
IPv6 addresses MUST be enclosed in square brackets.
It's highly recommended to use IP addresses in `hostname`
part: excessive DNS lookups may create significant overhead for the
benchmarking tool itself.
The option is used to set a transport, address and port of the server(s)
being tested.
The option is only mandatory for MQTT scenario, because there are no well
established mechanisms to locate MQTT servers.
For XMPP scenario the default is empty list which means server endpoints
will be located according to RFC6120 procedure, that is DNS A/AAAA/SRV lookup
of a domain part of `jid` parameter.
Leaving the default alone is also not recommended for the reason described above.
An URI from the `servers` list is picked in round-robin manner during
initial connections setup, but it's picked randomly for reconnection attempts.
Note that WebSockets connections are currently supported by MQTT scenario only.
Example:
```yaml
scenario: mqtt
interval: 10
capacity: 10000
certfile: cert.pem
servers:
- tls://127.0.0.1:8883
- tcp://192.168.1.1:1883
- wss://[::1]:443/mqtt
```
### Optional parameters
- **bind**: `[ip_address()]`
The list of IP addresses of local interfaces to bind. The typical
usage of the option is to set several binding interfaces in order
to establish more than 64k outgoing connections from the same machine.
The default is empty list: in this case a binding address will be chosen
automatically by the OS.
- **stats_dir**: `string()`
A path to the directory where statistics data will be stored.
The files in the directory are used by `gnuplot` to generate statistics graphs.
The default value is `stats`.
- **www_dir**: `string()`
A path to a directory where HTML and image files will be created.
The default is `www`. This is used by the statistics web interface.
- **www_port**: `pos_integer()`
A port number to start the statistics web interface at. The default is 8080.
- **gnuplot**: `string()`
The path to a gnuplot execution binary. By default RTB is trying to detect
the location of gnuplot automatically.
- **debug**: `true | false`
Whether to log debug messages or not. This is only needed to track down
issues of the server or the tool itself. **DON'T** enable in large scale benchmarking.
The default is `false`.
Example:
```yaml
bind:
- 192.168.1.1
- 192.168.1.2
- 192.168.1.3
stats_dir: /tmp/rtb/stats
www_dir: /tmp/rtb/www
www_port: 1234
gnuplot: /opt/bin/gnuplot
```
## Parameters of the XMPP scenario
This group of parameters are specific to the XMPP scenario only.
The parameters described here are applied per single session.
### Mandatory parameters
- **jid**: `pattern()`
A pattern for an XMPP address: bare or full. If it's bare, the default
`rtb` resource will be used. Refer to [Patterns](#patterns) section for
the detailed explanation of possible pattern values.
- **password**: `pattern()`
The pattern for a password. Refer to [Patterns](#patterns) section for
the detailed explanation of possible pattern values.
Example:
```yaml
jid: user%@domain.tld
password: pass%
```
### Optional parameters
#### Parameters for timings control.
- **negotiation_timeout**: `pos_integer() | false`
A timeout to wait for a stream negotiation to complete.
The value is in **seconds**. It can be set to `false` to disable timeout.
The default is 100 seconds.
- **connect_timeout**: `pos_integer() | false`
A timeout to wait for a TCP connection to be established.
The value is in **seconds**. It can be set to `false` to disable timeout.
The default is 100 seconds.
- **reconnect_interval**: `pos_integer() | false`
A timeout to wait before another reconnection attempt after previous
connection failure. Initially it is picked randomly between `1` and this
configured value. Then, exponential back off is applied between several
consecutive connection failures. The value is in **seconds**.
It can be set to `false` to disable reconnection attempts completely:
thus the failed session will never be restored.
The default is 60 (1 minute) - the value recommended by
[RFC6120](https://tools.ietf.org/html/rfc6120#section-3.3).
- **message_interval**: `pos_integer() | false`
An interval between sending messages. The value is in **seconds**.
It can be set to `false` to disable sending messages completely.
The default is 600 (10 minutes). See also `message_body_size` option.
- **muc_message_interval**: `pos_integer() | false`
An interval between sending groupchat messages. The value is in **seconds**.
It can be set to `false` to disable sending groupchat messages completely.
The default is 600 (10 minutes). If there are several MUC rooms configured,
the groupchat message is sent to a randomly chosen one, i.e. RTB doesn't
multicast the message to all joined rooms.
See also `message_body_size` and `muc_rooms` options.
The option doesn't have any effect if `muc_rooms` option is not set.
- **presence_interval**: `pos_integer() | false`
An interval between sending presence broadcats. The value is in **seconds**.
It can be set to `false` to disable sending presences completely.
The default is 600 (10 minutes). Note that at the first successful login a
presence broadcast is always sent unless the value is not set to `false`.
- **disconnect_interval**: `pos_integer() | false`
An interval to wait before forcing disconnect. The value is in **seconds**.
The default is 600 (10 minutes). If stream management is enabled
(this is the default, see `sm` option), then the session
will be resumed after a random timeout between 1 and the value of
`max` attribute of `` element reported by the server.
Otherwise, the next reconnection attempt will be performed according
to the value and logic of `reconnect_interval`.
- **proxy65_interval**: `pos_integer() | false`
An interval between file transfers via Proxy65 service (XEP-0065).
The value is in **seconds**. It can be set to `false` to disable
file transfer completely. The default is 600 (10 minutes).
See also `proxy65_size` option.
- **http_upload_interval**: `pos_integer() | false`
An interval between file uploads via HTTP Upload service (XEP-0363).
The value is in **seconds**. It can be set to `false` to disable
file uploads completely. The default is 600 (10 minutes).
See also `http_upload_size` option.
#### Parameters for payload/size control
- **message_to**: `pattern()`
The pattern of a JID to which messages will be sent. By default
a random JID within benchmark capacity is picked (whether it is
already connected or not). Refer to [Patterns](#patterns) section for
the detailed explanation of possible pattern values.
For example, to send messages to already connected JIDs, set:
```yaml
message_to: [email protected]
```
- **message_body_size**: `non_neg_integer()`
The size of `