https://github.com/vpalmisano/webrtcperf
WebRTC performance and quality evaluation tool.
https://github.com/vpalmisano/webrtcperf
chromium concurrent-webrtc-sessions edumeet janus jitsi load mediasoup nodejs performance statistics test test-automation video webrtc webrtcperf
Last synced: 1 day ago
JSON representation
WebRTC performance and quality evaluation tool.
- Host: GitHub
- URL: https://github.com/vpalmisano/webrtcperf
- Owner: vpalmisano
- License: agpl-3.0
- Created: 2021-03-03T17:08:05.000Z (about 4 years ago)
- Default Branch: devel
- Last Pushed: 2025-05-02T13:35:12.000Z (12 days ago)
- Last Synced: 2025-05-02T14:57:29.359Z (12 days ago)
- Topics: chromium, concurrent-webrtc-sessions, edumeet, janus, jitsi, load, mediasoup, nodejs, performance, statistics, test, test-automation, video, webrtc, webrtcperf
- Language: TypeScript
- Homepage:
- Size: 4.27 MB
- Stars: 182
- Watchers: 9
- Forks: 39
- Open Issues: 5
-
Metadata Files:
- Readme: README.md
- License: LICENSE
Awesome Lists containing this project
README
# WebRTC Perf
[GitHub page](https://github.com/vpalmisano/webrtcperf) | [Documentation](https://vpalmisano.github.io/webrtcperf)
[](https://github.com/vpalmisano/webrtcperf/actions/workflows/build.yaml)
WebRTC performance and quality evaluation tool.
It allows to validate the audio/video quality and the client CPU/memory usage
when multiple connections join the same WebRTC service.
Main features:
- A NodeJS application/library using Puppeteer for controlling chromium instances.
- It can be executed:
- using the pre built Docker image; this is the suggested way to run the tool
without installing any dependency;
- from sources (using git pull or npm install);
- using the pre built executables generated for each platform.
- It allows to inject custom Javascript source files that will run into the
browser page context for automating some tasks (e.g. pressing a button to join
a conference room).
- It allows to throttle the networking configuration, limiting the ingress/egress
available bandwidth, the RTT or the packet loss %.
- It uses a patched version of chromium (see `./chromium` directory) that allows
to disable the video decoding, lowering the CPU requirements when running multiple
browser sessions.
- It contains an RTC stats logging module that allows to collect metrics and
send them to a Prometheus Pushgateway server for live visualization with Grafana.
- It allows to override getUserMedia and getDisplayMedia calls.
- It allows to define alert rules and generate reports.
## Install
The tool can be executed from sources, using the pre built executables or using the Docker image.
Using Npm:
```bash
npm install -g @vpalmisano/webrtcperf
# Install FFMpeg:
sudo apt install ffmpeg # Linux
# or:
brew install ffmpeg # MacOS
# Run a Jitsi test:
webrtcperf \
--url="https://meet.jit.si/${JITSI_ROOM_URL}#config.prejoinPageEnabled=false" \
--sessions=1 \
--display='' \
--show-page-log=false
# Press to stop.
```
Using Docker:
```bash
docker pull ghcr.io/vpalmisano/webrtcperf
docker run -it --rm \
-v /dev/shm:/dev/shm \
ghcr.io/vpalmisano/webrtcperf \
--url="https://meet.jit.si/$JITSI_ROOM_URL#config.prejoinPageEnabled=false" \
--show-page-log=false \
--sessions=1 \
--tabs-per-session=1
```
Stop the tool pressing `q` (normal browser close) or `x` (it will close the
process immediately).
## Configuration options
See the [config documentation](https://vpalmisano.github.io/webrtcperf/documents/config.html).
## Statistics
Example output:
```
-- Mon, 06 Feb 2023 20:46:34 GMT -------------------------------------------------------------------
name count sum mean stddev 5p 95p min max
System CPU 1 15.89 0.00 15.89 15.89 15.89 15.89 %
System GPU 1 0.00 0.00 0.00 0.00 0.00 0.00 %
System Memory 1 72.18 0.00 72.18 72.18 72.18 72.18 %
CPU/page 1 84.42 84.42 0.00 84.42 84.42 84.42 84.42 %
Memory/page 1 1206.90 1206.90 0.00 1206.90 1206.90 1206.90 1206.90 MB
Pages 1 1 1 0 1 1 1 1
Errors 1 0 0 0 0 0 0 0
Warnings 1 0 0 0 0 0 0 0
Peer Connections 1 2 2 0 2 2 2 2
-- Inbound audio -----------------------------------------------------------------------------------
rate 2 28.73 14.36 14.36 0.00 28.73 0.00 28.73 Kbps
lost 1 0.00 0.00 0.00 0.00 0.00 0.00 %
jitter 2 0.00 0.00 0.00 0.00 0.00 0.00 s
avgJitterBufferDelay 1 35.29 0.00 35.29 35.29 35.29 35.29 ms
-- Inbound video -----------------------------------------------------------------------------------
received 2 2.66 1.33 1.32 0.01 2.64 0.01 2.64 MB
rate 2 967.41 483.71 483.71 0.00 967.41 0.00 967.41 Kbps
lost 1 0.00 0.00 0.00 0.00 0.00 0.00 %
jitter 2 0.01 0.01 0.01 0.02 0.01 0.02 s
avgJitterBufferDelay 1 50.48 0.00 50.48 50.48 50.48 50.48 ms
width 2 960 320 640 1280 640 1280 px
height 2 540 180 360 720 360 720 px
fps 1 15 0 15 15 15 15 fps
-- Outbound audio ----------------------------------------------------------------------------------
rate 2 42.84 21.42 21.42 0.00 42.84 0.00 42.84 Kbps
lost 1 0.00 0.00 0.00 0.00 0.00 0.00 %
roundTripTime 1 0.001 0.000 0.001 0.001 0.001 0.001 s
-- Outbound video ----------------------------------------------------------------------------------
sent 2 3.25 1.62 1.58 0.04 3.21 0.04 3.21 MB
rate 2 1131.25 565.63 565.63 0.00 1131.25 0.00 1131.25 Kbps
lost 1 0.00 0.00 0.00 0.00 0.00 0.00 %
roundTripTime 1 0.001 0.000 0.001 0.001 0.001 0.001 s
qualityLimitResolutionChanges 2 2 1 1 0 2 0 2
qualityLimitationCpu 2 0 0 0 0 0 0 0 %
qualityLimitationBandwidth 2 20 10 10 0 20 0 20 %
sentActiveEncodings 2 2 1 1 3 1 3 encodings
sentMaxBitrate 2 3700.00 1850.00 350.00 1500.00 2200.00 1500.00 2200.00 Kbps
width 2 640 640 0 1280 0 1280 px
height 2 360 360 0 720 0 720 px
fps 2 12 12 0 25 0 25 fps
pliCountReceived 2 1 0 1 2 1 2
```
Statistics values:
| Name | Count | Description |
| :------------------------ | :----------- | :----------- |
| cpu | Total sessions | The browser process cpu usage. |
| memory | Total sessions | The browser process memory usage. |
| tabs | Total sessions | The browser current opened tabs. |
| received | Total inbound streams | The `bytesReceived` value for each stream. |
| sent | Total outbound streams | The `bytesSent` value for each stream. |
| retransmitted | Total outbound streams | The `retransmittedBytesSent` value for each stream. |
| rate | Total streams | The stream bitrate. |
| lost | Total streams | The stream [lost packets](https://www.w3.org/TR/webrtc-stats/#dom-rtcreceivedrtpstreamstats-packetslost) %. |
| jitter | Total streams | The stream [jitter](https://www.w3.org/TR/webrtc-stats/#dom-rtcreceivedrtpstreamstats-jitter) in seconds. |
| avgJitterBufferDelay | Total decoded tracks | The inbound average [jitter buffer delay](https://www.w3.org/TR/webrtc-stats/#dom-rtcinboundrtpstreamstats-jitterbufferdelay). |
| qualityLimitResolutionChanges | Total outbound video streams | The `qualityLimitationResolutionChanges` [value](https://w3c.github.io/webrtc-stats/#dom-rtcoutboundrtpstreamstats-qualitylimitationresolutionchanges) for each outbound video stream. |
| width | Total sent or received videos | The sent or received video width. |
| height | Total sent or received videos | The sent or received video height. |
| fps | Total sent | The sent video frames per second. |
## Prometheus / Grafana
See the [prometheus stack](prometheus-stack/README.md).
## Examples
### Mediasoup demo
Starts one send-receive participant:
```sh
docker run -it --rm --name=webrtcperf-publisher \
-v /dev/shm:/dev/shm \
ghcr.io/vpalmisano/webrtcperf \
--url=$MEDIASOUP_DEMO_URL \
--url-query='roomId=test&displayName=Publisher($s-$t)' \
--sessions=1 \
--tabs-per-session=1
```
Starts 10 receive-only participants:
```sh
docker run -it --rm --name=webrtcperf-viewer \
-v /dev/shm:/dev/shm \
ghcr.io/vpalmisano/webrtcperf \
--url=$MEDIASOUP_DEMO_URL \
--url-query='roomId=test&displayName=Viewer($s-$t)&produce=false' \
--sessions=1 \
--tabs-per-session=10
```
### Edumeet
Starts one send-receive participant, with a random audio activation pattern:
```sh
docker run -it --rm \
-v /dev/shm:/dev/shm \
-v $PWD/examples:/scripts:ro \
ghcr.io/vpalmisano/webrtcperf \
--url=$EDUMEET_URL \
--url-query='displayName=Publisher($s-$t)' \
--script-path=/scripts/edumeet-sendrecv.js \
--sessions=1 \
--tabs-per-session=1
```
Starts 10 receive-only participants:
```sh
docker run -it --rm \
-v /dev/shm:/dev/shm \
-v $PWD/examples:/scripts:ro \
ghcr.io/vpalmisano/webrtcperf \
--url=$EDUMEET_URL \
--url-query='displayName=Viewer($s-$t)' \
--script-path=/scripts/edumeet-recv.js \
--sessions=1 \
--tabs-per-session=10
```
### Jitsi
Starts one send-receive participant:
```sh
docker run -it --rm \
-v /dev/shm:/dev/shm \
ghcr.io/vpalmisano/webrtcperf \
--url=$JITSI_ROOM_URL \
--url-query='#config.prejoinPageEnabled=false&userInfo.displayName=Participant($s-$t)' \
--sessions=1 \
--tabs-per-session=1
```
Starts 10 receive-only participants:
```sh
docker run -it --rm \
-v /dev/shm:/dev/shm \
ghcr.io/vpalmisano/webrtcperf \
--url=$ROOM_URL \
--url-query='#config.prejoinPageEnabled=false&userInfo.displayName=Participant($s-$t)' \
--sessions=1 \
--tabs-per-session=10
```
## Running from source code
The `DEBUG_LEVEL` environment variable can be used to enable debug messages;
see [debug-level](https://github.com/commenthol/debug-level#readme) for syntax.
```sh
git clone https://github.com/vpalmisano/webrtcperf.git
cd webrtcperf
# Optional: build the chromium customized version
# cd chromium
# ./build.sh setup
# ./build.sh apply_patch
# ./build.sh build
# install the package (on Ubuntu/Debian)
# dpkg -i ./chromium-browser-unstable_-1_amd64.deb
# cd ..
yarn build
# sendrecv test
DEBUG_LEVEL=DEBUG:* yarn start \
--url=https://127.0.0.1:3443/test \
--url-query='displayName=SendRecv($s/$S-$t/$T)' \
--script-path=./examples/edumeet-sendrecv.js \
--sessions=1 \
--tabs-per-session=1
# recv only
DEBUG_LEVEL=DEBUG:* yarn start \
--url=https://127.0.0.1:3443/test \
--url-query='displayName=Recv($s/$S-$t/$T)' \
--script-path=./examples/edumeet-recv.js \
--sessions=1 \
--tabs-per-session=10
```
## Using the VMAF calculator
Run a test adding the following options:
```sh
--script-params="{timestampWatermarkVideo:true,saveSendVideoTrack:'0',saveRecvVideoTrack:'1'}" \
--server-port=5000 \
--server-use-https=true \
--server-data=/data \
--vmaf-path /data
```
With `saveSendVideoTrack` and `saveRecvVideoTrack` you can specify the sessions that will be saved at sender and receiver side (in this example it will save all the video streams sent in the session with index `0` and received in session `1`).
The sent/received videos will be saved in the `/data` directory.
The tool will generate a `.vmaf.json` and a `.vmaf.png` files in the `data/vmaf` directory. Adding the `--vmaf-preview` option, a `.mp4` file containing the side-by-side video comparison will be generated.
## Using the VISQOL calculator
Run a test adding the following options:
```sh
--script-params="{saveSendAudioTrack:'0',saveRecvAudioTrack:'1'}" \
--server-port=5000 \
--server-use-https=true \
--server-data=/data \
--visqol-path /data
```
With `saveSendAudioTrack` and `saveRecvAudioTrack` you can specify the sessions that will be saved at sender and receiver side (in this example it will save all the audio streams sent in the session with index `0` and received in session `1`).
The sent/received audio files will be saved in the `/data` directory.
The tool will generate a `visqol.csv` file in the `data` directory.
## Authors
- Vittorio Palmisano [[github](https://github.com/vpalmisano)]
## License
[AGPL](./LICENSE)