https://github.com/switcherapi/feature-flag-benchmark
Evaluate performance of 5 different Feature Flag SDKs
https://github.com/switcherapi/feature-flag-benchmark
feature-flags feature-toggle switcher-api
Last synced: 12 months ago
JSON representation
Evaluate performance of 5 different Feature Flag SDKs
- Host: GitHub
- URL: https://github.com/switcherapi/feature-flag-benchmark
- Owner: switcherapi
- Created: 2022-07-29T04:56:43.000Z (over 3 years ago)
- Default Branch: master
- Last Pushed: 2024-12-13T01:38:00.000Z (over 1 year ago)
- Last Synced: 2025-02-06T05:14:41.107Z (about 1 year ago)
- Topics: feature-flags, feature-toggle, switcher-api
- Language: Java
- Homepage:
- Size: 28.3 KB
- Stars: 0
- Watchers: 0
- Forks: 1
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
Awesome Lists containing this project
README
# About
This benchmark compares 5 major Feature Flag SDKs against Switcher Client SDK for Java.
The tests included in this benchmark are focused only in performance and does not reflect nor measure any other SDK capabilities.
The intent is to gain more knowledge and define a baseline to improve Switcher Client SDK overall performance.
## Run
Build jar and run the benchmark
```
mvn clean verify
java -jar target/benchmarks.jar
```
## Tests explained
### [Async] testSwitcherRemoteThrottle, testHarness and testOptimizely
Uses in-memory cache by preloading or using async calls to remote service.
### [Local] testSwitcherLocal, testSplit and testUnleash
In-memory configuration loaded from file.
* Split SDK does not provide free access to fetch data from remote service - in this test the SDK is returning a default value.
### [Remote] testSwitcherRemote
These tests are 100% remote.
## Operations /s in 5s
| Benchmark | Type | Mode | Score | Units |
|:------------------------------------------------|:-------|:------|:----------------|:------|
| Simple Boolean (static feature flag) | Local | thrpt | 585,493,539.045 | ops/s |
| FeatureFlagBenchmark.testSwitcherRemoteThrottle | Async | thrpt | 34,327,818.480 | ops/s |
| FeatureFlagBenchmark.testSwitcherLocal | Local | thrpt | 17,861,430.894 | ops/s |
| FeatureFlagBenchmark.testHarness | Async | thrpt | 5,993,383.681 | ops/s |
| FeatureFlagBenchmark.testSplit | Local | thrpt | 5,176,741.297 | ops/s |
| FeatureFlagBenchmark.testUnleash | Local | thrpt | 2,674,586.707 | ops/s |
| FeatureFlagBenchmark.testOptimizely | Async | thrpt | 480,875.500 | ops/s |
| FeatureFlagBenchmark.testTogglz | Local | thrpt | 419,103.750 | ops/s |
| FeatureFlagBenchmark.testSwitcherRemote | Remote | thrpt | 187.247 | ops/s |
### Switcher Client SDK:
- when Async, can be up to 6x faster than the other SDKs.
- when Local, can be up to 3x faster than the other SDKs.
- 100% remote can only be enabled with Switcher Client SDK.
## Average calls/ns in 5s
| Benchmark | Mode | Score | Units |
|:------------------------------------------------|:-----|:--------------|:------|
| FeatureFlagBenchmark.testSwitcherRemoteThrottle | avgt | 31,661 | ns/op |
| FeatureFlagBenchmark.testSwitcherLocal | avgt | 63.852 | ns/op |
| FeatureFlagBenchmark.testHarness | avgt | 360.610 | ns/op |
| FeatureFlagBenchmark.testSplit | avgt | 430.571 | ns/op |
| FeatureFlagBenchmark.testUnleash | avgt | 800.930 | ns/op |
| FeatureFlagBenchmark.testOptimizely | avgt | 3,844.586 | ns/op |
| FeatureFlagBenchmark.testTogglz | avgt | 4,568.257 | ns/op |
| FeatureFlagBenchmark.testSwitcherRemote | avgt | 5,699,296.355 | ns/op |