https://github.com/openhft/jlbh

Last synced: 5 months ago
JSON representation

JLBH

Host: GitHub
URL: https://github.com/openhft/jlbh
Owner: OpenHFT
License: apache-2.0
Created: 2019-10-29T14:28:52.000Z (over 6 years ago)
Default Branch: ea
Last Pushed: 2024-05-29T07:13:51.000Z (about 2 years ago)
Last Synced: 2024-05-29T07:20:10.246Z (about 2 years ago)
Language: Java
Size: 215 KB
Stars: 71
Watchers: 17
Forks: 6
Open Issues: 2
Metadata Files:
- Readme: README.adoc
- License: LICENSE

Awesome Lists containing this project

README

= Chronicle JLBH
Chronicle Software
:css-signature: demo
:toc: macro
:toclevels: 2
:icons: font

image:https://maven-badges.herokuapp.com/maven-central/net.openhft/jlbh/badge.svg[caption="",link=https://maven-badges.herokuapp.com/maven-central/net.openhft/jlbh]
image:https://javadoc.io/badge2/net.openhft/JLBH/javadoc.svg[link="https://www.javadoc.io/doc/net.openhft/chronicle-wire/latest/index.html"]
//image:https://javadoc-badge.appspot.com/net.openhft/jlbh.svg?label=javadoc[JavaDoc, link=https://www.javadoc.io/doc/net.openhft/jlbh]
image:https://img.shields.io/github/license/OpenHFT/JLBH[GitHub]
image:https://img.shields.io/badge/release%20notes-subscribe-brightgreen[link="https://chronicle.software/release-notes/"]
image:https://sonarcloud.io/api/project_badges/measure?project=OpenHFT_JLBH&metric=alert_status[link="https://sonarcloud.io/dashboard?id=OpenHFT_JLBH"]
This project is actively maintained; see link:https://github.com/OpenHFT/JLBH/releases[GitHub releases] for the latest versions.

toc::[]

== About

Java Latency Benchmark Harness is a tool that allows you to benchmark your code running in context, rather than in a microbenchmark.
See <> for a series of articles introducing JLBH.
An excellent introduction can be found in http://www.rationaljava.com/2016/04/a-series-of-posts-on-jlbh-java-latency.html[this series of articles.]
link:src/main/adoc/project-requirements.adoc[The requirements document] contains detailed feature descriptions.

For terminology used throughout the project, see link:src/main/adoc/project-requirements.adoc#_7-glossary[the Glossary (section 7)].

Since those articles were written the main change has been to allow JLBH to be installed to an event loop, rather than it running in its own thread.
To do this, use the JLBH.eventLoopHandler method rather than JLBH.start.

NOTE: The JLBH harness itself runs on a single thread; benchmarked code may spawn threads, but the harness thread is unique.

For example, you can run the benchmark on your own event loop:

[source,java]
----
EventLoop eventLoop = new MediumEventLoop(null, "el", Pauser.busy(), true, null);
eventLoop.start();

JLBH jlbh = new JLBH(options);
jlbh.eventLoopHandler(eventLoop);
----

This installs JLBH onto the supplied loop instead of starting its own thread.

== Quick Start

To run a simple benchmark disable accounting for coordinated omission as follows:

[source,java]
----
JLBHOptions options = new JLBHOptions()
.warmUpIterations(100_000)
.iterations(1_000_000)
.throughput(1_000_000)
.accountForCoordinatedOmission(false) // disable correction
.jlbhTask(myTask);

new JLBH(options).start();
----

A minimal demonstration is provided in `src/test/java/net/openhft/chronicle/jlbh/ExampleJLBHMain.java`, showing how to run the harness from the command line.

Configure JLBH using the `JLBHOptions` builder:

[source,java]
----
JLBHOptions options = new JLBHOptions()
.throughput(50_000)
.iterations(1_000_000)
.jlbhTask(myTask);
new JLBH(options).start();
----

Commonly used option methods include:

* `.throughput(int)` - set target iterations per time unit
* `.iterations(long)` - number of iterations to execute
* `timeout(long)` aborts the benchmark if no samples are produced for the given number of milliseconds.

For a full list of configuration parameters see `src/main/java/net/openhft/chronicle/jlbh/JLBHOptions.java`.

For a visual overview of how a benchmark progresses, see the link:src/main/adoc/benchmark-lifecycle.adoc[benchmark lifecycle diagram].

== Additional Features

=== OS Jitter Measurement

JLBH can optionally track operating-system jitter by running a background thread that records scheduler delays as a separate probe.
This is useful when investigating latency spikes caused by kernel activity, but it incurs some overhead so it is enabled by default.
Disable it via `recordOSJitter(false)` when you want to avoid the extra monitoring.
The feature is listed in the requirements specification around lines 52 and 56 of `project-requirements.adoc`.

=== Adding custom probes

JLBH lets you time additional stages of your benchmark using `addProbe(String)`.
The returned `NanoSampler` records its own histogram.

[source,java]
----
class ProbedTask implements JLBHTask {
private JLBH jlbh;
private NanoSampler stage;

@Override
public void init(JLBH jlbh) {
this.jlbh = jlbh;
stage = jlbh.addProbe("my-stage");
}

@Override
public void run(long startTimeNs) {
long stepStart = System.nanoTime();
// ... benchmarked stage ...
stage.sampleNanos(System.nanoTime() - stepStart);

jlbh.sample(System.nanoTime() - startTimeNs);
}
}
----

For a runnable example see `src/test/java/net/openhft/chronicle/jlbh/SimpleOSJitterBenchmark.java`.

=== CSV Output

To export JLBH results in CSV format, invoke `JLBHResultSerializer.runResultToCSV(jlbhResult)`.
This writes the output to `result.csv` by default, as defined in `JLBHResultSerializer.RESULT_CSV`.

== Environment and Configuration

CPU affinity can be configured using the OpenHFT Affinity library as noted in the link:src/main/adoc/project-requirements.adoc[requirements document].

Histogramming of recorded latencies is precise.
As specified in the link:src/main/adoc/project-requirements.adoc[jlbh-requirements], JLBH generates high-resolution histograms of at least 35 bits.
This accuracy retains sub-nanosecond resolution over a wide range, so the reported percentiles closely reflect true end-to-end timings.

== Sample Benchmarks and Tests

The project ships with several example tasks and tests located under `src/test/java/net/openhft/chronicle/jlbh`.
They can serve as a starting point when writing your own benchmarks.

|===
| Class | Purpose

== Key Non-Functional Requirements

The following table summarises the main non-functional quality attributes derived from the Software Requirements Specification.

|===
| Area | Requirement

| Performance | Overhead per sample must remain below 100 ns when no additional probes are active. Histogram generation must support >=200 M iterations without heap pressure.
| Reliability | Harness must abort gracefully on interruptions or sample time-outs, and immutable result objects ensure thread-safe publication.
| Usability | Fluent builder API with sensible defaults yields a runnable benchmark in ≤10 LOC. ASCII table outputs are human-readable and CI-friendly.
| Portability | Pure-Java codebase with runtime-detected JDK optimisations; no native compilation.
| Maintainability | >=80 % unit-test coverage and adherence to Chronicle coding standards validated by SonarCloud.
| Security | No executable deserialisation; harness operates in-process. Users remain responsible for securing benchmarked code.
|===

[[further-reading]]
== Further Reading

http://www.rationaljava.com/2016/04/jlbh-introducing-java-latency.html[Introducing JLBH]

* What is JLBH
* What was the motivation for JLBH
* Differences between JMH and JLBH
* Quick start guide

=== Sample percentile output

The output of a short JLBH run may look similar to the following:

[source]
----
-------------------------------- SUMMARY (end to end) us -------------------------
Percentile run1 run2 run3 % Variation
50.0: 8.07 8.07 6.10 17.69
90.0: 11.66 11.66 9.71 11.82
99.0: 12.46 12.46 10.51 11.02
worst: 12.56 12.56 10.61 10.93
----

http://www.rationaljava.com/2016/04/jlbh-examples-1-why-code-should-be.html[Why Code Should be Benchmarked in Context]

* A side by side example using JMH and JLBH for Date serialisation
* Measuring Date serialisation in a microbenchmark
* Measuring Date serialisation as part of a proper application
* How to add a probe to your JLBH benchmark
* Understanding the importance of measuring code in context

Co-ordinated omission occurs when latency measurements ignore the time requests wait to be serviced, leading to unrealistically low results.

http://www.rationaljava.com/2016/04/jlbh-examples-2-accounting-for.html[Accounting for Coordinated Omission]

* Running JLBH with and without accounting for coordinated omission
* An example illustrating the numerical impact of coordinated omission
* A discussion about flow control

http://www.rationaljava.com/2016/04/jlbh-examples-3-affects-of-throughput.html[The Affects of Throughput on Latency]
Note: the blog deliberately uses "Affects" in the title while describing the effects of throughput on latency.

* A discussion about the effects of throughput on latency
* How to use JLBH to measure TCP loopback
* Adding probes to test both halves of the TCP round trip
* Watching the effect of increasing throughput on latency
* Understanding that you have to drop throughput to achieve good latencies at high percentiles.

http://www.rationaljava.com/2016/04/jlbh-examples-4-benchmarking-quickfix.html[Benchmarking QuickFix vs ChronicleFix]

* Using JLBH to benchmark QuickFIX
* Observing how QuickFix latencies degrade through the percentiles
* Comparing QuickFIX with Chronicle FIX

== Common Questions

=== Using JLBH as part of automated performance/performance regression testing

* Extract latency percentiles from `net.openhft.chronicle.jlbh.JLBHTest::shouldProvideResultData` for xUnit tests.
* Run them on a production-like CI server.
* When local hardware is adequate, execute them along with other tests.
* Integrate these benchmarks into the TDD cycle to expose latency-related design issues early.

=== Warm-up iterations

JLBH performs a warm-up phase before measurements start.
A rule of thumb is to warm up for about 30% of the run iterations.
Override this with `warmUpIterations(int)` if needed.

=== Throughput modelling

Configure the target rate using `throughput(int, TimeUnit)` and optionally a `LatencyDistributor` for bursty patterns.
Start with your production rate and tune to study latency trade-offs.

== Licensing and Release Information

The code is licensed under the terms described in the link:LICENSE.adoc[Apache 2.0 License].
Releases are available on link:https://github.com/OpenHFT/JLBH/releases[GitHub].

ecosyste.ms

Data

Tools

Indexes

Applications

Experiments

Awesome

https://github.com/openhft/jlbh

Awesome Lists containing this project

README