https://github.com/openhft/chronicle-core
Low level access to native memory, JVM and OS.
https://github.com/openhft/chronicle-core
Last synced: 8 months ago
JSON representation
Low level access to native memory, JVM and OS.
- Host: GitHub
- URL: https://github.com/openhft/chronicle-core
- Owner: OpenHFT
- License: apache-2.0
- Created: 2015-02-24T13:42:22.000Z (almost 11 years ago)
- Default Branch: ea
- Last Pushed: 2025-05-08T16:53:18.000Z (8 months ago)
- Last Synced: 2025-05-08T17:24:22.593Z (8 months ago)
- Language: Java
- Homepage:
- Size: 5.82 MB
- Stars: 601
- Watchers: 47
- Forks: 131
- Open Issues: 8
-
Metadata Files:
- Readme: README.adoc
- License: LICENSE
Awesome Lists containing this project
README
= Chronicle-Core: An Advanced Low-Level Library
Chronicle Software
:css-signature: demo
:toc: macro
:toclevels: 2
:icons: font
image:https://maven-badges.herokuapp.com/maven-central/net.openhft/chronicle-core/badge.svg[caption="",link=https://maven-badges.herokuapp.com/maven-central/net.openhft/chronicle-core]
image:https://javadoc.io/badge2/net.openhft/chronicle-core/javadoc.svg[link="https://www.javadoc.io/doc/net.openhft/chronicle-core/latest/index.html"]
//image:https://javadoc-badge.appspot.com/net.openhft/chronicle-wire.svg?label=javadoc[JavaDoc, link=https://www.javadoc.io/doc/net.openhft/chronicle-core]
image:https://img.shields.io/github/license/OpenHFT/Chronicle-Core[GitHub]
image:https://img.shields.io/badge/release%20notes-subscribe-brightgreen[link="https://chronicle.software/release-notes/"]
image:https://sonarcloud.io/api/project_badges/measure?project=OpenHFT_Chronicle-Core&metric=alert_status[link="https://sonarcloud.io/dashboard?id=OpenHFT_Chronicle-Core"]
image::images/Core_line.png[width=20%]
toc::[]
== About Chronicle-Core
Chronicle-Core is an advanced low-level library that equips developers with powerful tools to interact with the operating system, manage memory, handle resources, and more.
However, it should be used with caution due to its low-level operations which, if misused, can lead to complex issues.
Here is a summary of the library's key features:
**Operating System Calls:** Chronicle-Core provides access to various system calls such as retrieving the process ID, checking the operating system, and obtaining the hostname, among others.
[source,java]
----
int processId = OS.getProcessId();
boolean isWindows = OS.isWindows();
String hostname = OS.getHostName();
----
See the section on <>
**JVM Access Methods:** To access platform specific features of the JVM.
See the section on <>
**Memory Mapped Files:** The library offers an interface for managing memory mapped files, which is useful for high-performance I/O operations.
[source,java]
----
FileChannel fc = new CleaningRandomAccessFile(fileName, "rw").getChannel();
long address = OS.map(fc, MapMode.READ_WRITE, 0, 64 << 10);
OS.memory().writeLong(1024L, 0x1234567890ABCDEFL);
OS.unmap(address, 64 << 10);
----
**Deterministic Resource Management:** Chronicle-Core features components that can be closed or reference-counted, and released deterministically without waiting for garbage collection.
**Closeable Resources:** Chronicle-Core provides an interface for managing closeable resources, which are open when created and can't be used once closed.
This helps in preventing resource leaks.
[source,java]
----
public class AbstractCloseableTest {
@Test
public void close() {
MyCloseable mc = new MyCloseable();
assertFalse(mc.isClosed());
assertEquals(0, mc.performClose);
mc.throwExceptionIfClosed();
mc.close();
assertTrue(mc.isClosed());
assertEquals(1, mc.performClose);
}
}
----
**Resource Reference Counting:** The library enables the use of reference counting for deterministic resource release.
A reference-counted resource can add reservations until it's closed.
[source,java]
----
public class AbstractReferenceCountedTest {
@Test
public void reserve() {
assertTrue(Jvm.isResourceTracing());
MyReferenceCounted rc = new MyReferenceCounted();
assertEquals(1, rc.refCount());
ReferenceOwner a = ReferenceOwner.temporary("a");
rc.reserve(a);
assertEquals(2, rc.refCount());
//...
}
}
----
See the section on <>
**Thread safety checks:** The library enables the use of thread safety checks for single-threaded components.
See the section on <>
This library also wraps up low level access to
* <>
* <>
* <>
* <>
* <> for casting types, rounding double, faster hashing.
* <>
* <> A high performance wide range histogram.
== System properties from file
Chronicle-Core's `Jvm` class automatically loads system properties from a `system.properties` file if found in the current directory or parent directory.
This feature aids in streamlining your command line.
You can specify a different properties file with the `-Dsystem.properties=my.properties` command.
[source,java]
----
static {
Jvm.init();
}
----
The choice of file to load can be overridden on the command line with `-Dsystem.properties=my.properties`
In link:https://github.com/OpenHFT/Chronicle-Core/blob/ea/src/main/java/net/openhft/chronicle/core/Jvm.java[Jvm.java] it can be seen how to guarantee that JVM class is initialized before the system property is read.
For example with Jvm.getInteger or Jvm.getLong.
A number of relevant system properties are listed in link:https://github.com/OpenHFT/Chronicle-Core/blob/ea/systemProperties.adoc[systemProperties.adoc].
NOTE: Command line-specified system properties override those in the `system.properties` file.
== Chronicle-Core Initialization
Chronicle-Core offers an initialization class, link:https://github.com/OpenHFT/Chronicle-Core/blob/ea/src/main/java/net/openhft/chronicle/core/ChronicleInit.java[`ChronicleInit`], that enables developers to run their own code at startup.
This code can be executed before and/or after the execution of Chronicle's static initializers, which perform tasks such as system property loading.
`ChronicleInit` allows the developer to hook in their own code to be run at startup before and/or after the Chronicle static initialisers are run.
Chronicle static initialisers perform tasks such as loading system properties, so it is possible, for example, to override system properties using `ChronicleInit`.
To this end, `ChronicleInit` introduces the following system properties:
. "*chronicle.init.runnable*"
+
This system property specifies a fully qualified class name that will be run before any system property is read by Chronicle code, allowing the class to set them to the desired values.
The class should contain an empty static `init()` method that is called to trigger class load.
. "*chronicle.postinit.runnable*"
+
This system property specifies a fully qualified class name that will run only once after the Jvm initialisation static class.
The class should contain an empty static `postInit()` method that is called to trigger class load.
The alternative way to using the above system properties is to implement the `ChronicleInitRunnable` interface whose implementing classes may be listed in the `META-INF/services/net.openhft.chronicle.core.ChronicleInitRunnable` file in any JAR in classpath to be discovered via `ServiceLoader` JVM facility.
It can provide both init and post-init functionalities by implementing the `ChronicleInitRunnableRunnable.run()` and `ChronicleInitRunnable.postInit()` methods.
== Off Heap Memory Access
This allows you to access native memory using primitives and some thread safe operations.
[source,java]
----
Memory memory = OS.memory();
long address = memory.allocate(1024);
try {
memory.writeInt(address, 1);
assert memory.readInt(address) == 1;
final boolean swapped = memory.compareAndSwapInt(address, 1, 2);
assert swapped;
assert memory.readInt(address) == 2;
} finally {
memory.freeMemory(address, 1024);
}
----
== JVM Access Methods
Check the JVM is running in debug mode
[source,java]
----
if (Jvm.isDebug()) {
// running in debug.
----
Rethrow a checked exception as an unchecked one.
[source,java]
----
try {
// IO operation
} catch (IOException ioe) {
throw Jvm.rethrow(ioe);
}
----
Get a Field for a Class by name
[source,java]
----
Field theUnsafe = Jvm.getField(Unsafe.class, "theUnsafe");
Unsafe unsafe = (Unsafe) theUnsafe.get(null);
----
== OS Calls
Access to system calls
[source,java]
----
int processId = OS.getProcessId();
int maxProcessId = OS.getMaxProcessId();
int pageSize = OS.getPageSize();
boolean isWindows = OS.isWindows();
boolean is64bit = OS.is64Bit();
String hostname = OS.getHostName();
String username = OS.getUserName();
String targetDir = OS.getTarget(); // location the target directory during builds
----
Memory mapped files
[source,java]
----
FileChannel fc = new CleaningRandomAccessFile(fileName, "rw").getChannel();
// map in 64 KiB
long address = OS.map(fc, MapMode.READ_WRITE, 0, 64 << 10);
// use address
OS.memory().writeLong(1024L, 0x1234567890ABCDEFL);
// unmap memory region
OS.unmap(address, 64 << 10);
----
== Deterministic Resource Management
Component which are closeable or reference counted can be released deterministically without waiting for a GC.
=== Closeable Resources
A `Closeable` resources has a simple lifecycle.
It is open when created, and cannot be used once closed.
[source,Java]
----
public class AbstractCloseableTest {
@Test
public void close() {
MyCloseable mc = new MyCloseable();
assertFalse(mc.isClosed());
assertEquals(0, mc.performClose);
mc.throwExceptionIfClosed();
mc.close();
assertTrue(mc.isClosed());
assertEquals(1, mc.performClose);
mc.close();
assertTrue(mc.isClosed());
assertEquals(1, mc.performClose);
}
@Test(expected = IllegalStateException.class)
public void throwExceptionIfClosed() {
MyCloseable mc = new MyCloseable();
mc.close();
mc.throwExceptionIfClosed();
}
@Test
public void warnAndCloseIfNotClosed() {
Map map = Jvm.recordExceptions();
MyCloseable mc = new MyCloseable();
mc.warnAndCloseIfNotClosed();
Jvm.resetExceptionHandlers();
assertEquals("Discarded without closing\n" +
"java.lang.IllegalStateException: net.openhft.chronicle.core.StackTrace: Created Here",
map.keySet().stream()
.map(e -> e.message + "\n" + e.throwable)
.collect(Collectors.joining(", ")));
}
static class MyCloseable extends AbstractCloseable {
int performClose;
@Override
protected void performClose() {
performClose++;
}
}
}
----
=== Resource Reference Counting
Use reference counting to deterministically release resources.
A reference counted resource can add reservations until closed.
[source,Java]
----
public class AbstractReferenceCountedTest {
@Test
public void reserve() {
assertTrue(Jvm.isResourceTracing());
MyReferenceCounted rc = new MyReferenceCounted();
assertEquals(1, rc.refCount());
ReferenceOwner a = ReferenceOwner.temporary("a");
rc.reserve(a);
assertEquals(2, rc.refCount());
ReferenceOwner b = ReferenceOwner.temporary("b");
rc.reserve(b);
assertEquals(3, rc.refCount());
try {
rc.reserve(a);
fail();
} catch (IllegalStateException ignored) {
}
assertEquals(3, rc.refCount());
rc.release(b);
assertEquals(2, rc.refCount());
rc.release(a);
assertEquals(1, rc.refCount());
assertEquals(0, rc.performRelease);
rc.releaseLast();
assertEquals(0, rc.refCount());
assertEquals(1, rc.performRelease);
}
@Test
public void reserveWhenClosed() {
MyReferenceCounted rc = new MyReferenceCounted();
assertEquals(1, rc.refCount());
ReferenceOwner a = ReferenceOwner.temporary("a");
rc.reserve(a);
assertEquals(2, rc.refCount());
assertFalse(rc.isClosed());
rc.closeable.close();
assertEquals(2, rc.refCount());
assertTrue(rc.isClosed());
ReferenceOwner b = ReferenceOwner.temporary("b");
try {
rc.reserve(b);
fail();
} catch (IllegalStateException ignored) {
}
assertEquals(2, rc.refCount());
assertFalse(rc.tryReserve(b));
assertEquals(2, rc.refCount());
rc.release(a);
assertEquals(1, rc.refCount());
assertEquals(0, rc.performRelease);
rc.throwExceptionIfReleased();
rc.releaseLast();
assertEquals(0, rc.refCount());
assertEquals(1, rc.performRelease);
rc.throwExceptionBadResourceOwner();
try {
rc.throwExceptionIfClosed();
fail();
} catch (IllegalStateException ignored) {
}
try {
rc.throwExceptionIfReleased();
fail();
} catch (IllegalStateException ignored) {
}
}
@Test
public void throwExceptionBadResourceOwner() {
MyReferenceCounted rc = new MyReferenceCounted();
MyReferenceCounted rc2 = new MyReferenceCounted();
rc.reserve(rc2);
rc.throwExceptionBadResourceOwner();
rc2.closeable.close();
try {
rc.throwExceptionBadResourceOwner();
fail();
} catch (IllegalStateException ignored) {
}
rc.release(rc2);
rc.releaseLast();
}
@Test
public void throwExceptionIfClosed() {
MyReferenceCounted rc = new MyReferenceCounted();
rc.throwExceptionIfClosed();
rc.closeable.close();
try {
rc.throwExceptionIfClosed();
fail();
} catch (IllegalStateException ignored) {
}
}
static class MyReferenceCounted extends AbstractReferenceCounted {
final AbstractCloseable closeable;
int performRelease;
public MyReferenceCounted() {
this(new AbstractCloseableTest.MyCloseable());
}
public MyReferenceCounted(AbstractCloseable abstractCloseable) {
super(abstractCloseable);
closeable = abstractCloseable;
}
@Override
protected void performRelease() {
performRelease++;
}
}
}
----
[source,java]
----
MappedFile mf = MappedFile.mappedFile(tmp, chunkSize, 0);
MappedBytesStore bs = mf.acquireByteStore(chunkSize + (1 << 10));
assertEquals(2, mf.refCount());
assertEquals(3, bs.refCount());
assertEquals("refCount: 2, 0, 3", mf.referenceCounts());
mf.close();
assertEquals(2, bs.refCount());
assertEquals("refCount: 1, 0, 2", mf.referenceCounts());
bs2.releaseLast();
assertEquals(1, mf.refCount());
assertEquals(1, bs.refCount());
bs.releaseLast();
assertEquals(0, bs.refCount());
assertEquals(0, mf.refCount());
assertEquals("refCount: 0, 0, 0", mf.referenceCounts());
----
=== Releasing Resources
Releasing resources can be managed by starting the `BACKGROUND_RESOURCE_RELEASER` thread or alternatively it can be managed in a user defined thread.
To start the `BACKGROUND_RESOURCE_RELEASER` thread, both system properties `background.releaser` and `background.releaser.thread` should be set to `true`.
In this condition, the thread starts as a daemon thread and invokes `BackgroundResourceReleaser.runReleaseResources()`.
If only `background.releaser.thread` is set to `false`, resources will still be queued for releasing, but they need to be released explicitly by calling `BackgroundResourceReleaser.releasePendingResources()`.
If `background.releaser` is set to `false` regardless of `background.releaser.thread`, resources are not queued for release and release will be done synchronously (by calling the relevant close() function).
Calling `BackgroundResourceReleaser.stop()` releases pending resources and then stops the `BACKGROUND_RESOURCE_RELEASER` thread.
To make sure the shutdown hook does not prevent classes from unloading, deregister the shutdown hook by calling `PriorityHook.clear()`.
.Resource Release Configurations
[%header,cols=3]
|===
| `background.releaser.thread` | `background.releaser` | Release Behaviour
| `true` | `true` | Resources are queued and then released in the `BACKGROUND_RESOURCE_RELEASER` thread.
| `false` | `true` | Rresources are queued but should be released in a user thread by calling `BackgroundResourceReleaser.releasePendingResources()`.
| X | `false` | Resources are not queued and are released synchronously.
|===
== Thread Safety Checks
Classes that are designed for single-threaded use can implement `net.openhft.chronicle.core.io.SingleThreadedChecked`.
There are a number of implementations of this in Chronicle-Core, including `net.openhft.chronicle.core.io.AbstractCloseable`,
and many Chronicle library classes extend this class. When the user calls a method which _must_ be called single-threaded, these methods call a private method to check that `Thread.currentThread()` is the same as the first thread that used the object, and throws an exception if not. `AbstractCloseable` also provides very helpful functionality (if resource tracing is on) to keep track of the stack trace of where it was used first by the use-by thread - this provides invaluable when diagnosing unexpected sharing of an object between threads.
Thread safety checking can be turned off with the system property `disable.single.threaded.check`, once the application is thoroughly tested.
An object can be handed-off between threads by calling `singleThreadedCheckReset`.
=== Thread safety patterns
Some patterns which are commonly used in Chronicle's libraries are below.
==== Initialise in one thread, use in another
If using an object which is single-threaded, a common pattern is to construct and initialise your object on the main thread before handing it off to an event loop or worker thread. Call `singleThreadedCheckReset` after initialisation.
```java
// in the Main thread
ChronicleQueue q = createMyQueue();
ExcerptTailer tailer = q.createTailer().toEnd(); // toEnd() checks thread safety and thus initialises the used-by thread
tailer.singleThreadedCheckReset();
// now use the tailer in an event loop
```
==== Reset in constructor
If writing an object to use `net.openhft.chronicle.core.io.SingleThreadedChecked` then a common pattern (and a variant
of the above) is to call `singleThreadedCheckReset` at the end of the constructor e.g. from Chronicle Queue:
```java
public ExcerptTailer createTailer(String id) {
///
final StoreTailer storeTailer = new StoreTailer(this, pool, indexUpdater); // initialises state and sets the used-by thread
///
storeTailer.singleThreadedCheckReset();
return storeTailer;
}
```
==== Delegate through SingleThreadedCheck methods
If implementing `SingleThreadedCheck` then any methods should also delegate to any contained `SingleThreadedCheck`
objects i.e.
```java
@Override
public void singleThreadedCheckReset() {
// perform my reset
///
// call singleThreadedCheckReset on any single-threaded fields
this.bytes.singleThreadedCheckReset();
}
```
== Object Pooling
Chronicle-Core provides object pooling for strings and enums, allowing you to convert a `CharSequence` into a `String` of a specific `Enum` type efficiently.
[source,java]
----
Bytes> b = Bytes.from("Hello World");
b.readSkip(6);
StringInterner si = new StringInterner(128);
String s = si.intern(b);
String s2 = si.intern(b);
assertEquals("World", s);
assertSame(s, s2);
----
== Class Local Caching
Add caching of a data structure for each class using a lambda
[source,java]
----
public static final ClassLocal ENUM_INTERNER =
ClassLocal.withInitial(c -> new EnumInterner<>(c));
E enumValue = ENUM_INTERNER.get(enumClass).intern(stringBuilder);
----
== Maths Functions
Maths functions to support rounds
[source,java]
----
double a = 0.1;
double b = 0.3;
double c= Maths.round2(b - a); // 0.2 rounded to 2 decimal places
----
Checking type conversions
[source,java]
----
int i = Maths.toInt32(longValue);
----
== Serializable Lambdas
There is a number of FunctionalInterfaces you can utilise as method arguments.
This allows implicitly making a lambda Serializable.
[source,java]
----
// in KeyedVisitable
default R applyToKey(K key, @NotNull SerializableFunction function) {
// in code
String fullename = map.applyToKey("u:123223", u -> u.getFullName());
----
== Histogram
A high dynamic range histogram with tunable accuracy.
[source,java]
----
Histogram h = new Histogram(32, 4);
long start = instance.ticks(), prev = start;
for (int i = 0; i <= 1000_000_000; i++) {
long now = instance.ticks();
long time = now - prev;
h.sample(time);
prev = now;
}
System.out.println(h.toLongMicrosFormat(instance::toMicros));
----
== JLBH
JLBH has moved home and now lives in its own project, see https://github.com/OpenHFT/JLBH[JLBH].
== Loop Block Monitor tool
The tool to summarise the thread stack traces is here.
`net.openhft.chronicle.core.threads.MonitorProfileAnalyserMain`