{"id":19765789,"url":"https://github.com/openhft/chronicle-bytes","last_synced_at":"2025-05-14T21:05:07.408Z","repository":{"id":27773195,"uuid":"31261671","full_name":"OpenHFT/Chronicle-Bytes","owner":"OpenHFT","description":"Chronicle Bytes has a similar purpose to Java NIO's ByteBuffer with many extensions","archived":false,"fork":false,"pushed_at":"2025-03-24T16:04:58.000Z","size":8370,"stargazers_count":415,"open_issues_count":3,"forks_count":88,"subscribers_count":33,"default_branch":"ea","last_synced_at":"2025-04-13T16:53:41.436Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"http://chronicle.software","language":"Java","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/OpenHFT.png","metadata":{"files":{"readme":"README.adoc","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2015-02-24T13:39:36.000Z","updated_at":"2025-03-29T19:50:43.000Z","dependencies_parsed_at":"2023-10-04T17:53:32.281Z","dependency_job_id":"2a272eb8-8965-4a04-921d-8b1fa66f6d2a","html_url":"https://github.com/OpenHFT/Chronicle-Bytes","commit_stats":null,"previous_names":[],"tags_count":492,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/OpenHFT%2FChronicle-Bytes","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/OpenHFT%2FChronicle-Bytes/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/OpenHFT%2FChronicle-Bytes/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/OpenHFT%2FChronicle-Bytes/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/OpenHFT","download_url":"https://codeload.github.com/OpenHFT/Chronicle-Bytes/tar.gz/refs/heads/ea","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":254227611,"owners_count":22035669,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-12T04:19:26.840Z","updated_at":"2025-05-14T21:05:07.336Z","avatar_url":"https://github.com/OpenHFT.png","language":"Java","funding_links":[],"categories":[],"sub_categories":[],"readme":"= Chronicle Bytes\nChronicle Software\n:css-signature: demo\n:toc: macro\n:toclevels: 2\n:icons: font\n\nimage:https://maven-badges.herokuapp.com/maven-central/net.openhft/chronicle-bytes/badge.svg[caption=\"\",link=https://maven-badges.herokuapp.com/maven-central/net.openhft/chronicle-bytes]\nimage:https://javadoc.io/badge2/net.openhft/chronicle-bytes/javadoc.svg[link=\"https://www.javadoc.io/doc/net.openhft/chronicle-bytes/latest/index.html\"]\n//image:https://javadoc-badge.appspot.com/net.openhft/chronicle-bytes.svg?label=javadoc[JavaDoc, link=https://www.javadoc.io/doc/net.openhft/chronicle-bytes]\nimage:https://img.shields.io/github/license/OpenHFT/Chronicle-Bytes[GitHub]\nimage:https://img.shields.io/badge/release%20notes-subscribe-brightgreen[link=\"https://chronicle.software/release-notes/\"]\nimage:https://sonarcloud.io/api/project_badges/measure?project=OpenHFT_Chronicle-Bytes\u0026metric=alert_status[link=\"https://sonarcloud.io/dashboard?id=OpenHFT_Chronicle-Bytes\"]\n\nimage::images/Bytes_line.png[width=20%]\n\ntoc::[]\n\n== Introduction\n\nChronicle Bytes is a comprehensive library dedicated to providing low-level memory access wrappers.\nIt is constructed on the foundations of Chronicle Core, taking advantage of its direct memory and Operating System (OS) system call access.\n\nIn essence, Chronicle Bytes serves a similar purpose to Java's Non-blocking I/O (NIO) ByteBuffer, but with several additional features.\nThe API offers:\n\n- Support for 63-bit sizes.\n- Encoding for UTF-8 and ISO-8859-1 strings.\n- Thread-safe operations for off-heap memory.\n- Deterministic resource release through reference counting.\n- Compression for data types such as stop bit encoding.\n- Elastic ByteBuffer wrappers that resize as needed.\n- Direct parsing and writing of text to off-heap bytes.\n\n=== Supported Data Types\n\nThe following table outlines the operations and data types supported by Chronicle Bytes:\n\n|===\n| Operation | Indexed or Streaming | Binary | Text\n| Read/Write Binary Primitives | Both | float, double, boolean and unsigned/signed byte, short, 24-bit int, int, long, incompleteLong    | double, int, long, char, double with precision\n| Read/Write Text | Both | 8-bit/UTF-8 String with Length or Limit | 8-bit/UTF-8 String\n| Read/Write Other | Streaming | Histogram, Named Enum | BigDecimal, BigInteger, Date/Time/Zone, UUID, Hexadecimal\n| Data Driven Tests | From Files | No | Yes\n| CAS (Compare-and-Swap) | Indexed | int, long    |\n| Volatile Read/Write | Indexed | byte, short, int, long, float, double    |\n| Peek    | Both    | unsigned byte    |\n| Stop Bit Compression    | Streaming    | int, long, double, float, char    |\n| Search    | From Start    | indexOf String, findByte    |\n| addAndGet    | Indexed    | float, double, int, long    |\n| copy    | From Start    | write, copy    |\n| hash    | From Start    | byteSum, fastHash    |\n| BytesMarshallable    | Streaming    | Nested Data Structures, Expected Types Only.    |\n|===\n\n=== Explanation of Data Types\n\nHere is an explanation for each operation supported by Chronicle Bytes:\n\n|===\n| Read/Write Binary Primitives | Reading and writing of primitive data structures stored in binary form\n| Read/Write Text | Reading and writing of text data\n| Read/Write Other | Reading and writing of other data types\n| Data Driven Tests | Refer to https://en.wikipedia.org/wiki/Data-driven_testing[Data-Driven Testing]\n| CAS (Compare-and-Swap) | An atomic instruction used in multithreading for synchronization. It compares the contents of a memory location with a given value and, only if they match, updates the contents of that memory location to a new given value\n| Volatile Read/Write | Refer to http://tutorials.jenkov.com/java-concurrency/volatile.html[Volatile Variables]\n| Peek | An operation that retrieves the value of the bytes without affecting its read position\n| Stop Bit Compression | Refer to https://github.com/OpenHFT/RFC/tree/master/Stop-Bit-Encoding[Stop Bit Encoding]\n| Search | Any algorithm that addresses the search problem i.e., retrieving information stored within a data structure\n| AddAndGet | An operation that atomically adds a given value to the current value\n| Copy | Transferring data from one structure to another\n| Hash | Refer to https://en.wikipedia.org/wiki/Hash_function[Hash Function]\n| Bytes Marshallable | A serialization function\n|===\n\n=== Creating Bytes Instances\n\nThis section provides examples of how to create `Bytes` instances with different types of underlying data structures.\n\n* Creating a `Bytes` instance which wraps an on heap byte array:\n\n.Bytes which wraps an on heap byte[]\n[source,Java]\n----\nBytes\u003cbyte[]\u003e bytes = Bytes.allocateElasticOnHeap();\nbyte[] underlyingBytes = bytes.underlyingObject();\n----\n\n* Creating a `Bytes` instance which wraps a pre-sized on heap byte array:\n\n.Bytes which wraps an on heap byte[] pre-sized\n[source,Java]\n----\nBytes\u003cbyte[]\u003e bytes = Bytes.allocateElasticOnHeap(64);\nbyte[] underlyingBytes = bytes.underlyingObject();\n----\n\n* Creating a `Bytes` instance which wraps an on heap ByteBuffer:\n\n.Bytes which wraps an on heap ByteBuffer\n[source,Java]\n----\nBytes\u003cByteBuffer\u003e bytes = Bytes.elasticHeapByteBuffer(64);\nByteBuffer underlyingByteBuffer = bytes.underlyingObject();\n----\n\n* Creating a `Bytes` instance which wraps a direct ByteBuffer:\n\n.Bytes which wraps a direct ByteBuffer\n[source,Java]\n----\nBytes\u003cByteBuffer\u003e bytes = Bytes.elasticByteBuffer(64);\nByteBuffer underlyingByteBuffer = bytes.underlyingObject();\n\n// The memory can be freed deterministically when it is no longer needed.\nbytes.releaseLast();\n----\n\n* Creating a `Bytes` instance which wraps some native memory:\n\n.Bytes which wraps some native memory\n[source,Java]\n----\nBytes bytes = Bytes.allocateElasticDirect(64);\nlong memoryAddress = bytes.address();\n\n// The memory can be freed when it is no longer needed.\nbytes.releaseLast();\n----\n\n* Creating a `Bytes` instance which will wrap some native memory when used:\n\n.Bytes which will wrap some native memory when used\n[source,Java]\n----\nBytes bytes = Bytes.allocateElasticDirect();\n// use the bytes\n\n// The memory can be freed when it is no longer needed.\nbytes.releaseLast();\n----\n\n=== Understanding Bytes Capacity, Read/Write Position and Limit\n\nA `Bytes` buffer provides the following properties:\n\n- *start*: The smallest index in the buffer.\nThis is always '0L' for most implementations.\nFor Memory Mappings it is the offset within the file.\n- *capacity*: The maximum possible capacity for the buffer, which is usually 2147483632 (or more) for a `Bytes` buffer.\n- *realCapacity*: The current capacity of the buffer.\nThis value is dynamically extended as required.\n- *readLimit*: The maximum index that can be read from.\n- *writeLimit*: The maximum index that can be written to.\n- *readPosition*: The index from which to read.\nThis value should always be less than `writePosition`.\n- *writePosition*: The index at which to write.\n\n[#img-Bytes]\n.The diagram below illustrates a `Bytes` buffer with its read/write position markers.\nimage::docs/images/Figure1.png[450,450]\n\nIn the illustration above, note that `readPosition()` should always be less than or equals to `writePosition()` and greater than or equal to `start()`.\nAlso, `readLimit()` should always be less than or equals to `writeLimit()` and greater than or equal to `start()`.\n\n=== BytesStore\n\nBytesStore is a block of memory with fixed size into which you can write data and later read.\nYou can create a BytesStore using the bytes in a CharSequence, String, ByteBuffer or another BytesStore.\n\n[source,Java]\n----\n//Create a BytesStore bs using bytes in a String. This gives you a BytesStore with\n//fixed size 18.\nBytesStore bs = BytesStore.from(\"This is an example\");\n\n//or\n//Create a BytesStore from ByteBuffer\nByteBuffer buf = ByteBuffer.allocate(20);\nBytesStore bs2 = BytesStore.wrap(buf);\n----\n\nYou can see the buffer cursors of bs.\n\n[source,Java]\n----\n//Print cursors of bs.\nSystem.out.println(\"readLimit  \" + bs.readLimit());\nSystem.out.println(\"writeLimit \" + bs.writeLimit());\nSystem.out.println(\"readPosition \" + bs.readPosition());\nSystem.out.println(\"writePosition \" + bs.writePosition());\nSystem.out.println(\"capacity \" + bs.capacity());\nSystem.out.println(\"realCapacity \" + bs.realCapacity());\nSystem.out.println(\"start \" + bs.start());\n----\n\nprints\n\n[source,text]\n----\nreadLimit  18\nwriteLimit 18\nreadPosition 0\nwritePosition 0\ncapacity 18\nrealCapacity 18\nstart 0\n----\n\nYou can write into a BytesStore from an offset, however if your data is larger than the capacity of BytesStore, the ByteStore will not extend.\n\nNOTE: The returned object (BytesStore) is unchecked in terms of memory access, therefore the user code must make every effort not to exceed the underlying memory segment limit.\nOtherwise, the result is unspecified side effects including silently writing over other memory segments, and crashing the JVM.\n\n[source,Java]\n----\n//Write String \"Another example..\" into bs starting from offset 0.\nbs.writeUtf8(0,\"Another example..\");\n\n//Prints out: \"\u0011Another example..\"\nSystem.out.println( bs.to8bitString());\n----\n\n==== Bytes versus BytesStore\n\n- In contrast to a BytesStore, a Bytes extends if you write data into it which is larger than the realCapacity of the Bytes.\n\n[source,Java]\n----\n//Create a Bytes with initial capacity 14 and write into it \"This is an example\" that\n//needs 18 bytes.\nBytes\u003cbyte[]\u003e bb = Bytes.allocateElasticOnHeap(14);\nbb.write(\"This is an example\");\n----\n\nThe realCapacity of bb extends to 4096. Now, the cursors of bb Bytes are:\n\n[source,text]\n----\nreadLimit  18\nwriteLimit 2147483632\nreadPosition 0\nwritePosition 18\ncapacity 2147483632\nrealCapacity 4096\nstart 0\n----\n\n- Bytes gives you access to the cursors, thus you can use them to read/write from/into a desired index.\nIn contrast, you cannot use the cursors with a BytesStore.\n\n[source,Java]\n----\n//Write another data starting from index 5 which needs more bytes therefore bb extends.\nbb.writePosition(5);\nbb.write(\"sentence was overwritten from index 5 using writePosition cursor\");\n\n//Prints out: \"This sentence was overwritten from index 5 using writePosition cursor\"\nSystem.out.println(bb.toString());\n\n//Read bb from index 43\nbb.readPosition(43);\nString st = bb.to8bitString();\n\n//Prints out: \"using writePosition cursor\"\nSystem.out.println(st);\n----\n\n- In contrast to Bytes, BytesStore can be shared safely across threads provided that the referenced data is accessed in a thread-safe manner.\n\n=== bytesForRead() and bytesForWrite()\n\nThe `bytesForRead()` and `bytesForWrite()` methods can be used to create Bytes from a section of a BytesStore or a Bytes.\nThe size of the new Bytes depends on the `limit` and `position` cursors of the original Object.\n\n- Create bytesForRead/Write From BytesStore:\n\n. When you use bytesForWrite() `readLimit` and `writePosition` cursors are set to `start`.\n\n. When you use bytesForRead(), `readPosition` is set to `start`, and `readLimit` and `writePosition` cursors are set to `realCapacity`.\n\nThe new Bytes is not elastic and whether you use bytesForRead() or bytesForWrite() you can read and write from/into the new Bytes using cursors.\n\n[source,Java]\n----\n//Create a BytesStore bs using bytes in a String.\nBytesStore bs = BytesStore.from(\"This is an example\");\n\n//Create a Bytes from bs using bytesForRead().\nBytes bfr = bs.bytesForRead();\n\n//Create a Bytes from bs using bytesForwrite().\nBytes bfw = bs.bytesForWrite();\n----\n\nThe cursors of bs:\n\n[source,text]\n----\nreadLimit  18\nwriteLimit 18\nreadPosition 0\nwritePosition 0\ncapacity 18\nrealCapacity 18\nstart 0\n----\n\nThe cursors of bfr:\n\n[source,text]\n----\nreadLimit  18\nwriteLimit 18\nreadPosition 0\nwritePosition 18\ncapacity 18\nrealCapacity 18\nstart 0\n----\n\nThe cursors of bfw:\n\n[source,text]\n----\nreadLimit  0\nwriteLimit 18\nreadPosition 0\nwritePosition 0\ncapacity 18\nrealCapacity 18\nstart 0\n----\n\n- Create bytesForRead/Write from a Bytes:\n. When you use bytesForWrite() `writeLimit` is set to `capacity` and the new Bytes is elastic.\n\n. When you use bytesForRead() `readPosition` is set to `start`, and `writeLimit` is set to `realCapacity` and the new Bytes is *not* elastic.\n\nRegardless if bytesForRead() or bytesForWrite() is used, you can both read and write from/into the new Bytes using cursors.\n\n[source,Java]\n----\n//Create a Bytes bb with default size (256 bytes) and write a text into it.\nBytes\u003cByteBuffer\u003e bb = Bytes.elasticByteBuffer();\nbb.write(\"This is an example\");\n\n//Create a Bytes bfr2 from bb using bytesForRead().\nBytes bfr2 = bb.bytesForRead();\n\n//Create a Bytes bfw2 from bb using bytesForWrite().\nBytes bfw2 = bb.bytesForWrite();\n----\n\nThe cursors of bb:\n\n[source,text]\n----\nreadLimit  18\nwriteLimit 2147483632\nreadPosition 0\nwritePosition 18\ncapacity 2147483632\nrealCapacity 256\nstart 0\n----\n\nThe cursors of bfr2:\n\n[source,text]\n----\nreadLimit  18\nwriteLimit 256\nreadPosition 0\nwritePosition 18\ncapacity 2147483632\nrealCapacity 256\nstart 0\n----\n\nThe cursors of bfw2:\n\n[source,text]\n----\nreadLimit  18\nwriteLimit 2147483632\nreadPosition 0\nwritePosition 18\ncapacity 2147483632\nrealCapacity 256\nstart 0\n----\n\n=== Flipping Bytes\n\nThe standard Java `ByteBuffer` needs to be flipped to switch between reading and writing.\n\n`Bytes` holds a read position and a write position allowing you to write and immediately read without flipping.\n\nNOTE: The writePosition is the readLimit.\n\n=== Writing to a Hexadecimal dump\n\nWriting to a hexadecimal dump is useful for documenting the format for messages written.\nWe have used the hexadecimal dump here.\n\n.Writing primitives as binary and dumping\n[source,java]\n----\n// only used for documentation\nHexDumpBytes bytes = new HexDumpBytes();\nbytes.comment(\"true\").writeBoolean(true);\nbytes.comment(\"s8\").writeByte((byte) 1);\nbytes.comment(\"u8\").writeUnsignedByte(2);\nbytes.comment(\"s16\").writeShort((short) 3);\nbytes.comment(\"u16\").writeUnsignedShort(4);\nbytes.comment(\"char\").writeUnsignedShort('5'); // char\nbytes.comment(\"s24\").writeInt24(-6_666_666);\nbytes.comment(\"u24\").writeUnsignedInt24(16_666_666);\nbytes.comment(\"s32\").writeInt(6);\nbytes.comment(\"u32\").writeUnsignedShort(7);\nbytes.comment(\"s64\").writeLong(8);\nbytes.comment(\"f32\").writeFloat(9);\nbytes.comment(\"f64\").writeDouble(10);\n\nSystem.out.println(bytes.toHexString());\n----\n\nprints\n\n----\n59                                              # true\n01                                              # s8\n02                                              # u8\n03 00                                           # s16\n04 00                                           # u16\n35                                              # char\n56 46 9a                                        # s24\n2a 50 fe                                        # u24\n06 00 00 00                                     # s32\n07 00 00 00                                     # u32\n08 00 00 00 00 00 00 00                         # s64\n00 00 10 41                                     # f32\n00 00 00 00 00 00 24 40                         # f64\n----\n\nto read this data you can use\n\n.Reading the primitive values above\n[source,java]\n----\nboolean flag = bytes.readBoolean();\nbyte s8 = bytes.readByte();\nint u8 = bytes.readUnsignedByte();\nshort s16 = bytes.readShort();\nint u16 = bytes.readUnsignedShort();\nchar ch = bytes.readStopBitChar();\nint s24 = bytes.readInt24();\nlong u24 = bytes.readUnsignedInt24();\nint s32 = bytes.readInt();\nlong u32 = bytes.readUnsignedInt();\nlong s64 = bytes.readLong();\nfloat f32 = bytes.readFloat();\ndouble f64 = bytes.readDouble();\n----\n\n=== Writing and reading using offsets\n\nInstead of streaming the data, sometimes you need to control the placement of data, possibly at random.\n\n.Write and read primitive by offset\n[source,Java]\n----\nBytes\u003cByteBuffer\u003e bytes = Bytes.elasticHeapByteBuffer(64);\nbytes.writeBoolean(0, true);\nbytes.writeByte(1, (byte) 1);\nbytes.writeUnsignedByte(2, 2);\nbytes.writeShort(3, (short) 3);\nbytes.writeUnsignedShort(5, 4);\nbytes.writeInt(7, 6);\nbytes.writeUnsignedInt(11, 7);\nbytes.writeLong(15, 8);\nbytes.writeFloat(23, 9);\nbytes.writeDouble(27, 10);\nbytes.writePosition(35);\n\nSystem.out.println(bytes.toHexString());\n\nboolean flag = bytes.readBoolean(0);\nbyte s8 = bytes.readByte(1);\nint u8 = bytes.readUnsignedByte(2);\nshort s16 = bytes.readShort(3);\nint u16 = bytes.readUnsignedShort(5);\nint s32 = bytes.readInt(7);\nlong u32 = bytes.readUnsignedInt(11);\nlong s64 = bytes.readLong(15);\nfloat f32 = bytes.readFloat(23);\ndouble f64 = bytes.readDouble(27);\n----\n\nprints\n\n----\n00000000 59 01 02 03 00 04 00 06  00 00 00 07 00 00 00 08 Y······· ········\n00000010 00 00 00 00 00 00 00 00  00 10 41 00 00 00 00 00 ········ ··A·····\n00000020 00 24 40                                         ·$@\n----\n\nNOTE: While HexDumpBytes supports the offset methods, you need to provide the offset in binary and the dump making it more complex to use.\n\n==== Volatile read and ordered write\n\nChronicle Bytes supports variants of the write primitives which have a store barrier `writeOrderedXxxx`, and reads with a load barrier `readVolatileXxxx`\n\nNOTE: write ordered doesn't stall the pipeline to wait for the write to occur, making it possible for a single thread to read an old value after the ordered write.\n\n=== Working with text\n\nYou can also write and read text to Bytes for low level, direct to native memory text processing.\n\n.Writing primitives as text\n[source,Java]\n----\nBytes\u003cByteBuffer\u003e bytes = Bytes.elasticHeapByteBuffer(64);\nbytes.append(true).append('\\n');\nbytes.append(1).append('\\n');\nbytes.append(2L).append('\\n');\nbytes.append('3').append('\\n');\nbytes.append(4.1f).append('\\n');\nbytes.append(5.2).append('\\n');\nbytes.append(6.2999999, 3).append('\\n');\n\nSystem.out.println(bytes.toHexString());\n----\n\nprints\n\n----\n00000000 54 0a 31 0a 32 0a 33 0a  34 2e 31 0a 35 2e 32 0a T·1·2·3· 4.1·5.2·\n00000010 36 2e 33 30 30 0a                                6.300·\n----\n\n.Reading primitives as text\n[source,Java]\n----\nboolean flag = bytes.parseBoolean();\nint s32 = bytes.parseInt();\nlong s64 = bytes.parseLong();\nString ch = bytes.parseUtf8(StopCharTesters.SPACE_STOP);\nfloat f32 = bytes.parseFloat();\ndouble f64 = bytes.parseDouble();\ndouble f64b = bytes.parseDouble();\n----\n\nNOTE: There are fewer methods for text as 8, 16 and 24 bit can use methods for `int`, Unsigned int can use the `long` method.\n\n=== Reading and Writing Strings\n\nChronicle Bytes supports two encodings, ISO-8859-1 and UTF-8.\nIt also supports writing these as binary with a length prefix, and a string which should be terminated.\nBytes expects Strings to be read to a buffer for further processing, possibly with a String pool.\n\n[source,Java]\n----\nHexDumpBytes bytes = new HexDumpBytes();\nbytes.comment(\"write8bit\").write8bit(\"£ 1\");\nbytes.comment(\"writeUtf8\").writeUtf8(\"£ 1\");\nbytes.comment(\"append8bit\").append8bit(\"£ 1\").append('\\n');\nbytes.comment(\"appendUtf8\").appendUtf8(\"£ 1\").append('\\n');\n\nSystem.out.println(bytes.toHexString());\n----\n\nprints\n\n----\n03 a3 20 31                                     # write8bit\n04 c2 a3 20 31                                  # writeUtf8\na3 20 31 0a                                     # append8bit\nc2 a3 20 31 0a                                  # appendUtf8\n----\n\n[source,Java]\n----\nString a = bytes.read8bit();\nString b = bytes.readUtf8();\nString c = bytes.parse8bit(StopCharTesters.CONTROL_STOP);\nString d = bytes.parseUtf8(StopCharTesters.CONTROL_STOP);\n----\n\nBinary strings are prefixed with a https://github.com/OpenHFT/RFC/blob/master/Stop-Bit-Encoding/Stop-Bit-Encoding-1.0.adoc[Stop Bit Encoded] length.\n\n[source,Java]\n----\nHexDumpBytes bytes = new HexDumpBytes();\nbytes.comment(\"write8bit\").write8bit((String) null);\nbytes.comment(\"writeUtf8\").writeUtf8(null);\n\nSystem.out.println(bytes.toHexString());\n\nString a = bytes.read8bit();\nString b = bytes.readUtf8();\nassertEquals(null, a);\nassertEquals(null, b);\n----\n\nprints\n\n----\n80 00                                           # write8bit\n80 00                                           # writeUtf8\n----\n\nNOTE: `80 00` is the stop bit encoding for `-1` or `~0`\n\n=== Compare and Set operation\n\nIn binary, you can atomically replace an `int` or `long` on condition that it is an expected value.\n\n.Write two fields, remember where the `int` and `long` are\n[source,Java]\n----\nHexDumpBytes bytes = new HexDumpBytes();\n\nbytes.comment(\"s32\").writeUtf8(\"s32\");\nlong s32 = bytes.writePosition();\nbytes.writeInt(0);\n\nbytes.comment(\"s64\").writeUtf8(\"s64\");\nlong s64 = bytes.writePosition();\nbytes.writeLong(0);\n\nSystem.out.println(bytes.toHexString());\n----\n\nprints\n\n----\n03 73 33 32 00 00 00 00                         # s32\n03 73 36 34 00 00 00 00 00 00 00 00             # s64\n----\n\n.CAS two fields\n[source,Java]\n----\nassertTrue(bytes.compareAndSwapInt(s32, 0, Integer.MAX_VALUE));\nassertTrue(bytes.compareAndSwapLong(s64, 0, Long.MAX_VALUE));\n\nSystem.out.println(bytes.toHexString());\n----\n\nprints\n\n----\n03 73 33 32 ff ff ff 7f                         # s32\n03 73 36 34 ff ff ff ff ff ff ff 7f             # s64\n----\n\nINFO: You might wonder, how is the hex dump updated as well as the binary?\nThe `readPosition` actually holds the write position for both, which is why it has to be computed in this case.\n\n=== Stop bit compression\n\nStop Bit encoding is one form of simple compression.\nFor each 7 bits set, a byte is used with the high bit set when there is another byte to write.\n\nSee https://github.com/OpenHFT/RFC/blob/master/Stop-Bit-Encoding/Stop-Bit-Encoding-1.0.adoc[Stop Bit Encoding RFC] for more details\n\n.Writing with stop bit encoding\n[source,Java]\n----\nHexDumpBytes bytes = new HexDumpBytes();\n\nfor (long i : new long[]{\n        0, -1,\n        127, -127,\n        128, -128,\n        1 \u003c\u003c 14, 1 \u003c\u003c 21,\n        1 \u003c\u003c 28, 1L \u003c\u003c 35,\n        1L \u003c\u003c 42, 1L \u003c\u003c 49,\n        1L \u003c\u003c 56, Long.MAX_VALUE,\n        Long.MIN_VALUE}) {\n    bytes.comment(i + \"L\").writeStopBit(i);\n}\n\nfor (double d : new double[]{\n        0.0,\n        -0.0,\n        1.0,\n        1.0625,\n        -128,\n        -Double.MIN_NORMAL,\n        Double.NEGATIVE_INFINITY,\n        Double.NaN,\n        Double.POSITIVE_INFINITY}) {\n    bytes.comment(d + \"\").writeStopBit(d);\n}\n\nSystem.out.println(bytes.toHexString());\n----\n\nprints\n\n----\n00                                              # 0L\n80 00                                           # -1L\n7f                                              # 127L\nfe 00                                           # -127L\n80 01                                           # 128L\nff 00                                           # -128L\n80 80 01                                        # 16384L\n80 80 80 01                                     # 2097152L\n80 80 80 80 01                                  # 268435456L\n80 80 80 80 80 01                               # 34359738368L\n80 80 80 80 80 80 01                            # 4398046511104L\n80 80 80 80 80 80 80 01                         # 562949953421312L\n80 80 80 80 80 80 80 80 01                      # 72057594037927936L\nff ff ff ff ff ff ff ff 7f                      # 9223372036854775807L\nff ff ff ff ff ff ff ff ff 00                   # -9223372036854775808L\n00                                              # 0.0\n40                                              # -0.0\n9f 7c                                           # 1.0\n9f fc 20                                        # 1.0625\ne0 18                                           # -128.0\nc0 04                                           # -2.2250738585072014E-308\nff 7c                                           # -Infinity\nbf 7e                                           # NaN\nbf 7c                                           # Infinity\n----\n\nTo read these you need either `long x = bytes.readStopBit()` or `double d = bytes.readStopBitDouble()`\n\n=== BytesMarshallable objects\n\nChronicle Bytes supports serializing simple objects where the type is not stored.\nThis is similar to`RawWire` in Chronicle Wire.\n\n[source,Java]\n----\n@NotNull MyByteable mb1 = new MyByteable(false, (byte) 1, (short) 2, '3', 4, 5.5f, 6, 7.7);\n@NotNull MyByteable mb2 = new MyByteable(true, (byte) 11, (short) 22, 'T', 44, 5.555f, 66, 77.77);\nZonedDateTime zdt1 = ZonedDateTime.parse(\"2017-11-06T12:35:56.775Z[Europe/London]\");\nZonedDateTime zdt2 = ZonedDateTime.parse(\"2016-10-05T01:34:56.775Z[Europe/London]\");\nUUID uuid1 = new UUID(0x123456789L, 0xABCDEF);\nUUID uuid2 = new UUID(0x1111111111111111L, 0x2222222222222222L);\n@NotNull MyScalars ms1 = new MyScalars(\"Hello\", BigInteger.ONE, BigDecimal.TEN, zdt1.toLocalDate(), zdt1.toLocalTime(), zdt1.toLocalDateTime(), zdt1, uuid1);\n@NotNull MyScalars ms2 = new MyScalars(\"World\", BigInteger.ZERO, BigDecimal.ZERO, zdt2.toLocalDate(), zdt2.toLocalTime(), zdt2.toLocalDateTime(), zdt2, uuid2);\n@NotNull MyNested mn1 = new MyNested(mb1, ms1);\n@NotNull MyNested mn2 = new MyNested(mb2, ms2);\nbytes.comment(\"mn1\").writeUnsignedByte(1);\nmn1.writeMarshallable(bytes);\nbytes.comment(\"mn2\").writeUnsignedByte(2);\nmn2.writeMarshallable(bytes);\n----\n\n.MyByteable data structure\n[source,Java]\n----\nclass MyByteable implements BytesMarshallable {\n    boolean flag;\n    byte b;\n    short s;\n    char c;\n    int i;\n    float f;\n    long l;\n    double d;\n\n    public MyByteable(boolean flag, byte b, short s, char c, int i, float f, long l, double d) {\n        this.flag = flag;\n        this.b = b;\n        this.s = s;\n        this.c = c;\n        this.i = i;\n        this.f = f;\n        this.l = l;\n        this.d = d;\n    }\n----\n\n.MyScalars data structure\n[source,Java]\n----\nclass MyScalars implements BytesMarshallable {\n    String s;\n    BigInteger bi;\n    BigDecimal bd;\n    LocalDate date;\n    LocalTime time;\n    LocalDateTime dateTime;\n    ZonedDateTime zonedDateTime;\n    UUID uuid;\n\n    public MyScalars(String s, BigInteger bi, BigDecimal bd, LocalDate date, LocalTime time, LocalDateTime dateTime, ZonedDateTime zonedDateTime, UUID uuid) {\n        this.s = s;\n        this.bi = bi;\n        this.bd = bd;\n        this.date = date;\n        this.time = time;\n        this.dateTime = dateTime;\n        this.zonedDateTime = zonedDateTime;\n        this.uuid = uuid;\n    }\n----\n\nprints\n\n----\n01                                              # mn1\n                                                # byteable\n      4e                                              # flag\n      01                                              # b\n      02 00                                           # s\n      33                                              # c\n      04 00 00 00                                     # i\n      00 00 b0 40                                     # f\n      06 00 00 00 00 00 00 00                         # l\n      cd cc cc cc cc cc 1e 40                         # d\n                                                # scalars\n      05 48 65 6c 6c 6f                               # s\n      01 31                                           # bi\n      02 31 30                                        # bd\n      0a 32 30 31 37 2d 31 31 2d 30 36                # date\n      0c 31 32 3a 33 35 3a 35 36 2e 37 37 35          # time\n      17 32 30 31 37 2d 31 31 2d 30 36 54 31 32 3a 33 # dateTime\n      35 3a 35 36 2e 37 37 35 27 32 30 31 37 2d 31 31 # zonedDateTime\n      2d 30 36 54 31 32 3a 33 35 3a 35 36 2e 37 37 35\n      5a 5b 45 75 72 6f 70 65 2f 4c 6f 6e 64 6f 6e 5d # uuid\n      24 30 30 30 30 30 30 30 31 2d 32 33 34 35 2d 36\n      37 38 39 2d 30 30 30 30 2d 30 30 30 30 30 30 61\n      62 63 64 65 66\n02                                              # mn2\n                                                # byteable\n      59                                              # flag\n      0b                                              # b\n      16 00                                           # s\n      54                                              # c\n      2c 00 00 00                                     # i\n      8f c2 b1 40                                     # f\n      42 00 00 00 00 00 00 00                         # l\n      e1 7a 14 ae 47 71 53 40                         # d\n                                                # scalars\n      05 57 6f 72 6c 64                               # s\n      01 30                                           # bi\n      01 30                                           # bd\n      0a 32 30 31 36 2d 31 30 2d 30 35                # date\n      0c 30 31 3a 33 34 3a 35 36 2e 37 37 35          # time\n      17 32 30 31 36 2d 31 30 2d 30 35 54 30 31 3a 33 # dateTime\n      34 3a 35 36 2e 37 37 35 2c 32 30 31 36 2d 31 30 # zonedDateTime\n      2d 30 35 54 30 31 3a 33 34 3a 35 36 2e 37 37 35\n      2b 30 31 3a 30 30 5b 45 75 72 6f 70 65 2f 4c 6f\n      6e 64 6f 6e 5d 24 31 31 31 31 31 31 31 31 2d 31 # uuid\n      31 31 31 2d 31 31 31 31 2d 32 32 32 32 2d 32 32\n      32 32 32 32 32 32 32 32 32 32\n----\n\n== Data driven tests\n\nThe purpose of a Lambda function is to create a simple, highly reproducible, easily testable component.\n\nOnce you have your data dumped as hexadecimal, you can create tests using that data, and make variations of those tests.\n\n=== What do we mean by a Lambda function?\n\nIn this context a Lambda function is one which is entirely input driven and produces a list of messages (one or more outputs).\n\nThe simplest Lambda function is stateless, however this has limited application.\nThey are useful for message translation.\n\nIf you need a stateful Lambda function, you can consider the input to the function to be every message it has ever consumed.\nObviously this is inefficient, however with appropriate caches in your lamdba function, you can process and produce result incrementally.\n\n=== Data in and out.\n\nWe module a Lambda function as having an interface for inputs and another for outputs.\nThese interfaces can be the same.\n\n.Sample interface for Lambda function\n[source,Java]\n----\ninterface IBytesMethod {\n    @MethodId(0x81L) // \u003c1\u003e\n    void myByteable(MyByteable byteable);\n\n    @MethodId(0x82L)\n    void myScalars(MyScalars scalars);\n\n    @MethodId(0x83L)\n    void myNested(MyNested nested);\n}\n----\n\n\u003c1\u003e assign a unique id to each method to simplify decoding/encoding.\n\nEach method needs a DTO to describe the data for that message.\n\n[source,Java]\n----\nclass MyByteable implements BytesMarshallable {\n    boolean flag;\n    byte b;\n    short s;\n    char c;\n    int i;\n    float f;\n    long l;\n    double d;\n// rest of class\n}\n\nclass MyScalars implements BytesMarshallable {\n    String s;\n    BigInteger bi;\n    BigDecimal bd;\n    LocalDate date;\n    LocalTime time;\n    LocalDateTime dateTime;\n    ZonedDateTime zonedDateTime;\n    UUID uuid;\n// rest of class\n}\n\nclass MyNested implements BytesMarshallable {\n    MyByteable byteable;\n    MyScalars scalars;\n// rest of class\n}\n----\n\nThe implementation needs to take it's output interface and implement the input interface\n\n.A simple pass through implementation\n[source,Java]\n----\nstatic class IBMImpl implements IBytesMethod {\n    final IBytesMethod out;\n\n    IBMImpl(IBytesMethod out) { this.out = out; }\n\n    @Override\n    public void myByteable(MyByteable byteable) { out.myByteable(byteable); }\n\n    @Override\n    public void myScalars(MyScalars scalars) { out.myScalars(scalars); }\n\n    @Override\n    public void myNested(MyNested nested) { out.myNested(nested); }\n}\n\n----\n\nOnce we have interfaces, DTOs, and an implementation we can setup a test harness\n\n.Setup a test harness for a Lambda function\n[source,Java]\n----\nprotected void btmttTest(String input, String output)\nthrows IOException {\n    BytesTextMethodTester tester = new BytesTextMethodTester\u003c\u003e(\n            input,\n            IBMImpl::new,\n            IBytesMethod.class,\n            output);\n    tester.run();\n    assertEquals(tester.expected(), tester.actual());\n}\n----\n\nThis allows us to give two files, one for expected inputs and one for expected outputs.\n\n[source,Java]\n----\n@Test\npublic void run()\nthrows IOException {\n    btmttTest(\"btmtt/prim-input.txt\", \"btmtt/prim-output.txt\");\n}\n----\n\nNOTE: In this case the input and outputs are expected to be the same.\n\n.Sample input/output file\n[source]\n----\n81 01                                           # myByteable\n   4e                                              # flag\n   01                                              # b\n   02 00                                           # s\n   33                                              # c\n   04 00 00 00                                     # i\n   00 00 b0 40                                     # f\n   06 00 00 00 00 00 00 00                         # l\n   cd cc cc cc cc cc 1e 40                         # d\n### End Of Block\n81 01                                           # myByteable\n   59                                              # flag\n   0b                                              # b\n   16 00                                           # s\n   54                                              # c\n   2c 00 00 00                                     # i\n   8f c2 b1 40                                     # f\n   42 00 00 00 00 00 00 00                         # l\n   e1 7a 14 ae 47 71 53 40                         # d\n### End Of Block\n82 01                                           # myScalars\n   05 48 65 6c 6c 6f                               # s\n   01 31                                           # bi\n   02 31 30                                        # bd\n   0a 32 30 31 37 2d 31 31 2d 30 36                # date\n   0c 31 32 3a 33 35 3a 35 36 2e 37 37 35          # time\n   17 32 30 31 37 2d 31 31 2d 30 36 54 31 32 3a 33 # dateTime\n   35 3a 35 36 2e 37 37 35 27 32 30 31 37 2d 31 31 # zonedDateTime\n   2d 30 36 54 31 32 3a 33 35 3a 35 36 2e 37 37 35\n   5a 5b 45 75 72 6f 70 65 2f 4c 6f 6e 64 6f 6e 5d # uuid\n   24 30 30 30 30 30 30 30 31 2d 32 33 34 35 2d 36\n   37 38 39 2d 30 30 30 30 2d 30 30 30 30 30 30 61\n   62 63 64 65 66\n### End Of Block\n83 01                                           # myNested\n                                                # byteable\n      59                                              # flag\n      0b                                              # b\n      16 00                                           # s\n      54                                              # c\n      2c 00 00 00                                     # i\n      8f c2 b1 40                                     # f\n      42 00 00 00 00 00 00 00                         # l\n      e1 7a 14 ae 47 71 53 40                         # d\n                                                # scalars\n      05 57 6f 72 6c 64                               # s\n      01 30                                           # bi\n      01 30                                           # bd\n      0a 32 30 31 36 2d 31 30 2d 30 35                # date\n      0c 30 31 3a 33 34 3a 35 36 2e 37 37 35          # time\n      17 32 30 31 36 2d 31 30 2d 30 35 54 30 31 3a 33 # dateTime\n      34 3a 35 36 2e 37 37 35 2c 32 30 31 36 2d 31 30 # zonedDateTime\n      2d 30 35 54 30 31 3a 33 34 3a 35 36 2e 37 37 35\n      2b 30 31 3a 30 30 5b 45 75 72 6f 70 65 2f 4c 6f\n      6e 64 6f 6e 5d 24 31 31 31 31 31 31 31 31 2d 31 # uuid\n      31 31 31 2d 31 31 31 31 2d 32 32 32 32 2d 32 32\n      32 32 32 32 32 32 32 32 32 32\n### End Of Block\n### End Of Test\n----\n\nIn this case, the test calls the methods with the matching method ids which in turn uses the same ids to encode the output.\n\nNOTE: Creating and maintain such tests can be an overhead you don't need.\nIn this case, you can use Chronicle Wire's YAML testing format to check functionality.\nWire can be used for most of the tests even if you intend to use Bytes for production.\n\n=== Comparison of access to native memory\n\n|===\n| Feature | ByteBuffer | Netty IOBuffer | Aeron UnsafeBuffer | Chronicle Bytes\n\n| Read/write primitives in native memory\n| Yes\n| Yes\n| Yes\n| Yes\n\n| Separate mutable interfaces\n| Runtime check\n| Runtime check\n| Yes\n| Yes\n\n| Read/Write UTF-8 strings\n| No\n| No\n| Limited to Strings\n| Supports any CharSequence + Appendable\n\n| Read/Write ISO-8859-1 strings\n| No\n| No\n| ?\n| Supports any CharSequence + Appendable\n\n| Support Endianness\n| Supports both Big and Little\n| Supports both Big and Little\n| Supports both Big and Little\n| Supports Native only\n\n| Size of buffer\n| Limited to 31-bit\n| Limited to 31-bit\n| Limited to 31-bit\n| Supports up to 63-bit\n\n| Elastic ByteBuffers\n| No\n| Yes\n| No\n| Yes\n\n| Disable bounds checks\n| No\n| No\n| Can be set globally\n| Can be set per buffer\n\n| Wrap an address\n| No\n| No\n| Yes\n| Yes\n\n| Thread-safe read/write, CAS, and atomic add operations\n| No\n| No\n| Supports int and long\n| Supports int, long, float, and double\n\n| Streaming access\n| Yes\n| Yes\n| No\n| Yes\n\n| Deterministic release of memory\n| Via internal API\n| Via internal API\n| Caller's responsibility\n| Yes, supported natively\n\n| Separate read and write position\n| No\n| Yes\n| Not applicable\n| Yes\n\n| Sync to disk of memory-mapped files\n| No\n| No\n| No\n| Supports both MS_ASYNC and MS_SYNC\n|===\n\nThe above table provides a clear comparison of the various memory access features supported by ByteBuffer, Netty IOBuffer, Aeron UnsafeBuffer, and Chronicle Bytes.\n\n== Converting bytes instances to string\n\n`Bytes` instances provided a number of `toString` methods depending on the encoding of the data within the buffer. As `Bytes` supports encoding of string data in both ISO-8859-1 and UTF-8, you will need to pick an appropriate `toString` method based on the encoding of the data within the buffer.\n\n`toString` methods provided:\n\n* `toString` renders the bytes as if they were ISO-8859-1 encoded.\n* `to8bitString` renders the bytes as if they were ISO-8859-1 encoded.\n* `toUtf8String` renders the bytes as if they were UTF-8 encoded.\n\n== Utilizing Memory Mapped Files\n\nMemory mapped files function like Bytes instances, expanding as required.\nA specific chunk size determines the growth rate as more Bytes are employed.\nA `syncMode` can be set to ensure data is flushed as each chunk is released.\n\nBelow is an example of opening two MappedBytes for the same memory region:\n\n.Open two MappedBytes for the same region of memory\n[source,java]\n----\n// Specify the file name, chunk size, and overlap size of each chunk\ntry (MappedBytes bytesW = MappedBytes.mappedBytes(file, 64 \u003c\u003c 10, 16 \u003c\u003c 10);\n        MappedBytes bytesR = MappedBytes.mappedBytes(file,  64 \u003c\u003c 10, 16 \u003c\u003c 10)) {\n        bytesW.syncMode(SyncMode.ASYNC); // Asynchronously flush each chunk to disk\n    }\n----\n\nMemory mapped files are commonly used by Chronicle Map and Chronicle Queue.\n\n== Debugging Chronicle-Bytes\n\nWhen utilizing IntelliJ IDEA, you can configure a custom renderer to view the bytes.\nPlease refer to the images below for guidance on setting this up:\n\nimage::/docs/images/customize-data-views-menu.jpg[]\n\nimage::/docs/images/customize-data-views.png[]\n\n=== Note on Code Coverage\n\nWhen conducting code coverage, it may be beneficial to exclude `AbstractBytes` as it can significantly reduce the execution speed of unit tests.\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fopenhft%2Fchronicle-bytes","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fopenhft%2Fchronicle-bytes","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fopenhft%2Fchronicle-bytes/lists"}