https://github.com/marschall/streaming-jobkeygenerator
a faster replacement for DefaultJobKeyGenerator
https://github.com/marschall/streaming-jobkeygenerator
spring-batch
Last synced: 3 months ago
JSON representation
a faster replacement for DefaultJobKeyGenerator
- Host: GitHub
- URL: https://github.com/marschall/streaming-jobkeygenerator
- Owner: marschall
- Created: 2021-09-02T20:54:06.000Z (over 3 years ago)
- Default Branch: master
- Last Pushed: 2021-09-04T16:43:28.000Z (over 3 years ago)
- Last Synced: 2025-01-16T02:45:00.406Z (5 months ago)
- Topics: spring-batch
- Language: Java
- Homepage:
- Size: 19.5 KB
- Stars: 0
- Watchers: 3
- Forks: 0
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
Awesome Lists containing this project
README
StreamingJobKeyGenerator
========================
A faster replacement for `DefaultJobKeyGenerator` that incrementally creates the job key and avoid large intermediate allocations.
See [marschall/jobkeygenerator-benchmarks](https://github.com/marschall/jobkeygenerator-benchmarks) for benchmarks
Implementation Notes
--------------------
- The code uses a small, limited size, `CharBuffer` to incrementally build the key and a small, limited size, `ByteBuffer` to hold the incremental UTF-8 input for the `MessageDigest`. A `CharsetEncoder` is used for the UTF-8 encoding from the `CharBuffer` to the `ByteBuffer`.
The data flow looks like this:
String -> CharBuffer -(CharsetEncoder)-> ByteBuffer -> MessageDigest
The data flow looks like a lot of wasteful memory copies but the relative cost is quite low and decreasing in newer JDKs.
- The code avoids generating the full key as a `String` using a `StringBuilder` but instead uses a limited size `CharBuffer` work buffer to incrementally build the key.
- The code avoids converting the full key to UTF-8 into a single, potentially large, `byte[]` but instead uses a limited size `ByteBuffer` work buffer to incrementally build the hash input.
- The code uses a `CharsetEncoder` to avoid creating a `OutputStreamWriter` instance to avoid creating a `byte[8192]` buffer.
- The code avoids creating a `BigInteger` and using `String#format` for hex printing. Instead it implements a custom hex printer that reuses the existing `ByteBuffer` as a work buffer to avoid intermediate allocations.
- The code runs noticeably faster in JDK 17 compared to JDK 11, so does `DefaultJobKeyGenerator` as well.
- It benefits from the following improvements in JDK 11:
- [JDK-8054307](https://bugs.openjdk.java.net/browse/JDK-8054307)
- It benefits from the following improvements in JDK 17:
- [JDK-8011135](https://bugs.openjdk.java.net/browse/JDK-8011135)
- [JDK-8250902](https://bugs.openjdk.java.net/browse/JDK-8250902)
- [JDK-8259498](https://bugs.openjdk.java.net/browse/JDK-8259498)
- [JDK-8259065](https://bugs.openjdk.java.net/browse/JDK-8259065)