https://github.com/apache/hbase-thirdparty
Mirror of Apache HBase Third Party Libs
https://github.com/apache/hbase-thirdparty
database hbase java
Last synced: 4 months ago
JSON representation
Mirror of Apache HBase Third Party Libs
- Host: GitHub
- URL: https://github.com/apache/hbase-thirdparty
- Owner: apache
- License: apache-2.0
- Created: 2017-06-21T07:00:07.000Z (over 8 years ago)
- Default Branch: master
- Last Pushed: 2025-03-19T05:57:11.000Z (9 months ago)
- Last Synced: 2025-04-02T12:07:21.300Z (9 months ago)
- Topics: database, hbase, java
- Language: Java
- Homepage: https://hbase.apache.org/
- Size: 410 KB
- Stars: 16
- Watchers: 38
- Forks: 50
- Open Issues: 4
-
Metadata Files:
- Readme: README.md
- Changelog: CHANGES.md
- License: LICENSE.txt
Awesome Lists containing this project
README
# HBase Thirdparty
This project packages relocated third-party libraries used by Apache HBase.
> DISCLAIMER: This project is for Apache HBase internal use. Included libs
> and/or their versions are subject to change at the dictate of hbase without
> regard to the concern of others!
We have a number of submodules, one per ornery lib -- protobuf, netty, &c. --
where we need special-handling and then a bucket for all the rest,
hbase-shaded-miscellaneous. This latter includes protobuf-util, gson, and guava.
General philosophy is many modules rather than a few fat ones so we can keep
dependency narrow; a fat jar would put a load of unnecessaries on the
CLASSPATH. The hbase-shaded-miscellaneous is a sort of all-the-rest but it
is also libs that depend on each other and are awkward to disentangle.
All shading is done using the same relocation offset of
org.apache.hbase.thirdparty. We add this prefix to the relocated thirdparty
library class names.
See the pom.xml for the explicit version of each third-party lib included.
Note that in hbase-shaded-protobuf, we unzip the protobuf jar to src/main/java
rather than to a dir under target because the jar plugin wants src here (its
hard to convince it otherwise). We also apply some patches. Current set are:
```
HBASE-15789_V4.patch
HBASE-17087.patch
HBASE-17239.patch
```
Ideally we would be pushing this set up into protobuf project.
### Steps to generate patches for protobuf-java:
In case build fails due to protobuf-java version change, we can follow below steps to generate new patch files.
1) Clone the protobuf git repo.
```sh
git clone https://github.com/protocolbuffers/protobuf
```
2) Checkout to the appropriate release tag.
```sh
cd protobuf
git checkout v29.2
```
NOTE: Ensure to update above tag based on your target protobuf-java version. Also, protobuf repo has a special way of versioning so that different languages can have different major versions. For example, here v29.2 maps to 4.29.2 for protobuf-java.
3) Create a branch for applying the patches.
```sh
git checkout -b apply_patches
```
4) Apply patches from hbase-thirdparty's 'hbase-shaded-protobuf/src/main/patches' one by one, in-order. For example:
```sh
git apply --directory java/core /hbase-thirdparty/hbase-shaded-protobuf/src/main/patches/HBASE-15789_V3.patch
```
NOTE: Ensure to replace based on your setup.
5) Resolve any conflicts. Next, stage all changes and commit the change.
6) Generate a patch from previous commit and save the patch file, if there are code changes/conflict.
```sh
git diff HEAD^ HEAD > HBASE-15789.patch
```
7) Trim the prefix directory 'java/core' from the generated patch.
```sh
sed -i '' 's|java/core/src/main/java/|src/main/java/|g' HBASE-15789.patch
```
8) Repeat steps 4 to 7 for each patch.
9) Copy and overwrite old patches at 'hbase-shaded-protobuf/src/main/patches' with the updated patches, in case there was any code changes/conflict.
---
## Dual JDK Requirements
Starting with version 4.1.12, this project requires both JDK 8 and JDK 17 to accommodate different HBase versions:
- **HBase 2.x**: Uses JDK 8 compatible modules (including `hbase-shaded-jetty` with Jetty 9)
- **HBase 3.x**: Uses JDK 17 modules (including `hbase-shaded-jetty-12-plus-*` with Jetty 12)
Jetty 12 requires JDK 17 for compilation, but HBase 2.x deployments cannot move to Jetty 12 for JDK 8 compatibility. Our solution provides a single release containing modules for both JDK versions, eliminating the need for separate branches or releases.
### JDK 8 Required Modules
This project has specific JDK requirements for different modules:
- **hbase-unsafe**: Must be built with JDK 8 because it uses old APIs that have been removed from more modern JDKs. Due to a bug in JDK, we cannot generate this code using a more modern version of the JDK. See [HBASE-26773](https://issues.apache.org/jira/browse/HBASE-26773) for details.
- **hbase-shaded-protobuf**: Must be built with JDK 8 because it depends on internal Java APIs such as `sun.misc.Unsafe`. These internal APIs are inaccessible when compiling with newer JDKs but release target set to JDK 8.
### JDK 17 Compilation with JDK 8 Target
**All other modules** (including Jetty 12 modules) use JDK 17 for compilation but with release target set to JDK 8.
### Maven Enforcer Plugin Profiles
The project uses Maven enforcer plugin profiles to ensure these requirements are met:
- **enforce-jdk8-bytecode**: Ensures hbase-unsafe and hbase-shaded-protobuf are built with JDK 8.
- **enforce-jdk17-bytecode**: For **Jetty 12 modules** (hbase-shaded-jetty-12-plus-*), the enforcer plugin allows JDK 17 bytecode to be included from Jetty 12 dependencies, while still using JDK 8 as the release target.
### Why toolchains are required?
Maven needs explicit toolchain configuration to automatically select JDK 8 for hbase-unsafe and hbase-shaded-protobuf modules, and JDK 17 for all other modules including Jetty 12 modules. Environment variables alone are insufficient.
### Files
- `dev-support/generate-toolchains.sh` - Script to generate toolchains.xml with configurable paths
- `toolchains.xml` - Generated Maven toolchains configuration file (not checked in)
## Build/Deploy
### Local Development Setup
1. **Install both JDK versions**: JDK 8 and JDK 17
2. **Set environment variables**:
```sh
export JAVA8_HOME=/path/to/your/jdk8
export JAVA17_HOME=/path/to/your/jdk17
```
3. **Choose your toolchain setup approach** (see options below)
### Toolchain Setup Options
**Option 1: Project-local toolchains.xml**
```sh
# Generate and use project-specific toolchains
export JAVA8_HOME=/path/to/your/jdk8
export JAVA17_HOME=/path/to/your/jdk17
# Below command will generate toolchains.xml in project root
./dev-support/generate-toolchains.sh
# Run build by passing toolchains.xml file to maven
mvn clean install -t toolchains.xml
```
**Option 2: Global Maven toolchains setup**
```sh
# Setup toolchains in ~/.m2/ directory
export JAVA8_HOME=/path/to/your/jdk8
export JAVA17_HOME=/path/to/your/jdk17
# Below command will generate toolchains.xml in project root
./dev-support/generate-toolchains.sh
# Copy the generated file to global .m2 directory
cp toolchains.xml ~/.m2/toolchains.xml
# Run build as usual
mvn clean install
```
### CI/Jenkins Setup
The Jenkins CI environment uses a Docker-based build system that automatically handles the dual JDK requirements without any manual configuration.
#### Multi-JDK Docker Environment
The Jenkins build uses a custom Dockerfile (`dev-support/jenkins/Dockerfile`) that:
- Downloads and installs both JDK 8 (Adoptium Temurin) and JDK 17 (Adoptium Temurin)
- Places them at standardized paths:
- Java 8: `/usr/lib/jvm/java-8`
- Java 17: `/usr/lib/jvm/java-17`
- Includes Maven 3.9.8 for the build process
#### Maven Wrapper with Automatic Toolchains
The Docker image creates a Maven wrapper that automatically handles toolchain configuration by replacing the original `mvn` command with a wrapper that always passes the `-t ${BASEDIR}/dev-support/toolchains-jenkins.xml` parameter to ensure the correct toolchains file is used for every Maven invocation.
#### Automatic JDK Selection
The system uses a pre-configured toolchains file (`dev-support/toolchains-jenkins.xml`) that:
- Defines JDK 1.8 toolchain pointing to `/usr/lib/jvm/java-8`
- Defines JDK 17 toolchain pointing to `/usr/lib/jvm/java-17`
- Allows Maven to automatically select the correct JDK for each module based on the toolchain requirements in their POMs
#### Jenkins Build Process
In Jenkins, the build process is completely automated:
1. Docker container starts with both JDKs pre-installed
2. Maven wrapper automatically passes the toolchains configuration
3. Each module uses the appropriate JDK version:
- `hbase-unsafe` and `hbase-shaded-protobuf`: Built with JDK 8
- All other modules: Built with JDK 17 (with JDK 8 release target)
4. No manual toolchain setup or environment configuration required
#### Environment Variables
The Jenkinsfile sets the following environment variables:
```bash
SET_JAVA_HOME="/usr/lib/jvm/java-17" # Default JDK for the build
JAVA8_HOME="/usr/lib/jvm/java-8" # JDK 8 location
JAVA17_HOME="/usr/lib/jvm/java-17" # JDK 17 location
```
This automated setup ensures consistent builds across all Jenkins jobs without requiring developers or maintainers to manually configure toolchains in the CI environment.
## Release
To cut a release candidate, update JIRA. The hbase-thirdparty currently uses
hbase JIRA but with versions specified with a 'thirdparty-x.y.z'.
Use the `dev-support/create-release` scripts found in `hbase.git:master`. This
requires having the `gpg-agent` running and configured correctly. Specify the
project name and git repo path. See `do-release-docker.sh -h` for details. For
example:
```sh
% mkdir ~/tmp/hbase-thirdparty-4.1.6RC0/
% ~/src/hbase/dev-support/create-release/do-release-docker.sh \
-d ~/tmp/hbase-thirdparty-4.1.6RC0 \
-p hbase-thirdparty
```
Try the new artifact by having hbase use the staged jar. Do this in your hbase pom:
```
diff --git a/pom.xml b/pom.xml
index 112f95a892..dab9e7a6bd 100755
--- a/pom.xml
+++ b/pom.xml
@@ -1451,7 +1451,7 @@
3.1.0-RC3
2.12
1.0.1
- 2.1.0
+ 2.2.0
@@ -3774,4 +3774,11 @@
file:///tmp
+
+
+ staging
+ staging
+ https://repository.apache.org/content/repositories/orgapachehbase-1296
+
+
```
Send out an email with details on staging repo and pointers to the uploaded
artifacts.