{"id":14988097,"url":"https://github.com/apache/hbase-thirdparty","last_synced_at":"2025-08-23T00:16:58.429Z","repository":{"id":37752387,"uuid":"94973335","full_name":"apache/hbase-thirdparty","owner":"apache","description":"Mirror of Apache HBase Third Party Libs","archived":false,"fork":false,"pushed_at":"2025-03-19T05:57:11.000Z","size":420,"stargazers_count":16,"open_issues_count":4,"forks_count":50,"subscribers_count":38,"default_branch":"master","last_synced_at":"2025-04-02T12:07:21.300Z","etag":null,"topics":["database","hbase","java"],"latest_commit_sha":null,"homepage":"https://hbase.apache.org/","language":"Java","has_issues":false,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/apache.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGES.md","contributing":null,"funding":null,"license":"LICENSE.txt","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2017-06-21T07:00:07.000Z","updated_at":"2025-03-29T19:09:43.000Z","dependencies_parsed_at":"2023-02-12T08:31:31.331Z","dependency_job_id":"7e1d2a7b-eac4-4d88-a080-ee026c10bb59","html_url":"https://github.com/apache/hbase-thirdparty","commit_stats":{"total_commits":282,"total_committers":27,"mean_commits":"10.444444444444445","dds":0.6347517730496454,"last_synced_commit":"a98a74b5d2cc4090317b8293c6c52292b5f33c99"},"previous_names":[],"tags_count":67,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/apache%2Fhbase-thirdparty","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/apache%2Fhbase-thirdparty/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/apache%2Fhbase-thirdparty/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/apache%2Fhbase-thirdparty/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/apache","download_url":"https://codeload.github.com/apache/hbase-thirdparty/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":247721889,"owners_count":20985085,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["database","hbase","java"],"created_at":"2024-09-24T14:16:05.000Z","updated_at":"2025-08-23T00:16:58.367Z","avatar_url":"https://github.com/apache.png","language":"Java","funding_links":[],"categories":[],"sub_categories":[],"readme":"# HBase Thirdparty\n\u003c!---\n# Licensed to the Apache Software Foundation (ASF) under one\n# or more contributor license agreements.  See the NOTICE file\n# distributed with this work for additional information\n# regarding copyright ownership.  The ASF licenses this file\n# to you under the Apache License, Version 2.0 (the\n# \"License\"); you may not use this file except in compliance\n# with the License.  You may obtain a copy of the License at\n#\n#     http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\n--\u003e\n\nThis project packages relocated third-party libraries used by Apache HBase.\n\n\u003e DISCLAIMER: This project is for Apache HBase internal use.  Included libs\n\u003e and/or their versions are subject to change at the dictate of hbase without\n\u003e regard to the concern of others!\n\nWe have a number of submodules, one per ornery lib -- protobuf, netty, \u0026c. --\nwhere we need special-handling and then a bucket for all the rest,\nhbase-shaded-miscellaneous. This latter includes protobuf-util, gson, and guava.\n\nGeneral philosophy is many modules rather than a few fat ones so we can keep\ndependency narrow; a fat jar would put a load of unnecessaries on the\nCLASSPATH. The hbase-shaded-miscellaneous is a sort of all-the-rest but it\nis also libs that depend on each other and are awkward to disentangle.\n\nAll shading is done using the same relocation offset of\norg.apache.hbase.thirdparty. We add this prefix to the relocated thirdparty\nlibrary class names.\n\nSee the pom.xml for the explicit version of each third-party lib included.\n\nNote that in hbase-shaded-protobuf, we unzip the protobuf jar to src/main/java\nrather than to a dir under target because the jar plugin wants src here (its\nhard to convince it otherwise). We also apply some patches. Current set are:\n\n```\nHBASE-15789_V4.patch\nHBASE-17087.patch\nHBASE-17239.patch\n```\n\nIdeally we would be pushing this set up into protobuf project.\n\n### Steps to generate patches for protobuf-java:\nIn case build fails due to protobuf-java version change, we can follow below steps to generate new patch files.\n\n1) Clone the protobuf git repo.\n    ```sh\n    git clone https://github.com/protocolbuffers/protobuf\n    ```\n2) Checkout to the appropriate release tag.\n    ```sh\n    cd protobuf\n    git checkout v29.2\n    ```\n   NOTE: Ensure to update above tag based on your target protobuf-java version. Also, protobuf repo has a special way of versioning so that different languages can have different major versions. For example, here v29.2 maps to 4.29.2 for protobuf-java.\n3) Create a branch for applying the patches.\n    ```sh\n    git checkout -b apply_patches\n    ```\n4) Apply patches from hbase-thirdparty's 'hbase-shaded-protobuf/src/main/patches' one by one, in-order. For example:\n    ```sh\n    git apply --directory java/core \u003cBASE_DIR_TO_HBASE_THIRDPARTY_CODE\u003e/hbase-thirdparty/hbase-shaded-protobuf/src/main/patches/HBASE-15789_V3.patch\n    ```\n   NOTE: Ensure to replace \u003cBASE_DIR_TO_HBASE_THIRDPARTY_CODE\u003e based on your setup.\n5) Resolve any conflicts. Next, stage all changes and commit the change.\n6) Generate a patch from previous commit and save the patch file, if there are code changes/conflict.\n    ```sh\n    git diff HEAD^ HEAD \u003e HBASE-15789.patch\n    ```\n7) Trim the prefix directory 'java/core' from the generated patch.\n    ```sh\n    sed -i '' 's|java/core/src/main/java/|src/main/java/|g' HBASE-15789.patch\n    ```\n8) Repeat steps 4 to 7 for each patch.\n9) Copy and overwrite old patches at 'hbase-shaded-protobuf/src/main/patches' with the updated patches, in case there was any code changes/conflict.\n\n---\n\n## Dual JDK Requirements\n\nStarting with version 4.1.12, this project requires both JDK 8 and JDK 17 to accommodate different HBase versions:\n\n- **HBase 2.x**: Uses JDK 8 compatible modules (including `hbase-shaded-jetty` with Jetty 9)\n- **HBase 3.x**: Uses JDK 17 modules (including `hbase-shaded-jetty-12-plus-*` with Jetty 12)\n\nJetty 12 requires JDK 17 for compilation, but HBase 2.x deployments cannot move to Jetty 12 for JDK 8 compatibility. Our solution provides a single release containing modules for both JDK versions, eliminating the need for separate branches or releases.\n\n### JDK 8 Required Modules\n\nThis project has specific JDK requirements for different modules:\n\n- **hbase-unsafe**: Must be built with JDK 8 because it uses old APIs that have been removed from more modern JDKs. Due to a bug in JDK, we cannot generate this code using a more modern version of the JDK. See [HBASE-26773](https://issues.apache.org/jira/browse/HBASE-26773) for details.\n- **hbase-shaded-protobuf**: Must be built with JDK 8 because it depends on internal Java APIs such as `sun.misc.Unsafe`. These internal APIs are inaccessible when compiling with newer JDKs but release target set to JDK 8.\n\n### JDK 17 Compilation with JDK 8 Target\n\n**All other modules** (including Jetty 12 modules) use JDK 17 for compilation but with release target set to JDK 8.\n\n### Maven Enforcer Plugin Profiles\n\nThe project uses Maven enforcer plugin profiles to ensure these requirements are met:\n- **enforce-jdk8-bytecode**: Ensures hbase-unsafe and hbase-shaded-protobuf are built with JDK 8.\n- **enforce-jdk17-bytecode**: For **Jetty 12 modules** (hbase-shaded-jetty-12-plus-*), the enforcer plugin allows JDK 17 bytecode to be included from Jetty 12 dependencies, while still using JDK 8 as the release target.\n\n### Why toolchains are required?\n\nMaven needs explicit toolchain configuration to automatically select JDK 8 for hbase-unsafe and hbase-shaded-protobuf modules, and JDK 17 for all other modules including Jetty 12 modules. Environment variables alone are insufficient.\n\n### Files\n- `dev-support/generate-toolchains.sh` - Script to generate toolchains.xml with configurable paths\n- `toolchains.xml` - Generated Maven toolchains configuration file (not checked in)\n\n## Build/Deploy\n\n### Local Development Setup\n\n1. **Install both JDK versions**: JDK 8 and JDK 17\n2. **Set environment variables**:\n   ```sh\n   export JAVA8_HOME=/path/to/your/jdk8\n   export JAVA17_HOME=/path/to/your/jdk17\n   ```\n3. **Choose your toolchain setup approach** (see options below)\n\n### Toolchain Setup Options\n\n**Option 1: Project-local toolchains.xml**\n```sh\n# Generate and use project-specific toolchains\nexport JAVA8_HOME=/path/to/your/jdk8\nexport JAVA17_HOME=/path/to/your/jdk17\n\n# Below command will generate toolchains.xml in project root\n./dev-support/generate-toolchains.sh\n\n# Run build by passing toolchains.xml file to maven\nmvn clean install -t toolchains.xml\n```\n\n**Option 2: Global Maven toolchains setup**\n```sh\n# Setup toolchains in ~/.m2/ directory\nexport JAVA8_HOME=/path/to/your/jdk8\nexport JAVA17_HOME=/path/to/your/jdk17\n\n# Below command will generate toolchains.xml in project root\n./dev-support/generate-toolchains.sh\n\n# Copy the generated file to global .m2 directory\ncp toolchains.xml ~/.m2/toolchains.xml\n\n# Run build as usual\nmvn clean install\n```\n\n### CI/Jenkins Setup\n\nThe Jenkins CI environment uses a Docker-based build system that automatically handles the dual JDK requirements without any manual configuration.\n\n#### Multi-JDK Docker Environment\n\nThe Jenkins build uses a custom Dockerfile (`dev-support/jenkins/Dockerfile`) that:\n- Downloads and installs both JDK 8 (Adoptium Temurin) and JDK 17 (Adoptium Temurin)\n- Places them at standardized paths:\n  - Java 8: `/usr/lib/jvm/java-8`\n  - Java 17: `/usr/lib/jvm/java-17`\n- Includes Maven 3.9.8 for the build process\n\n#### Maven Wrapper with Automatic Toolchains\n\nThe Docker image creates a Maven wrapper that automatically handles toolchain configuration by replacing the original `mvn` command with a wrapper that always passes the `-t ${BASEDIR}/dev-support/toolchains-jenkins.xml` parameter to ensure the correct toolchains file is used for every Maven invocation.\n\n#### Automatic JDK Selection\n\nThe system uses a pre-configured toolchains file (`dev-support/toolchains-jenkins.xml`) that:\n- Defines JDK 1.8 toolchain pointing to `/usr/lib/jvm/java-8`\n- Defines JDK 17 toolchain pointing to `/usr/lib/jvm/java-17`\n- Allows Maven to automatically select the correct JDK for each module based on the toolchain requirements in their POMs\n\n#### Jenkins Build Process\n\nIn Jenkins, the build process is completely automated:\n1. Docker container starts with both JDKs pre-installed\n2. Maven wrapper automatically passes the toolchains configuration\n3. Each module uses the appropriate JDK version:\n   - `hbase-unsafe` and `hbase-shaded-protobuf`: Built with JDK 8\n   - All other modules: Built with JDK 17 (with JDK 8 release target)\n4. No manual toolchain setup or environment configuration required\n\n#### Environment Variables\n\nThe Jenkinsfile sets the following environment variables:\n```bash\nSET_JAVA_HOME=\"/usr/lib/jvm/java-17\"    # Default JDK for the build\nJAVA8_HOME=\"/usr/lib/jvm/java-8\"        # JDK 8 location\nJAVA17_HOME=\"/usr/lib/jvm/java-17\"      # JDK 17 location\n```\n\nThis automated setup ensures consistent builds across all Jenkins jobs without requiring developers or maintainers to manually configure toolchains in the CI environment.\n\n\n## Release\n\nTo cut a release candidate, update JIRA. The hbase-thirdparty currently uses\nhbase JIRA but with versions specified with a 'thirdparty-x.y.z'.\n\nUse the `dev-support/create-release` scripts found in `hbase.git:master`. This\nrequires having the `gpg-agent` running and configured correctly. Specify the\nproject name and git repo path. See `do-release-docker.sh -h` for details. For\nexample:\n\n```sh\n% mkdir ~/tmp/hbase-thirdparty-4.1.6RC0/\n% ~/src/hbase/dev-support/create-release/do-release-docker.sh \\\n    -d ~/tmp/hbase-thirdparty-4.1.6RC0 \\\n    -p hbase-thirdparty\n```\n\nTry the new artifact by having hbase use the staged jar. Do this in your hbase pom:\n\n```\ndiff --git a/pom.xml b/pom.xml\nindex 112f95a892..dab9e7a6bd 100755\n--- a/pom.xml\n+++ b/pom.xml\n@@ -1451,7 +1451,7 @@\n     \u003cspotbugs.version\u003e3.1.0-RC3\u003c/spotbugs.version\u003e\n     \u003cwagon.ssh.version\u003e2.12\u003c/wagon.ssh.version\u003e\n     \u003cxml.maven.version\u003e1.0.1\u003c/xml.maven.version\u003e\n-    \u003chbase-thirdparty.version\u003e2.1.0\u003c/hbase-thirdparty.version\u003e\n+    \u003chbase-thirdparty.version\u003e2.2.0\u003c/hbase-thirdparty.version\u003e\n     \u003c!-- Intraproject jar naming properties --\u003e\n     \u003c!-- TODO this is pretty ugly, but works for the moment.\n       Modules are pretty heavy-weight things, so doing this work isn't too bad. --\u003e\n@@ -3774,4 +3774,11 @@\n       \u003curl\u003efile:///tmp\u003c/url\u003e\n     \u003c/site\u003e\n   \u003c/distributionManagement\u003e\n+  \u003crepositories\u003e\n+    \u003crepository\u003e\n+      \u003cid\u003estaging\u003c/id\u003e\n+      \u003cname\u003estaging\u003c/name\u003e\n+      \u003curl\u003ehttps://repository.apache.org/content/repositories/orgapachehbase-1296\u003c/url\u003e\n+    \u003c/repository\u003e\n+  \u003c/repositories\u003e\n \u003c/project\u003e\n```\n\nSend out an email with details on staging repo and pointers to the uploaded\nartifacts.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fapache%2Fhbase-thirdparty","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fapache%2Fhbase-thirdparty","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fapache%2Fhbase-thirdparty/lists"}