https://github.com/t-snyder/learn-07-pulsar
Pulsar minikube deployment with cert-manager, istio ambient mode, tls connections
https://github.com/t-snyder/learn-07-pulsar
cert-manager istio-ambient-mode pulsar pulsar-helm-deployment pulsar-kube-deployment tls
Last synced: about 2 months ago
JSON representation
Pulsar minikube deployment with cert-manager, istio ambient mode, tls connections
- Host: GitHub
- URL: https://github.com/t-snyder/learn-07-pulsar
- Owner: t-snyder
- License: apache-2.0
- Created: 2025-02-28T11:33:48.000Z (about 2 months ago)
- Default Branch: main
- Last Pushed: 2025-03-03T05:46:13.000Z (about 2 months ago)
- Last Synced: 2025-03-03T06:28:46.958Z (about 2 months ago)
- Topics: cert-manager, istio-ambient-mode, pulsar, pulsar-helm-deployment, pulsar-kube-deployment, tls
- Language: Shell
- Homepage:
- Size: 123 KB
- Stars: 0
- Watchers: 1
- Forks: 0
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
- License: LICENSE
Awesome Lists containing this project
README
# Apache Pulsar External Connection Prototypes
## Purpose
The purpose of this set of prototypes is to provide simple external connection prototypes
for both pulsar cli and a java pulsar client. The prototypes first explore unencrypted
connections within the Proto-01-kube-basic, and then encrypted connections via tls with
cert-manager providing the issuers, certificates and secrets. The final 3rd prototype uses
the helm chart for tls deployment.
## Original Unmet Goal
The original purpose of these learning prototypes was to explore Kubernetes Gateway API
with Istio and Cert-Manager for external TLS connections. However, after several failed
attempts I determined that Pulsar external connections rely on the Pulsar proxy for
providing clients the correct broker. Without a deep dive into the Pulsar Proxy
code to figure out how it was handling this, and to then come up with a way for the
Gateway API to work within these boundaries I did not see a way forward. As the Proxy is
working I will leave the rest for another time when I have more time and a definite need.
## Project Dependencies
| Core Infrastructure | Version |
| --------------- | --------------- |
| Minikube | 1.34.0 |
| Kubernetes | 1.31.0 |
| Docker | 27.2.0 |
## Computer Configuration:
| Name | Description |
| --------------- | --------------------------------------- |
| Ubuntu | 20.04.6 LTS |
| Processor | Intel® Core™ i7-7700K CPU @ 4.20GHz × 8 |
| Memory | 64 GB |
## Deploying the Core Infrastructure Dependencies
Instructions for deploying the Core Infrastructure Dependencies listed above are NOT included within this set
of Prototypes as there are numerous targeted deployment instructions for each better suited for your
particular OS.
## Dependencies Deployed within the Prototype Scripts ( As Required )
| Deployed Name | Version |
| ---------------------- | ------- |
| Cert-manager | 1.15.5 |
| Istio | 1.23.2 |
| Kubernetes Gateway API | 1.2.0 |
| Metallb | 0.9.6 |
## Dependency Documentation Referenced
1. Cert-Manager - https://cert-manager.io/docs/
2. Istio Ambient Mode - https://istio.io/latest/docs/ambient/
3. Kubernetes Gateway API - https://kubernetes.io/docs/concepts/services-networking/gateway/
4. Apache Pulsar - https://pulsar.apache.org/docs/4.0.x
## Kubernetes deployment component generation
The kube based deployments were originally generated by reviewing the output of the helm
chart for minikube and tls. The following is the prime example.
***helm install --dry-run --values ${PROTODIR}/helm/values-.yaml --namespace pulsar pulsar-mini apache/pulsar > output.txt***
It should be noted that minikube has to be running in order to run this dry-run command.
## Prototype Script Functionality
**Note :**
*The commands within the shell files below are meant to be copy pasted (one or a few lines at a time) into a terminal, and not run as an automated bash script.*
## The Prototypes
The Pulsar components deployed within each of the following Prototypes consist of only the
required Pulsar components as follows:
1. Zookeeper
2. Bookkeeper (Bookie)
3. Toolset - for Pulsar CLI client
4. Broker
5. Proxy
Additional components which can be deployed within the Helm chart are excluded from this
deployment. These include Prometheus and Graphana as well as helpers such as PodMonitors.
### Proto-01-kube-basic
The purpose for this prototype is to provide unencrypted external access to the Pulsar
deployment within minikube. The prototype kube yaml components were generated by running a dry-run from the Pulsar
minikube Helm chart as follows:
***helm install --dry-run --values ${PROTODIR}/helm/values-01.yaml --namespace pulsar pulsar-mini apache/pulsar > kube-pulsar.txt***
From this output the kube deployment components were obtained. This script does a minimal
installation of only the main required Pulsar components. The pods deployed as listed below.
| Component | # Deployed |
| ------------- | ---------- |
| Zookeeper | 1 |
| Bookie | 1 |
| Toolset | 1 |
| Broker | 1 |
| Proxy | 3 |
The deployment script ***Step 01 Deploy Minikube, metallb, Pulsar*** within the scripts
folder performs the following:
1. Deploys a fresh minikube with minikube addons (dashboard, metallb);
2. Configures Metallb loadbalancer
3. deploys Pulsar and all its components into the Cluster
4. Tests access from the Pulsar CLI client
5. Allows running of the simple java test program (eclipse, maven) found in the
pulsar-client directory within this project.
### Proto-02-kube-basic-tls
The main purpose of this prototype is to provide tls encrypted external access to the Pulsar
deployment within minikube. The prototype kube yaml components were generated by running a dry-run from the Pulsar
minikube Helm chart as follows:
***helm install --dry-run --values ${PROTODIR}/helm/values-02.yaml --namespace pulsar pulsar-mini apache/pulsar > kube-pulsar.txt***
From this output the kube deployment components were obtained. This script does a minimal
installation of only the main required Pulsar components. The pods deployed as listed below.
| Component | # Deployed |
| ------------- | ---------- |
| Zookeeper | 3 |
| Bookie | 4 |
| Toolset | 1 |
| Broker | 3 |
| Proxy | 3 |
The deployment script ***Step 01 Deploy Minikube, Metallb,Cert-manager, Pulsar*** within the
scripts folder performs the following:
1. Deploys a fresh minikube with minikube addons (dashboard, metallb);
2. Configures Metallb loadbalancer
3. Deploys Kubernetes Gateway API CRDs (cert-manager deploy uses)
4. Deploys istio in Ambient mode.
5. Deploys Cert-Manager
6. Deploys Pulsar and all required components into the Cluster
7. Sets pulsar namespace to istio ambient mode which initiates mTLS between pods
8. Tests access from the Pulsar CLI client
9. Allows running of the simple java test program (eclipse, maven) found in the
pulsar-client directory within this project.
### Proto-03-helm-basic-tls
The main purpose of this prototype is to provide tls encrypted external access to the Pulsar
deployment within minikube. The deployment uses the Pulsar Helm chart to deploy the following
Pulsar components.
| Component | # Deployed |
| ------------- | ---------- |
| Zookeeper | 3 |
| Bookie | 4 |
| Toolset | 1 |
| Broker | 3 |
| Proxy | 3 |
The deployment script ***Step 01 Deploy Minikube, Metallb,Cert-manager, Pulsar*** within the
scripts folder performs the following:
1. Deploys a fresh minikube with minikube addons (dashboard, metallb);
2. Configures Metallb loadbalancer
3. Deploys Kubernetes Gateway API CRDs (cert-manager deploy uses)
4. Deploys istio in Ambient mode.
5. Deploys Cert-Manager
6. Deploys Pulsar and all required components via the Pulsar Helm chart into the Cluster
7. Sets pulsar namespace to istio ambient mode which initiates mTLS between pods
8. Tests access from the Pulsar CLI client
9. Allows running of the simple java test program (eclipse, maven) found in the
pulsar-client directory within this project.