{"id":26018773,"url":"https://github.com/t-snyder/learn-07-pulsar","last_synced_at":"2025-03-06T06:34:02.915Z","repository":{"id":280388448,"uuid":"940554074","full_name":"t-snyder/learn-07-pulsar","owner":"t-snyder","description":"Pulsar minikube deployment with cert-manager, istio ambient mode, tls connections","archived":false,"fork":false,"pushed_at":"2025-03-03T05:46:13.000Z","size":126,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-03-03T06:28:46.958Z","etag":null,"topics":["cert-manager","istio-ambient-mode","pulsar","pulsar-helm-deployment","pulsar-kube-deployment","tls"],"latest_commit_sha":null,"homepage":"","language":"Shell","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/t-snyder.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2025-02-28T11:33:48.000Z","updated_at":"2025-03-03T06:11:36.000Z","dependencies_parsed_at":"2025-03-03T06:38:55.371Z","dependency_job_id":null,"html_url":"https://github.com/t-snyder/learn-07-pulsar","commit_stats":null,"previous_names":["t-snyder/learn-07-pulsar"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/t-snyder%2Flearn-07-pulsar","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/t-snyder%2Flearn-07-pulsar/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/t-snyder%2Flearn-07-pulsar/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/t-snyder%2Flearn-07-pulsar/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/t-snyder","download_url":"https://codeload.github.com/t-snyder/learn-07-pulsar/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":242163293,"owners_count":20082209,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["cert-manager","istio-ambient-mode","pulsar","pulsar-helm-deployment","pulsar-kube-deployment","tls"],"created_at":"2025-03-06T06:34:02.269Z","updated_at":"2025-03-06T06:34:02.905Z","avatar_url":"https://github.com/t-snyder.png","language":"Shell","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Apache Pulsar External Connection Prototypes\n\n## Purpose\nThe purpose of this set of prototypes is to provide simple external connection prototypes\nfor both pulsar cli and a java pulsar client. The prototypes first explore unencrypted\nconnections within the Proto-01-kube-basic, and then encrypted connections via tls with\ncert-manager providing the issuers, certificates and secrets. The final 3rd prototype uses\nthe helm chart for tls deployment.\n\n## Original Unmet Goal\nThe original purpose of these learning prototypes was to explore Kubernetes Gateway API\nwith Istio and Cert-Manager for external TLS connections. However, after several failed\nattempts I determined that Pulsar external connections rely on the Pulsar proxy for \nproviding clients the correct broker. Without a deep dive into the Pulsar Proxy\ncode to figure out how it was handling this, and to then come up with a way for the \nGateway API to work within these boundaries I did not see a way forward. As the Proxy is\nworking I will leave the rest for another time when I have more time and a definite need.\n\n\n## Project Dependencies\n| Core Infrastructure | Version         |\n| --------------- | --------------- |\n| Minikube        | 1.34.0          |\n| Kubernetes      | 1.31.0          |\n| Docker          | 27.2.0          |\n\n## Computer Configuration:\n\n| Name            | Description                             |\n| --------------- | --------------------------------------- |\n| Ubuntu          | 20.04.6 LTS                             |\n| Processor       | Intel® Core™ i7-7700K CPU @ 4.20GHz × 8 |\n| Memory          | 64 GB                                   |\n\n## Deploying the Core Infrastructure Dependencies\nInstructions for deploying the Core Infrastructure Dependencies listed above are NOT included within this set\nof Prototypes as there are numerous targeted deployment instructions for each better suited for your\nparticular OS.\n\n## Dependencies Deployed within the Prototype Scripts ( As Required )\n\n| Deployed Name          | Version |\n| ---------------------- | ------- |\n| Cert-manager           | 1.15.5  |\n| Istio                  | 1.23.2  |\n| Kubernetes Gateway API | 1.2.0   |\n| Metallb                | 0.9.6   |\n\n## Dependency Documentation Referenced\n  1. Cert-Manager           - https://cert-manager.io/docs/\n  2. Istio Ambient Mode     - https://istio.io/latest/docs/ambient/\n  3. Kubernetes Gateway API - https://kubernetes.io/docs/concepts/services-networking/gateway/\n  4. Apache Pulsar          - https://pulsar.apache.org/docs/4.0.x\n  \n## Kubernetes deployment component generation\nThe kube based deployments were originally generated by reviewing the output of the helm\nchart for minikube and tls. The following is the prime example.\n\n***helm install --dry-run --values ${PROTODIR}/helm/values-\u003cProto #\u003e.yaml --namespace pulsar pulsar-mini apache/pulsar \u003e output.txt***\n\nIt should be noted that minikube has to be running in order to run this dry-run command.\n\n   \n## Prototype Script Functionality\n**Note :**\u003cbr\u003e\n*The commands within the shell files below are meant to be copy pasted (one or a few lines at a time) into a terminal, and not run as an automated bash script.*\n\n\n## The Prototypes\nThe Pulsar components deployed within each of the following Prototypes consist of only the\nrequired Pulsar components as follows:\n   1. Zookeeper\n   2. Bookkeeper (Bookie)\n   3. Toolset    - for Pulsar CLI client\n   4. Broker\n   5. Proxy\n\nAdditional components which can be deployed within the Helm chart are excluded from this \ndeployment. These include Prometheus and Graphana as well as helpers such as PodMonitors.\n\n   \n### Proto-01-kube-basic\nThe purpose for this prototype is to provide unencrypted external access to the Pulsar\ndeployment within minikube. The prototype kube yaml components were generated by running a dry-run from the Pulsar\nminikube Helm chart as follows:\n\n***helm install --dry-run --values ${PROTODIR}/helm/values-01.yaml --namespace pulsar pulsar-mini apache/pulsar \u003e kube-pulsar.txt***\n\nFrom this output the kube deployment components were obtained. This script does a minimal\ninstallation of only the main required Pulsar components. The pods deployed as listed below.\n\n| Component     | # Deployed |\n| ------------- | ---------- |\n|    Zookeeper  |     1      |\n|    Bookie     |     1      |\n|    Toolset    |     1      |\n|    Broker     |     1      |\n|    Proxy      |     3      |\n\nThe deployment script ***Step 01 Deploy Minikube, metallb, Pulsar*** within the scripts\nfolder performs the following:\n   1. Deploys a fresh minikube with minikube addons (dashboard, metallb);\n   2. Configures Metallb loadbalancer\n   3. deploys Pulsar and all its components into the Cluster\n   4. Tests access from the Pulsar CLI client\n   5. Allows running of the simple java test program (eclipse, maven) found in the\n      pulsar-client directory within this project.\n\n\n### Proto-02-kube-basic-tls\nThe main purpose of this prototype is to provide tls encrypted external access to the Pulsar\ndeployment within minikube. The prototype kube yaml components were generated by running a dry-run from the Pulsar\nminikube Helm chart as follows:\n\n***helm install --dry-run --values ${PROTODIR}/helm/values-02.yaml --namespace pulsar pulsar-mini apache/pulsar \u003e kube-pulsar.txt***\n\nFrom this output the kube deployment components were obtained. This script does a minimal\ninstallation of only the main required Pulsar components. The pods deployed as listed below.\n\n| Component     | # Deployed |\n| ------------- | ---------- |\n|    Zookeeper  |     3      |\n|    Bookie     |     4      |\n|    Toolset    |     1      |\n|    Broker     |     3      |\n|    Proxy      |     3      |\n\nThe deployment script ***Step 01 Deploy Minikube, Metallb,Cert-manager, Pulsar*** within the \nscripts folder performs the following:\n   1. Deploys a fresh minikube with minikube addons (dashboard, metallb);\n   2. Configures Metallb loadbalancer\n   3. Deploys Kubernetes Gateway API CRDs (cert-manager deploy uses)\n   4. Deploys istio in Ambient mode.\n   5. Deploys Cert-Manager\n   6. Deploys Pulsar and all required components into the Cluster\n   7. Sets pulsar namespace to istio ambient mode which initiates mTLS between pods\n   8. Tests access from the Pulsar CLI client\n   9. Allows running of the simple java test program (eclipse, maven) found in the\n      pulsar-client directory within this project.\n\n\n### Proto-03-helm-basic-tls\nThe main purpose of this prototype is to provide tls encrypted external access to the Pulsar\ndeployment within minikube. The deployment uses the Pulsar Helm chart to deploy the following\nPulsar components.\n\n| Component     | # Deployed |\n| ------------- | ---------- |\n|    Zookeeper  |     3      |\n|    Bookie     |     4      |\n|    Toolset    |     1      |\n|    Broker     |     3      |\n|    Proxy      |     3      |\n\nThe deployment script ***Step 01 Deploy Minikube, Metallb,Cert-manager, Pulsar*** within the \nscripts folder performs the following:\n   1. Deploys a fresh minikube with minikube addons (dashboard, metallb);\n   2. Configures Metallb loadbalancer\n   3. Deploys Kubernetes Gateway API CRDs (cert-manager deploy uses)\n   4. Deploys istio in Ambient mode.\n   5. Deploys Cert-Manager\n   6. Deploys Pulsar and all required components via the Pulsar Helm chart into the Cluster\n   7. Sets pulsar namespace to istio ambient mode which initiates mTLS between pods\n   8. Tests access from the Pulsar CLI client\n   9. Allows running of the simple java test program (eclipse, maven) found in the\n      pulsar-client directory within this project.\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ft-snyder%2Flearn-07-pulsar","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Ft-snyder%2Flearn-07-pulsar","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ft-snyder%2Flearn-07-pulsar/lists"}