{"id":19704669,"url":"https://github.com/datastax/kaap","last_synced_at":"2026-02-18T08:01:17.528Z","repository":{"id":176894142,"uuid":"554321551","full_name":"datastax/kaap","owner":"datastax","description":"KAAP, Kubernetes Autoscaling for Apache Pulsar","archived":false,"fork":false,"pushed_at":"2026-02-04T15:30:45.000Z","size":3763,"stargazers_count":55,"open_issues_count":43,"forks_count":25,"subscribers_count":6,"default_branch":"main","last_synced_at":"2026-02-05T03:04:41.422Z","etag":null,"topics":["apache-pulsar","kubernetes","kubernetes-operators","pulsar"],"latest_commit_sha":null,"homepage":"https://datastax.github.io/kaap/","language":"Java","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/datastax.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2022-10-19T16:05:23.000Z","updated_at":"2026-02-04T15:36:36.000Z","dependencies_parsed_at":"2024-01-31T13:53:18.187Z","dependency_job_id":"4f9c6d18-eda0-4006-88a5-8ee6bb859634","html_url":"https://github.com/datastax/kaap","commit_stats":null,"previous_names":["datastax/kaap"],"tags_count":49,"template":false,"template_full_name":null,"purl":"pkg:github/datastax/kaap","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/datastax%2Fkaap","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/datastax%2Fkaap/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/datastax%2Fkaap/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/datastax%2Fkaap/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/datastax","download_url":"https://codeload.github.com/datastax/kaap/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/datastax%2Fkaap/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":29573400,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-02-18T07:57:19.261Z","status":"ssl_error","status_checked_at":"2026-02-18T07:57:18.820Z","response_time":162,"last_error":"SSL_connect returned=1 errno=0 peeraddr=140.82.121.6:443 state=error: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["apache-pulsar","kubernetes","kubernetes-operators","pulsar"],"created_at":"2024-11-11T21:24:02.475Z","updated_at":"2026-02-18T08:01:17.481Z","avatar_url":"https://github.com/datastax.png","language":"Java","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Kubernetes Autoscaling for Apache Pulsar (KAAP)\n\nKubernetes Autoscaling for Apache Pulsar (KAAP) simplifies running [Apache Pulsar](https://pulsar.apache.org) on Kubernetes by applying the familiar [Operator pattern](https://kubernetes.io/docs/concepts/extend-kubernetes/operator/) to Pulsar's components, and horizonally scaling resources up or down based on CPU and memory workloads.\n\nKAAP operator's broker autoscaling integrates with the Pulsar broker's load balancer, which has insight into all other brokers' workloads. With this information, KAAP can make smarter resource management decisions than the Kubernetes HorizontalPodAutoscaler.\n\nKAAP's Bookkeeper autoscaling solution is similarly Pulsar-native. Bookkeeper nodes are scaled up in response to running low on storage, and because of Bookkeeper's segment-based design, the new storage is available immediately for use by the cluster, with no log stream rebalancing required.\n\nWhen KAAP sees low storage usage on a Bookkeeper node, the node is automatically scaled down (decommissioned) to free up volume usage and reduce storage costs. This scale-down is done in a safe, controlled manner which ensures no data loss and guarantees the configured replication factor for all messages. For example, if your replication factor is 3 (write and ack quorum of 3), 3 replicas are maintained at all times during the scale down to ensure data can be recovered, even if there is a failure during the scale-down phase. Scaling down bookies has been a consistent pain point in Pulsar, and KAAP automates this without sacrifing Pulsar's data guarantees.\n\nOperating and maintaining Apache Pulsar clusters traditionally involves complex manual configurations, making it challenging for developers and operators to effectively manage the system's lifecycle. However, with the KAAP operator, these complexities are abstracted away, enabling developers to focus on their applications rather than the underlying infrastructure.\n\nSome of the key features and benefits of the KAAP operator include:\n\n- **Easy Deployment**: Deploying an Apache Pulsar cluster on Kubernetes is simplified through declarative configurations and automation provided by the operator.\n\n- **Scalability**: The KAAP operator enables effortless scaling of Pulsar clusters by automatically handling the creation and configuration of new Pulsar brokers and bookies as per defined rules. The broker autoscaling is integrated with the Pulsar broker load balancer to make smart resource management decisions, and bookkeepers are scaled up and down based on storage usage in a safe, controlled manner.\n\n- **High Availability**: The operator implements best practices for high availability, ensuring that Pulsar clusters are fault-tolerant and can sustain failures without service disruptions.\n\n- **Lifecycle Management**: The operator takes care of common Pulsar cluster lifecycle tasks, such as cluster creation, upgrade, configuration updates, and graceful shutdowns.\n\nWe also offer the KAAP Stack if you're looking for more Kubernetes-native tooling deployed with your Pulsar cluster. Along with the PulsarCluster CRDs, KAAP stack also includes:\n\n* Pulsar Operator\n* Prometheus Stack (Grafana)\n* Pulsar Grafana dashboards\n* Cert Manager\n* Keycloak\n\nThe KAAP Stack is also included in this repository.\n\nWhether you are a developer looking to leverage the power of Apache Pulsar in your Kubernetes environment, or an operator seeking to streamline the management of Pulsar clusters, the KAAP Operator provides a robust and user-friendly solution.\n\nIf you're running [Luna Streaming](https://www.datastax.com/products/luna-streaming), the DataStax distribution of Apache Pulsar, KAAP is 100% compatible with your existing Pulsar cluster.\n\n## Documentation\n\nFull documentation is available in the [DataStax Streaming Documentation](https://docs.datastax.com/en/streaming/kaap-operator/latest/index.html) or at [this repo's GitHub Pages site](https://datastax.github.io/kaap/docs/).\n\n## Install KAAP Operator\n\nThis example installs only the KAAP operator without a managed Pulsar cluster.\nSee the KAAP Stack below to install KAAP with an operator, a Pulsar cluster, and the Prometheus monitoring stack.\n\n1. Install the DataStax KAAP Helm repository:\n```\nhelm repo add kaap https://datastax.github.io/kaap\nhelm repo update\n```\n2. Install the KAAP operator Helm chart:\n```\nhelm install kaap kaap/kaap\n```\nResult:\n```\nNAME: kaap\nLAST DEPLOYED: Wed Dec 17 12:11:51 2025\nNAMESPACE: default\nSTATUS: deployed\nREVISION: 1\nDESCRIPTION: Install complete\nTEST SUITE: None\n```\n3. Ensure the operator is up and running:\n```\nkubectl get deployment\n```\nResult:\n```\nNAME                  READY   UP-TO-DATE   AVAILABLE   AGE\nkaap                  1/1     1            1           52m\n```\n\n4. You've now installed the KAAP operator.\nBy default, when KAAP is installed, the PulsarCluster CRDs are also created.\nThis setting is defined in the KAAP values.yaml file as `crd: {create: true}`.\n\n5. To see available CRDs:\n```\nkubectl get crds | grep kaap\n```\nResult:\n```\nautorecoveries.kaap.oss.datastax.com             2023-06-28T15:37:39Z\nbastions.kaap.oss.datastax.com                   2023-06-28T15:37:39Z\nbookkeepers.kaap.oss.datastax.com                2023-06-28T15:37:39Z\nbrokers.kaap.oss.datastax.com                    2023-06-28T15:37:40Z\nfunctionsworkers.kaap.oss.datastax.com           2023-06-28T15:37:40Z\nproxies.kaap.oss.datastax.com                    2023-06-28T15:37:40Z\npulsarclusters.kaap.oss.datastax.com             2023-06-28T15:37:41Z\nzookeepers.kaap.oss.datastax.com                 2023-06-28T15:37:41Z\n```\n\nFor more information, see the [DataStax Streaming Documentation](https://docs.datastax.com/en/streaming/kaap-operator/latest/index.html) or [this repo's GitHub Pages site](https://datastax.github.io/kaap/docs/).\n\n## Install KAAP Stack with a Pulsar cluster\n\nThis example deploys a KAAP operator, and also deploys a minimally-sized Pulsar cluster for development testing.\n\n\u003e [!IMPORTANT]\n\u003e If using a local Kubernetes environment like [kind](https://kind.sigs.k8s.io/) or [minikube](https://minikube.sigs.k8s.io/docs/) you will need to allocate at least 16 GB of memory to run the\n\u003e example development cluster.\n\n\n1. Install the DataStax KAAP Helm repository:\n```\nhelm repo add kaap https://datastax.github.io/kaap\nhelm repo update\n```\n2. Install the KAAP Stack operator Helm chart with the custom `dev-cluster` values file:\n```\nhelm install kaap-stack kaap/kaap-stack --values helm/examples/dev-cluster/values.yaml\n```\nResult:\n```\nNAME: kaap-stack\nLAST DEPLOYED: Wed Dec 17 12:25:16 2025\nNAMESPACE: default\nSTATUS: deployed\nREVISION: 1\nDESCRIPTION: Install complete\nTEST SUITE: None\n```\n3. Ensure the operator is up and running:\n```\nkubectl get deployment\n```\nResult:\n```\nNAME                                  READY   UP-TO-DATE   AVAILABLE   AGE\nkaap                                  1/1     1            1           5m19s\npulsar-autorecovery                   1/1     1            1           3m19s\npulsar-bastion                        1/1     1            1           3m19s\npulsar-grafana                        1/1     1            1           5m19s\npulsar-kube-prometheus-sta-operator   1/1     1            1           5m19s\npulsar-kube-state-metrics             1/1     1            1           5m19s\npulsar-proxy                          1/1     1            1           3m19s\n```\n\n4. You've now installed KAAP Stack operator with a Pulsar cluster.\nBy default, when KAAP is installed, the PulsarCluster CRDs are also created.\nThis setting is defined in the KAAP values.yaml file as `crd: {create: true}`.\n\n5. To see available CRDs:\n```\nkubectl get crds | grep kaap\n```\nResult:\n```\nautorecoveries.kaap.oss.datastax.com             2023-06-28T15:37:39Z\nbastions.kaap.oss.datastax.com                   2023-06-28T15:37:39Z\nbookkeepers.kaap.oss.datastax.com                2023-06-28T15:37:39Z\nbrokers.kaap.oss.datastax.com                    2023-06-28T15:37:40Z\nfunctionsworkers.kaap.oss.datastax.com           2023-06-28T15:37:40Z\nproxies.kaap.oss.datastax.com                    2023-06-28T15:37:40Z\npulsarclusters.kaap.oss.datastax.com             2023-06-28T15:37:41Z\nzookeepers.kaap.oss.datastax.com                 2023-06-28T15:37:41Z\n```\n\nFor more, see the [DataStax Streaming Documentation](https://docs.datastax.com/en/streaming/kaap-operator/latest/index.html) or [this repo's GitHub Pages site](https://datastax.github.io/kaap/docs/).\n\n## Uninstall KAAP\n\nTo uninstall KAAP:\n```\nhelm uninstall kaap\n```\nResult:\n```\nrelease \"kaap\" uninstalled\n```\n\nFor more, see the [DataStax Streaming Documentation](https://docs.datastax.com/en/streaming/kaap-operator/latest/index.html) or [this repo's GitHub Pages site](https://datastax.github.io/kaap/docs/).\n\n## Resources\nFor more, see the [DataStax Streaming Documentation](https://docs.datastax.com/en/streaming/kaap-operator/latest/index.html) or [this repo's GitHub Pages site](https://datastax.github.io/kaap/docs/).\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdatastax%2Fkaap","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fdatastax%2Fkaap","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdatastax%2Fkaap/lists"}