https://github.com/strimzi/kafka-kubernetes-config-provider
Kubernetes Configuration Provider for Apache Kafka®
https://github.com/strimzi/kafka-kubernetes-config-provider
Last synced: 20 days ago
JSON representation
Kubernetes Configuration Provider for Apache Kafka®
- Host: GitHub
- URL: https://github.com/strimzi/kafka-kubernetes-config-provider
- Owner: strimzi
- License: apache-2.0
- Created: 2021-06-03T11:56:29.000Z (almost 4 years ago)
- Default Branch: main
- Last Pushed: 2024-12-17T11:40:38.000Z (4 months ago)
- Last Synced: 2025-03-30T21:07:15.285Z (28 days ago)
- Language: Java
- Homepage:
- Size: 69.3 KB
- Stars: 26
- Watchers: 12
- Forks: 12
- Open Issues: 1
-
Metadata Files:
- Readme: README.md
- License: LICENSE
- Code of conduct: CODE_OF_CONDUCT.md
- Governance: GOVERNANCE.md
Awesome Lists containing this project
README
[](https://dev.azure.com/cncf/strimzi/_build/latest?definitionId=32&branchName=main)
[](https://github.com/strimzi/kafka-kubernetes-config-provider/releases/latest)
[](https://maven-badges.herokuapp.com/maven-central/io.strimzi/kafka-kubernetes-config-provider)
[](http://www.apache.org/licenses/LICENSE-2.0)
[](https://twitter.com/strimziio)
# Kubernetes Configuration Provider for Apache Kafka®
[Apache Kafka®](https://kafka.apache.org) supports pluggable configuration providers which can load configuration data from external sources.
The configuration providers in this repo can be used to load data from Kubernetes Secrets and Config Maps.
It can be used in all Kafka components and does not depend on the other Strimzi components.
So you could, for example, use it with your producer or consumer applications even if you don't use the Strimzi operators to provide your Kafka cluster.
One of the example use-cases is to load certificates or JAAS configuration from Kubernetes Secrets.
## Using it with Strimzi
From Strimzi Kafka Operators release 0.24.0, the Kubernetes Configuration Provider is included in all the Kafka deployments.
You can use it for example with Kafka Connect, Kafka Mirror Maker 1 or 2 and with Kafka Connect connectors.
The following example shows how to use it with Kafka Connect and Connectors:
1) Deploy Kafka Connect and enable the Kubernetes Configuration Provider:
```yaml
apiVersion: kafka.strimzi.io/v1beta2
kind: KafkaConnect
metadata:
name: my-connect
annotations:
strimzi.io/use-connector-resources: "true"
spec:
# ...
config:
# ...
config.providers: secrets,configmaps
config.providers.secrets.class: io.strimzi.kafka.KubernetesSecretConfigProvider
config.providers.configmaps.class: io.strimzi.kafka.KubernetesConfigMapConfigProvider
# ...
```
2) Create a configuration Config Map
```yaml
apiVersion: v1
kind: ConfigMap
metadata:
name: connector-configuration
data:
option1: value1
option2: value2
```
3) Create the Role and RoleBinding:
```yaml
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
name: connector-configuration-role
rules:
- apiGroups: [""]
resources: ["configmaps"]
resourceNames: ["connector-configuration"]
verbs: ["get"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
name: connector-configuration-role-binding
subjects:
- kind: ServiceAccount
name: my-connect-connect
namespace: myproject
roleRef:
kind: Role
name: connector-configuration-role
apiGroup: rbac.authorization.k8s.io
```
Use the Service Account already used by your Kafka Connect deployment, which is named `-connect` where `` is the name of your KafkaConnect custom resource.
4) Create the connector:
```yaml
apiVersion: kafka.strimzi.io/v1beta2
kind: KafkaConnector
metadata:
name: my-connector
labels:
strimzi.io/cluster: my-connect
spec:
# ...
config:
option: ${configmaps:myproject/connector-configuration:option1}
# ...
```
## Adding the Kubernetes Configuration Provider to Apache Kafka clients
You can add Kubernetes Configuration Provider as any other Java dependency using Maven or any other build tool.
For example:
```xml
io.strimzi
kafka-kubernetes-config-provider
0.1.0
```
## Adding the Kubernetes Configuration Provider to Apache Kafka server components
You can also use the Kubernetes Configuration Provider with your own Apache Kafka deployments not managed by Strimzi.
To add the Kubernetes Configuration Provider to your own Apache Kafka server distribution, you can download the ZIP or TAR.GZ files frm the GitHub release page and unpack it into Kafka's `libs` directory.
## Using the configuration provider
First, you need to initializer the configuration provider.
For use with Kubernetes Secrets, use:
```properties
config.providers=secrets
config.providers.secrets.class=io.strimzi.kafka.KubernetesSecretConfigProvider
```
For use with Kubernetes Config Maps, use:
```properties
config.providers=configmaps
config.providers.configmaps.class=io.strimzi.kafka.KubernetesConfigMapConfigProvider
```
You can also use both together:
```properties
config.providers=secrets,configmaps
config.providers.secrets.class=io.strimzi.kafka.KubernetesSecretConfigProvider
config.providers.configmaps.class=io.strimzi.kafka.KubernetesConfigMapConfigProvider
```
Once you initialize it, you can use it to load data from Kubernetes.
The config provider configuration is in the form `/:`.
Where:
* The `` is the namespace where the Secret or Config Map exists
* The `` is the name of the Secret or Config Map
* The `` is the key under which the configuration value is stored in the Secret or Config Map
For example:
```properties
option=${secrets:my-namespace/my-secret:my-key}
```
Alternatively, you can also use the form `:`.
Where:
* The `` is the name of the Secret or Config Map
* The `` is the key under which the configuration value is stored in the Secret or Config Map
In this case, the current namespace of the Kubernetes client will be used.
The following example shows how to use it in a Kafka Consumer consuming from Apache Kafka cluster on Kubernetes using Strimzi:
1) Deploy your Kafka cluster with TLS authentication enabled
2) Create KafkaUser resource using TLS authentication
3) In your consumer properties, use the following options to load the public key of the CA used by the Apache Kafka cluster and the user certificates for authentication
```properties
config.providers=secrets,configmaps
config.providers.secrets.class=io.strimzi.kafka.KubernetesSecretConfigProvider
config.providers.configmaps.class=io.strimzi.kafka.KubernetesConfigMapConfigProvider
security.protocol=SSL
ssl.keystore.type=PEM
ssl.keystore.certificate.chain=${secrets:myproject/my-user:user.crt}
ssl.keystore.key=${secrets:myproject/my-user:user.key}
ssl.truststore.type=PEM
ssl.truststore.certificates=${secrets:myproject/my-cluster-cluster-ca-cert:ca.crt}
```
## Using patterns
You can also use [Glob patterns](https://en.wikipedia.org/wiki/Glob_(programming)) to specify the keys in the Secret or ConfigMap that should be used.
The Kubernetes Config Provider will find all fields matching the pattern and join them together into a single value.
This is useful for example when you want to load multiple certificates from a Secret into a single configuration option.
With an example Kubernetes Secret container multiple certificate files:
```yaml
apiVersion: v1
kind: Secret
metadata:
name: certificates
data:
ca.crt: ...
ca2.crt: ...
ca3.crt: ...
```
You can use the provider to load all of the certificates by using the key pattern `*.crt`:
```properties
config.providers=secrets
config.providers.secrets.class=io.strimzi.kafka.KubernetesSecretConfigProvider
...
ssl.truststore.certificates=${secrets:myproject/certificates:*.crt}
```
By default, a new line will be used as a separator when joining the different values.
But you can configure the separator if you want to use a different one:
```properties
config.providers=secrets,configmaps
config.providers.secrets.class=io.strimzi.kafka.KubernetesSecretConfigProvider
config.providers.secrets.param.separator=,
```
## Configuring the Kubernetes client
The Kubernetes Config Provider is using the [Fabric8 Kubernetes Client](https://github.com/fabric8io/kubernetes-client).
You can configure it using Java system properties, environment variables, Kube config files or ServiceAccount tokens.
All available configuration options are described in the [client documentation](https://github.com/fabric8io/kubernetes-client#configuring-the-client).
By default, it will try to automatically find the available configuration - for example from the Kube config file (`~/.kube/config`) or from the ServiceAccount if running inside Kubernetes Pod.
### RBAC rights
The Kubernetes account used by the Kubernetes Configuration Provider needs to have access to the Config Maps or Secrets.
The only RBAC rights it needs is the `get` access rights on a given resource.
For example:
```yaml
- apiGroups: [""]
resources: ["secrets"]
resourceNames: ["my-user", "my-cluster-cluster-ca-cert"]
verbs: ["get"]
```
It does not need any other access rights.