Ecosyste.ms: Awesome

An open API service indexing awesome lists of open source software.
Awesome Lists | Featured Topics | Projects
https://github.com/aidanmelen/terraform-kubernetes-confluent-platform

A Terraform module for managing Confluent for Kubernetes (CFK).
https://github.com/aidanmelen/terraform-kubernetes-confluent-platform
cfk confluent confluent-for-kubernetes kafka kubernetes operator terraform terraform-module
Last synced: about 1 month ago
JSON representation
A Terraform module for managing Confluent for Kubernetes (CFK).
Host: GitHub
URL: https://github.com/aidanmelen/terraform-kubernetes-confluent-platform
Owner: aidanmelen
License: apache-2.0
Created: 2022-07-24T03:21:46.000Z (over 2 years ago)
Default Branch: main
Last Pushed: 2022-11-13T22:09:30.000Z (about 2 years ago)
Last Synced: 2024-09-27T19:21:05.992Z (about 2 months ago)
Topics: cfk, confluent, confluent-for-kubernetes, kafka, kubernetes, operator, terraform, terraform-module
Language: Go
Homepage: https://registry.terraform.io/modules/aidanmelen/confluent-platform/kubernetes
Size: 1.33 MB
Stars: 7
Watchers: 3
Forks: 1
Open Issues: 1
Metadata Files:
- Readme: README.md
- License: LICENSE
Awesome Lists containing this project

README

        [![Pre-Commit](https://github.com/aidanmelen/terraform-kubernetes-confluent-platform/actions/workflows/pre-commit.yaml/badge.svg)](https://github.com/aidanmelen/terraform-kubernetes-confluent-platform/actions/workflows/pre-commit.yaml)

[![cookiecutter-tf-module](https://img.shields.io/badge/cookiecutter--tf--module-enabled-brightgreen)](https://github.com/aidanmelen/cookiecutter-tf-module)

[![Terratest](https://img.shields.io/badge/Terratest-enabled-blueviolet)](https://github.com/aidanmelen/terraform-kubernetes-confluent-platform/tree/main/test)

# terraform-kubernetes-confluent-platform

A Terraform module for managing [Confluent for Kubernetes (CFK)](https://docs.confluent.io/operator/current/overview.html).

## Prerequisites

It is recommended that you at least install Custom Resource Definitions (CRDs) on the Kubernetes cluster before applying the Terraform.

Please see [The Confluent for Kubernetes Quickstart](https://docs.confluent.io/operator/current/co-quickstart.html#co-long-quickstart) for more information.

## Override CFK Manifest Values

Similar to the [values file for Helm](https://helm.sh/docs/chart_template_guide/values_files/); The variables supplied to the Confluent Platform module will be [deep-merged](https://github.com/privx-de/terraform-deepmerge) with the default `local` values of the module.

## Example

```hcl

module "confluent_platform" {

  source  = "aidanmelen/confluent-platform/kubernetes"

  version = ">= 0.9.5"

  namespace = var.namespace

  # assumes the confluent operator was deployed in another terraform run

  confluent_operator = {

    create = false

  }

  # uncomment to override the modules default local values

  /*

  zookeeper      = yamldecode(file("${path.module}/values/zookeeper.yaml"))

  kafka          = yamldecode(file("${path.module}/values/kafka.yaml"))

  connect        = yamldecode(file("${path.module}/values/connect.yaml"))

  ksqldb         = yamldecode(file("${path.module}/values/ksqldb.yaml"))

  controlcenter  = yamldecode(file("${path.module}/values/controlcenter.yaml"))

  schemaregistry = yamldecode(file("${path.module}/values/schemaregistry.yaml"))

  kafkarestproxy = yamldecode(file("${path.module}/values/kafkarestproxy.yaml"))

  */

  create_controlcenter = var.create_controlcenter

}

```

## Usage

Release the Confluent Platform with:

```bash

cd examples/complete

terraform init

terraform apply

```

## Troubleshooting

```

Error: Failed to determine GroupVersionResource for manifest

```

This Terraform module uses the [kubernetes_manifest](https://registry.terraform.io/providers/hashicorp/kubernetes/latest/docs/resources/manifest) resource to deploy CFK custom resources. The following excerpt is from [Beta Support for CRDs in the Terraform Provider for Kubernetes](https://www.hashicorp.com/blog/beta-support-for-crds-in-the-terraform-provider-for-kubernetes):

> Custom resource definitions must be applied before custom resources. As above, this is because the provider queries the Kubernetes API for the OpenAPI specification for the resource supplied in the manifest attribute. If the CRD doesn’t exist in the OpenAPI specification during plan time then Terraform can’t use it to create custom resources.

Please see [Troubleshoot Confluent for Kubernetes](https://github.com/confluentinc/confluent-kubernetes-examples/tree/master/troubleshooting) for other troubleshooting needs.

## Tests

Run Terratest using the [Makefile](https://github.com/aidanmelen/terraform-aws-security-group-v2/tree/main/Makefile) targets:

1. `make setup`

2. `make tests`

### Results

```

Terratest Suite (Module v0.9.5) (Terraform v1.2.6)

--- PASS: TestTerraformCompleteExample (175.48s)

--- PASS: TestTerraformConfluentOperatorExample (22.24s)

--- PASS: TestTerraformConfluentPlatformExample (236.10s)

--- PASS: TestTerraformConfluentPlatformSinglenodeExample (232.32s)

--- PASS: TestTerraformConnectorExample (222.19s)

--- PASS: TestTerraformKafkaTopicsExample (168.37s)

--- PASS: TestTerraformSchemaExample (336.87s)

```

## Makefile Targets

```

help                                This help.

build                               Build docker dev container

run                                 Run docker dev container

setup                               Setup project

lint                                Lint with pre-commit

lint-all                            Lint all files with pre-commit

tests                               Tests with Terratest

test-confluent-operator             Test the confluent_operator example

test-confluent-platform             Test the confluent_platform example

test-confluent-platform-singlenode  Test the confluent_platform_singlenode example

test-complete                       Test the complete example

test-kafka-topic                    Test the kafka_topic example

test-schema                         Test the schema example

test-connector                      Test the connector example

release                             Tag remote triggering Terraform Registry release

clean                               Clean project

```

## Requirements

| Name | Version |

|------|---------|

|  [terraform](#requirement\_terraform) | >= 0.14.8 |

|  [helm](#requirement\_helm) | >= 2.0.0 |

|  [kubernetes](#requirement\_kubernetes) | >= 2.12.1 |

## Providers

| Name | Version |

|------|---------|

|  [kubernetes](#provider\_kubernetes) | >= 2.12.1 |

## Modules

| Name | Source | Version |

|------|--------|---------|

|  [confluent\_operator](#module\_confluent\_operator) | ./modules/confluent_operator | n/a |

|  [confluent\_platform\_override\_values](#module\_confluent\_platform\_override\_values) | Invicton-Labs/deepmerge/null | 0.1.5 |

|  [confluent\_role\_bindings](#module\_confluent\_role\_bindings) | ./modules/confluent_role_binding | n/a |

|  [connectors](#module\_connectors) | ./modules/connector | n/a |

|  [kafka\_rest\_classes](#module\_kafka\_rest\_classes) | ./modules/kafka_rest_class | n/a |

|  [kafka\_topics](#module\_kafka\_topics) | ./modules/kafka_topic | n/a |

|  [schemas](#module\_schemas) | ./modules/schema | n/a |

## Resources

| Name | Type |

|------|------|

| [kubernetes_manifest.components](https://registry.terraform.io/providers/hashicorp/kubernetes/latest/docs/resources/manifest) | resource |

| [kubernetes_resource.components](https://registry.terraform.io/providers/hashicorp/kubernetes/latest/docs/data-sources/resource) | data source |

## Inputs

| Name | Description | Type | Default | Required |

|------|-------------|------|---------|:--------:|

|  [confluent\_operator](#input\_confluent\_operator) | Controls if the Confluent Operator resources should be created. This is required when the Confluent Operator is not already running on the kubernetes cluster. | `any` | 
{
  "create": true
} | no |

|  [confluent\_operator\_app\_version](#input\_confluent\_operator\_app\_version) | The default Confluent Operator app version. This may be overriden by component override values. This version must be compatible with the `confluent_platform_version`. Please see confluent docs for more information: https://docs.confluent.io/platform/current/installation/versions-interoperability.html#confluent-operator | `string` | `"2.4.0"` | no |

|  [confluent\_platform\_version](#input\_confluent\_platform\_version) | The default Confluent Platform app version. This may be overriden by component override values. This version must be compatible with the `confluent_operator_app_version`. Please see confluent docs for more information: https://docs.confluent.io/platform/current/installation/versions-interoperability.html#confluent-operator | `string` | `"7.2.0"` | no |

|  [confluent\_role\_bindings](#input\_confluent\_role\_bindings) | A map of Confluent Role Bindings to create. The key is the confluent role binding name. The value map is the input for the `confluent_role_binding` submodule. | `any` | `{}` | no |

|  [connect](#input\_connect) | The Connect override values. | `any` | `{}` | no |

|  [connectors](#input\_connectors) | A map of Connectors to create. The key is the connector name. The value map is the input for the `connector` submodule. | `any` | `{}` | no |

|  [controlcenter](#input\_controlcenter) | The ControlCenter override values. | `any` | `{}` | no |

|  [create](#input\_create) | Controls if the Confluent Platform and Operator resources should be created (affects all resources). | `bool` | `true` | no |

|  [create\_connect](#input\_create\_connect) | Controls if the Connect component of the Confluent Platform should be created. | `bool` | `true` | no |

|  [create\_controlcenter](#input\_create\_controlcenter) | Controls if the ControlCenter component of the Confluent Platform should be created. | `bool` | `true` | no |

|  [create\_kafka](#input\_create\_kafka) | Controls if the Kafka component of the Confluent Platform should be created. | `bool` | `true` | no |

|  [create\_kafkarestproxy](#input\_create\_kafkarestproxy) | Controls if the KafkaRestProxy component of the Confluent Platform should be created. | `bool` | `true` | no |

|  [create\_ksqldb](#input\_create\_ksqldb) | Controls if the KsqlDB component of the Confluent Platform should be created. | `bool` | `true` | no |

|  [create\_schemaregistry](#input\_create\_schemaregistry) | Controls if the SchemaRegistry component of the Confluent Platform should be created. | `bool` | `true` | no |

|  [create\_timeout](#input\_create\_timeout) | The create timeout for each Confluent Platform component. | `string` | `"1h"` | no |

|  [create\_zookeeper](#input\_create\_zookeeper) | Controls if the Zookeeper component of the Confluent Platform should be created. | `bool` | `true` | no |

|  [delete\_timeout](#input\_delete\_timeout) | The delete timeout for each Confluent Platform component. | `string` | `"10m"` | no |

|  [kafka](#input\_kafka) | The Kafka override values. | `any` | `{}` | no |

|  [kafka\_rest\_classes](#input\_kafka\_rest\_classes) | A map of Kafka Rest Classes to create. The key is the kafka rest class name. The value map is the input for the `kafka_rest_class` submodule. | `any` | `{}` | no |

|  [kafka\_topics](#input\_kafka\_topics) | A map of Kafka Topics to create. The key is the topic name. The value map is the input for the `kafka_topic` submodule. | `any` | `{}` | no |

|  [kafkarestproxy](#input\_kafkarestproxy) | The KafkaRestProxy override values. | `any` | `{}` | no |

|  [ksqldb](#input\_ksqldb) | The KsqlDB override values. | `any` | `{}` | no |

|  [namespace](#input\_namespace) | The namespace to release Confluent Platform into. When `confluent_operator` is specified, this will also ensure the Confluent Operator is released into the same namespace. | `string` | `"confluent"` | no |

|  [schemaregistry](#input\_schemaregistry) | The SchemaRegistry override values. | `any` | `{}` | no |

|  [schemas](#input\_schemas) | A map of Schemas to create. The key is the schema name. The value map is the input for the `schema` submodule. | `any` | `{}` | no |

|  [update\_timeout](#input\_update\_timeout) | The update timeout for each Confluent Platform component. | `string` | `"1h"` | no |

|  [zookeeper](#input\_zookeeper) | The Zookeeper override values. | `any` | `{}` | no |

## Outputs

| Name | Description |

|------|-------------|

|  [confluent\_operator](#output\_confluent\_operator) | Map of attributes for the Confluent Operator. |

|  [confluent\_role\_binding\_manifests](#output\_confluent\_role\_binding\_manifests) | Map of attribute maps for all the ConfluentRoleBinding manifests created. |

|  [confluent\_role\_binding\_objects](#output\_confluent\_role\_binding\_objects) | Map of attribute maps for all the ConfluentRoleBinding objects created. |

|  [confluent\_role\_bindings](#output\_confluent\_role\_bindings) | Map of attribute maps for all ConfluentRoleBinding submodules created. |

|  [connect\_manifest](#output\_connect\_manifest) | The Connect manifest. |

|  [connect\_object](#output\_connect\_object) | The Connect object. |

|  [connector\_manifests](#output\_connector\_manifests) | Map of attribute maps for all the Connector manifests created. |

|  [connector\_objects](#output\_connector\_objects) | Map of attribute maps for all the Connector objects created. |

|  [connectors](#output\_connectors) | Map of attribute maps for all Connector submodules created. |

|  [controlcenter\_manifest](#output\_controlcenter\_manifest) | The ControlCenter manifest. |

|  [controlcenter\_object](#output\_controlcenter\_object) | The ControlCenter object. |

|  [kafka\_manifest](#output\_kafka\_manifest) | The Kafka manifest. |

|  [kafka\_object](#output\_kafka\_object) | The Kafka object. |

|  [kafka\_rest\_class\_manifests](#output\_kafka\_rest\_class\_manifests) | Map of attribute maps for all the KafkaRestClass manifests created. |

|  [kafka\_rest\_class\_objects](#output\_kafka\_rest\_class\_objects) | Map of attribute maps for all the KafkaRestClass objects created. |

|  [kafka\_rest\_classes](#output\_kafka\_rest\_classes) | Map of attribute maps for all KafkaRestClass submodules created. |

|  [kafka\_topic\_manifests](#output\_kafka\_topic\_manifests) | Map of attribute maps for all the KafkaTopic manifests created. |

|  [kafka\_topic\_objects](#output\_kafka\_topic\_objects) | Map of attribute maps for all the KafkaTopic objects created. |

|  [kafka\_topics](#output\_kafka\_topics) | Map of attribute maps for all KafkaTopic submodules created. |

|  [kafkarestproxy\_manifest](#output\_kafkarestproxy\_manifest) | The KafkaRestProxy manifest. |

|  [kafkarestproxy\_object](#output\_kafkarestproxy\_object) | The KafkaRestProxy object. |

|  [ksqldb\_manifest](#output\_ksqldb\_manifest) | The KsqlDB manifest. |

|  [ksqldb\_object](#output\_ksqldb\_object) | The KsqlDB object. |

|  [namespace](#output\_namespace) | The default namespace for the Confluent Platform. |

|  [schema\_config\_map](#output\_schema\_config\_map) | Map of attribute maps for all the Schema ConfigMap created. |

|  [schema\_manifests](#output\_schema\_manifests) | Map of attribute maps for all the Schema manifests created. |

|  [schema\_objects](#output\_schema\_objects) | Map of attribute maps for all the Schema objects created. |

|  [schemaregistry\_manifest](#output\_schemaregistry\_manifest) | The SchemaRegistry manifest. |

|  [schemaregistry\_object](#output\_schemaregistry\_object) | The SchemaRegistry object. |

|  [schemas](#output\_schemas) | Map of attribute maps for all Schema submodules created. |

|  [zookeeper\_manifest](#output\_zookeeper\_manifest) | The Zookeeper manifest. |

|  [zookeeper\_object](#output\_zookeeper\_object) | The Zookeeper object. |

## License

Apache 2 Licensed. See [LICENSE](https://github.com/aidanmelen/terraform-kubernetes-confluent-platform/blob/main/LICENSE) for full details.