https://github.com/materializeinc/terraform-google-materialize
Terraform module for deploying Materialize on GCP with all required infrastructure components.
https://github.com/materializeinc/terraform-google-materialize
gcp materialize terraform
Last synced: about 1 month ago
JSON representation
Terraform module for deploying Materialize on GCP with all required infrastructure components.
- Host: GitHub
- URL: https://github.com/materializeinc/terraform-google-materialize
- Owner: MaterializeInc
- License: apache-2.0
- Created: 2024-10-28T14:06:41.000Z (over 1 year ago)
- Default Branch: main
- Last Pushed: 2025-02-27T18:33:40.000Z (over 1 year ago)
- Last Synced: 2025-03-01T11:09:22.519Z (over 1 year ago)
- Topics: gcp, materialize, terraform
- Language: HCL
- Homepage: https://materialize.com/docs/self-managed/v25.1/
- Size: 101 KB
- Stars: 0
- Watchers: 0
- Forks: 1
- Open Issues: 3
-
Metadata Files:
- Readme: README.md
- Contributing: CONTRIBUTING.md
- License: LICENSE
Awesome Lists containing this project
README
# Materialize on Google Cloud Platform
Terraform module for deploying Materialize on Google Cloud Platform (GCP) with all required infrastructure components.
This module sets up:
- GKE cluster for Materialize workloads
- Cloud SQL PostgreSQL instance for metadata storage
- Cloud Storage bucket for persistence
- Required networking and security configurations
- Service accounts with proper IAM permissions
> [!WARNING]
> This module is intended for demonstration/evaluation purposes as well as for serving as a template when building your own production deployment of Materialize.
>
> This module should not be directly relied upon for production deployments: **future releases of the module will contain breaking changes.** Instead, to use as a starting point for your own production deployment, either:
> - Fork this repo and pin to a specific version, or
> - Use the code as a reference when developing your own deployment.
The module has been tested with:
- GKE version 1.28
- PostgreSQL 15
- terraform-helm-materialize v0.1.12 (Materialize Operator v25.1.7)
## `materialize_instances` variable
The `materialize_instances` variable is a list of objects that define the configuration for each Materialize instance.
### `environmentd_extra_args`
Optional list of additional command-line arguments to pass to the `environmentd` container. This can be used to override default system parameters or enable specific features.
```hcl
environmentd_extra_args = [
"--system-parameter-default=max_clusters=1000",
"--system-parameter-default=max_connections=1000",
"--system-parameter-default=max_tables=1000",
]
```
These flags configure default limits for clusters, connections, and tables. You can provide any supported arguments [here](https://materialize.com/docs/sql/alter-system-set/#other-configuration-parameters).
## Requirements
| Name | Version |
|------|---------|
| [terraform](#requirement\_terraform) | >= 1.0 |
| [deepmerge](#requirement\_deepmerge) | ~> 1.0 |
| [google](#requirement\_google) | >= 6.0 |
| [helm](#requirement\_helm) | ~> 2.0 |
| [kubernetes](#requirement\_kubernetes) | ~> 2.0 |
## Providers
No providers.
## Modules
| Name | Source | Version |
|------|--------|---------|
| [certificates](#module\_certificates) | ./modules/certificates | n/a |
| [database](#module\_database) | ./modules/database | n/a |
| [gke](#module\_gke) | ./modules/gke | n/a |
| [load\_balancers](#module\_load\_balancers) | ./modules/load_balancers | n/a |
| [materialize\_nodepool](#module\_materialize\_nodepool) | ./modules/nodepool | n/a |
| [networking](#module\_networking) | ./modules/networking | n/a |
| [operator](#module\_operator) | github.com/MaterializeInc/terraform-helm-materialize | v0.1.66 |
| [storage](#module\_storage) | ./modules/storage | n/a |
## Resources
No resources.
## Inputs
| Name | Description | Type | Default | Required |
|------|-------------|------|---------|:--------:|
| [cert\_manager\_chart\_version](#input\_cert\_manager\_chart\_version) | Version of the cert-manager helm chart to install. | `string` | `"v1.17.1"` | no |
| [cert\_manager\_install\_timeout](#input\_cert\_manager\_install\_timeout) | Timeout for installing the cert-manager helm chart, in seconds. | `number` | `300` | no |
| [cert\_manager\_namespace](#input\_cert\_manager\_namespace) | The name of the namespace in which cert-manager is or will be installed. | `string` | `"cert-manager"` | no |
| [database\_config](#input\_database\_config) | Cloud SQL configuration |
object({
tier = optional(string, "db-custom-2-4096")
version = optional(string, "POSTGRES_15")
password = string
username = optional(string, "materialize")
db_name = optional(string, "materialize")
}) | n/a | yes |
| [database\_vpc\_wait\_duration](#input\_database\_vpc\_wait\_duration) | Duration to wait for VPC resources to be ready before creating the database (e.g., '60s', '2m') | `string` | `"60s"` | no |
| [helm\_chart](#input\_helm\_chart) | Chart name from repository or local path to chart. For local charts, set the path to the chart directory. | `string` | `"materialize-operator"` | no |
| [helm\_values](#input\_helm\_values) | Values to pass to the Helm chart | `any` | `{}` | no |
| [install\_cert\_manager](#input\_install\_cert\_manager) | Whether to install cert-manager. | `bool` | `true` | no |
| [install\_materialize\_operator](#input\_install\_materialize\_operator) | Whether to install the Materialize operator | `bool` | `true` | no |
| [install\_metrics\_server](#input\_install\_metrics\_server) | Whether to install the metrics-server for the Materialize Console. Defaults to false since GKE installs one by default in the kube-system namespace. Only set to true if the GKE cluster was deployed with [monitoring explicitly turned off](https://cloud.google.com/kubernetes-engine/docs/how-to/configure-metrics#:~:text=To%20disable%20system%20metric%20collection,for%20the%20%2D%2Dmonitoring%20flag). Refer to the [GKE docs](https://cloud.google.com/kubernetes-engine/docs/how-to/configure-metrics#:~:text=To%20disable%20system%20metric%20collection,for%20the%20%2D%2Dmonitoring%20flag) for more information, including impact to GKE customer support efforts. | `bool` | `false` | no |
| [labels](#input\_labels) | Labels to apply to all resources | `map(string)` | `{}` | no |
| [materialize\_instances](#input\_materialize\_instances) | Configuration for Materialize instances | list(object({
name = string
namespace = optional(string)
database_name = string
create_database = optional(bool, true)
create_load_balancer = optional(bool, true)
internal_load_balancer = optional(bool, true)
environmentd_version = optional(string)
cpu_request = optional(string, "1")
memory_request = optional(string, "1Gi")
memory_limit = optional(string, "1Gi")
in_place_rollout = optional(bool, false)
request_rollout = optional(string)
force_rollout = optional(string)
balancer_memory_request = optional(string, "256Mi")
balancer_memory_limit = optional(string, "256Mi")
balancer_cpu_request = optional(string, "100m")
license_key = optional(string)
authenticator_kind = optional(string, "None")
external_login_password_mz_system = optional(string)
environmentd_extra_args = optional(list(string), [])
})) | `[]` | no |
| [materialize\_node\_group\_disk\_size\_gb](#input\_materialize\_node\_group\_disk\_size\_gb) | Size of the disk attached to each Materialize worker node | `number` | `100` | no |
| [materialize\_node\_group\_local\_ssd\_count](#input\_materialize\_node\_group\_local\_ssd\_count) | Number of local NVMe SSDs to attach to each Materialize node. In GCP, each disk is 375GB. | `number` | `1` | no |
| [materialize\_node\_group\_machine\_type](#input\_materialize\_node\_group\_machine\_type) | Machine type for Materialize worker nodes | `string` | `"n2-highmem-8"` | no |
| [materialize\_node\_group\_max\_nodes](#input\_materialize\_node\_group\_max\_nodes) | Maximum number of Materialize worker nodes | `number` | `2` | no |
| [materialize\_node\_group\_min\_nodes](#input\_materialize\_node\_group\_min\_nodes) | Minimum number of Materialize worker nodes | `number` | `1` | no |
| [namespace](#input\_namespace) | Kubernetes namespace for Materialize | `string` | `"materialize"` | no |
| [network\_config](#input\_network\_config) | Network configuration for the GKE cluster | object({
subnet_cidr = string
pods_cidr = string
services_cidr = string
}) | n/a | yes |
| [operator\_namespace](#input\_operator\_namespace) | Namespace for the Materialize operator | `string` | `"materialize"` | no |
| [operator\_version](#input\_operator\_version) | Version of the Materialize operator to install | `string` | `null` | no |
| [orchestratord\_version](#input\_orchestratord\_version) | Version of the Materialize orchestrator to install | `string` | `null` | no |
| [prefix](#input\_prefix) | Prefix to be used for resource names | `string` | `"materialize"` | no |
| [project\_id](#input\_project\_id) | The ID of the project where resources will be created | `string` | n/a | yes |
| [region](#input\_region) | The region where resources will be created | `string` | `"us-central1"` | no |
| [storage\_bucket\_version\_ttl](#input\_storage\_bucket\_version\_ttl) | Sets the TTL (in days) on non current storage bucket objects. This must be set if storage\_bucket\_versioning is turned on. | `number` | `7` | no |
| [storage\_bucket\_versioning](#input\_storage\_bucket\_versioning) | Enable bucket versioning. This should be enabled for production deployments. | `bool` | `false` | no |
| [system\_node\_group\_disk\_size\_gb](#input\_system\_node\_group\_disk\_size\_gb) | Size of the disk attached to each system node | `number` | `100` | no |
| [system\_node\_group\_machine\_type](#input\_system\_node\_group\_machine\_type) | Machine type for system nodes | `string` | `"n2-highmem-8"` | no |
| [system\_node\_group\_max\_nodes](#input\_system\_node\_group\_max\_nodes) | Maximum number of system nodes | `number` | `2` | no |
| [system\_node\_group\_min\_nodes](#input\_system\_node\_group\_min\_nodes) | Minimum number of system nodes | `number` | `1` | no |
| [system\_node\_group\_node\_count](#input\_system\_node\_group\_node\_count) | Number of nodes in the system node group | `number` | `1` | no |
| [use\_local\_chart](#input\_use\_local\_chart) | Whether to use a local chart instead of one from a repository | `bool` | `false` | no |
| [use\_self\_signed\_cluster\_issuer](#input\_use\_self\_signed\_cluster\_issuer) | Whether to install and use a self-signed ClusterIssuer for TLS. To work around limitations in Terraform, this will be treated as `false` if no materialize instances are defined. | `bool` | `true` | no |
## Outputs
| Name | Description |
|------|-------------|
| [connection\_strings](#output\_connection\_strings) | Formatted connection strings for Materialize |
| [database](#output\_database) | Cloud SQL instance details |
| [gke\_cluster](#output\_gke\_cluster) | GKE cluster details |
| [load\_balancer\_details](#output\_load\_balancer\_details) | Details of the Materialize instance load balancers. |
| [network](#output\_network) | Network details |
| [operator](#output\_operator) | Materialize operator details |
| [service\_accounts](#output\_service\_accounts) | Service account details |
| [storage](#output\_storage) | GCS bucket details |
## Connecting to Materialize instances
Access to the database is through the balancerd pods on:
* Port 6875 for SQL connections.
* Port 6876 for HTTP(S) connections.
Access to the web console is through the console pods on port 8080.
#### TLS support
TLS support is provided by using `cert-manager` and a self-signed `ClusterIssuer`.
More advanced TLS support using user-provided CAs or per-Materialize `Issuer`s are out of scope for this Terraform module. Please refer to the [cert-manager documentation](https://cert-manager.io/docs/configuration/) for detailed guidance on more advanced usage.
## Upgrade Notes
#### v0.9.0
Environmentd now selects swap nodes by default.
#### v0.8.0
You must upgrade to at least v0.7.x before upgrading to v0.8.x of this terraform code.
Breaking changes:
* The system node group is renamed and significantly modified, forcing a recreation.
* Both node groups are now locked to consistent configurations and ON\\_DEMAND scheduling.
* OpenEBS is removed, and with it all support for lgalloc, our legacy spill to disk mechanism.
#### v0.7.0
This is an intermediate version to handle some changes that must be applied in stages.
It is recommended to upgrade to v0.8.x after upgrading to this version.
Breaking changes:
* Swap is enabled by default.
* Support for lgalloc, our legacy spill to disk mechanism, is deprecated, and will be removed in the next version.
* We now always use two node groups, one for system workloads and one for Materialize workloads.
* Variables for configuring these node groups have been renamed, so they may be configured separately.
To avoid downtime when upgrading to future versions, you must perform a rollout at this version.
1. Ensure your `environmentd_version` is at least `v26.0.0`.
2. Update your `request_rollout` (and `force_rollout` if already at the correct `environmentd_version`).
3. Run `terraform apply`.
You must upgrade to at least v0.6.x before upgrading to v0.7.0 of this terraform code.
It is strongly recommended to have enabled swap on v0.6.x before upgrading to v0.7.0 or higher.
#### v0.6.1
We now have some initial support for swap.
We recommend upgrading to at least v0.5.10 before upgrading to v0.6.x of this terraform code.
To use swap:
1. Set `swap_enabled` to `true`.
2. Ensure your `environmentd_version` is at least `v26.0.0`.
3. Update your `request_rollout` (and `force_rollout` if already at the correct `environmentd_version`).
4. Run `terraform apply`.
This will create a new node group configured for swap, and migrate your clusterd pods there.
#### v0.6.0
This version is missing the updated helm chart.
Skip this version, go to v0.6.1.
#### v0.3.0
We now install `cert-manager` and configure a self-signed `ClusterIssuer` by default.
Due to limitations in Terraform, it cannot plan Kubernetes resources using CRDs that do not exist yet. We have worked around this for new users by only generating the certificate resources when creating Materialize instances that use them, which also cannot be created on the first run.
For existing users upgrading Materialize instances not previously configured for TLS:
1. Leave `install_cert_manager` at its default of `true`.
2. Set `use_self_signed_cluster_issuer` to `false`.
3. Run `terraform apply`. This will install cert-manager and its CRDs.
4. Set `use_self_signed_cluster_issuer` back to `true` (the default).
5. Update the `request_rollout` field of the Materialize instance.
6. Run `terraform apply`. This will generate the certificates and configure your Materialize instance to use them.
#### Storage Bucket Versioning
By default storage bucket versioning is turned off. This both reduces
costs and allows for easier cleanup of resources for testing. When running in
production, versioning should be turned on with a sufficient TTL to meet any
data-recovery requirements. See `storage_bucket_versioning` and `storage_bucket_version_ttl`.