Ecosyste.ms: Awesome
An open API service indexing awesome lists of open source software.
https://github.com/ackeecz/terraform-gcp-elasticsearch
Terraform module for deploying Elasticsearch cluster on GCP
https://github.com/ackeecz/terraform-gcp-elasticsearch
elasticsearch gcp terraform terraform-modules
Last synced: 3 months ago
JSON representation
Terraform module for deploying Elasticsearch cluster on GCP
- Host: GitHub
- URL: https://github.com/ackeecz/terraform-gcp-elasticsearch
- Owner: AckeeCZ
- License: mit
- Created: 2020-04-29T12:52:50.000Z (almost 5 years ago)
- Default Branch: master
- Last Pushed: 2023-03-16T15:51:15.000Z (almost 2 years ago)
- Last Synced: 2024-04-15T00:06:29.180Z (10 months ago)
- Topics: elasticsearch, gcp, terraform, terraform-modules
- Language: HCL
- Homepage:
- Size: 175 KB
- Stars: 10
- Watchers: 4
- Forks: 7
- Open Issues: 2
-
Metadata Files:
- Readme: README.md
- Changelog: CHANGELOG.md
- License: LICENSE
Awesome Lists containing this project
README
# Ackee GCE Elasticsearch Terraform module
This module is primary written for provisioning of GCE instance from our ES image (https://github.com/AckeeCZ/packer-elasticsearch)
It does a few things :
* Generates GCP IAM Service Account with Storage Admin role for backups and insert it into ES keystore
* Downloads RAW disk from GCS and create an image from it. (you can generate your own image with Packer using https://github.com/AckeeCZ/packer-elasticsearch)
* Create SSH key for instance provisioning
* Create (GCP) firewall rules so GKE "gateway" pods can reach GCE cluster## Usage
```hcl
module "elasticsearch_prod" {
source = "AckeeCZ/elasticsearch/gcp"project = "my-gcp-project"
region = "europe-west3"
zone = "europe-west3-c"
instance_name = "elasticsearch-prod"
cluster_name = "elasticsearch"
cluster_ipv4_cidr = "10.128.0.0/14"
node_count = "3"
heap_size = "1500m"
raw_image_source = "https://storage.googleapis.com/ackee-images/ackee-elasticsearch-7-disk-79.tar.gz"
data_disk_size = "10"
custom_pre_start_commands = "/usr/share/elasticsearch/bin/elasticsearch-plugin install -b analysis-stempel"
}
```## Running plugins needed for index startup
If you need some plugins that is required for index startup (e.g., [analysis-stempel](https://www.elastic.co/guide/en/elasticsearch/plugins/current/analysis-stempel.html)),
you must install them utilizing `custom_pre_start_commands` variable, which is called before running `systemctl start elasticsearch`.See Usage part above for example with `analysis-stempel` plugin installation
## Upgrading to v12
To keep ingesting Elasticsearch logs with the new Google Cloud Ops agent, please upgrade your ES instances to image generated from `v1.0.0` of the [packer-elasticsearch](https://github.com/AckeeCZ/packer-elasticsearch)
AFTER upgrading this module to `v12.0.0`## Before you do anything in this module
Install pre-commit hooks by running following commands:
```shell script
brew install pre-commit
pre-commit install
```## Requirements
| Name | Version |
|------|---------|
| [terraform](#requirement\_terraform) | >= 1.0 |## Providers
| Name | Version |
|------|---------|
| [google](#provider\_google) | n/a |
| [google-beta](#provider\_google-beta) | n/a |
| [kubernetes](#provider\_kubernetes) | n/a |
| [random](#provider\_random) | n/a |
| [tls](#provider\_tls) | n/a |## Modules
No modules.
## Resources
| Name | Type |
|------|------|
| [google-beta_google_compute_forwarding_rule.elasticsearch](https://registry.terraform.io/providers/hashicorp/google-beta/latest/docs/resources/google_compute_forwarding_rule) | resource |
| [google-beta_google_compute_health_check.elasticsearch](https://registry.terraform.io/providers/hashicorp/google-beta/latest/docs/resources/google_compute_health_check) | resource |
| [google-beta_google_compute_instance_group.elasticsearch](https://registry.terraform.io/providers/hashicorp/google-beta/latest/docs/resources/google_compute_instance_group) | resource |
| [google-beta_google_compute_region_backend_service.elasticsearch](https://registry.terraform.io/providers/hashicorp/google-beta/latest/docs/resources/google_compute_region_backend_service) | resource |
| [google-beta_google_compute_subnetwork.proxy](https://registry.terraform.io/providers/hashicorp/google-beta/latest/docs/resources/google_compute_subnetwork) | resource |
| [google_compute_disk.data](https://registry.terraform.io/providers/hashicorp/google/latest/docs/resources/compute_disk) | resource |
| [google_compute_firewall.elasticsearch_allow_external_subnets](https://registry.terraform.io/providers/hashicorp/google/latest/docs/resources/compute_firewall) | resource |
| [google_compute_firewall.elasticsearch_allow_external_tags](https://registry.terraform.io/providers/hashicorp/google/latest/docs/resources/compute_firewall) | resource |
| [google_compute_firewall.elasticsearch_allow_healthchecks](https://registry.terraform.io/providers/hashicorp/google/latest/docs/resources/compute_firewall) | resource |
| [google_compute_firewall.elasticsearch_allow_ilb_traffic](https://registry.terraform.io/providers/hashicorp/google/latest/docs/resources/compute_firewall) | resource |
| [google_compute_image.elasticsearch](https://registry.terraform.io/providers/hashicorp/google/latest/docs/resources/compute_image) | resource |
| [google_compute_instance.elasticsearch](https://registry.terraform.io/providers/hashicorp/google/latest/docs/resources/compute_instance) | resource |
| [google_project_iam_member.elasticsearch_backup_role](https://registry.terraform.io/providers/hashicorp/google/latest/docs/resources/project_iam_member) | resource |
| [google_service_account.elasticsearch_backup](https://registry.terraform.io/providers/hashicorp/google/latest/docs/resources/service_account) | resource |
| [google_service_account_key.elasticsearch_backup](https://registry.terraform.io/providers/hashicorp/google/latest/docs/resources/service_account_key) | resource |
| [google_storage_bucket.backup_repository](https://registry.terraform.io/providers/hashicorp/google/latest/docs/resources/storage_bucket) | resource |
| [kubernetes_cron_job_v1.backup_cleanup](https://registry.terraform.io/providers/hashicorp/kubernetes/latest/docs/resources/cron_job_v1) | resource |
| [kubernetes_stateful_set.elasticsearch](https://registry.terraform.io/providers/hashicorp/kubernetes/latest/docs/resources/stateful_set) | resource |
| [random_string.es_name_suffix](https://registry.terraform.io/providers/hashicorp/random/latest/docs/resources/string) | resource |
| [tls_private_key.provision](https://registry.terraform.io/providers/hashicorp/tls/latest/docs/resources/private_key) | resource |
| [google_compute_network.default](https://registry.terraform.io/providers/hashicorp/google/latest/docs/data-sources/compute_network) | data source |
| [google_compute_zones.available](https://registry.terraform.io/providers/hashicorp/google/latest/docs/data-sources/compute_zones) | data source |## Inputs
| Name | Description | Type | Default | Required |
|------|-------------|------|---------|:--------:|
| [add\_random\_suffix](#input\_add\_random\_suffix) | Add random suffix to all resources with possible duplicates if the same module is deployed multiple times | `bool` | `false` | no |
| [allowed\_ipv4\_subnets](#input\_allowed\_ipv4\_subnets) | IPv4 subnets allowed to communicate with ES instances. | `list(string)` | `[]` | no |
| [allowed\_tags](#input\_allowed\_tags) | Network tags allowed to communicate with ES instances. | `list(string)` | `[]` | no |
| [backend\_service\_timeout\_sec](#input\_backend\_service\_timeout\_sec) | How many seconds to wait for the backend before considering it a failed request. Default is 30 seconds. Valid range is [1, 86400]. | `number` | `30` | no |
| [backup\_failed\_jobs\_history\_limit](#input\_backup\_failed\_jobs\_history\_limit) | Set retention for failed jobs history | `number` | `5` | no |
| [backup\_repository\_create](#input\_backup\_repository\_create) | If GCS bucket should be created. Set to false, if you previously created bucket | `bool` | `true` | no |
| [backup\_repository\_name](#input\_backup\_repository\_name) | Custom name of Elasticsearch backup repository, same name is going to be used for backup bucket | `string` | `""` | no |
| [backup\_schedule](#input\_backup\_schedule) | Backup schedule in cron format | `string` | `"0 3 * * *"` | no |
| [backup\_storage\_class](#input\_backup\_storage\_class) | The storage class you set for an object affects the object's availability and pricing model | `string` | `"STANDARD"` | no |
| [backup\_successful\_jobs\_history\_limit](#input\_backup\_successful\_jobs\_history\_limit) | Set retention for successful jobs history | `number` | `3` | no |
| [cluster\_ca\_certificate](#input\_cluster\_ca\_certificate) | Public CA certificate that is the root of trust for the GKE K8s cluster | `string` | n/a | yes |
| [cluster\_endpoint](#input\_cluster\_endpoint) | Cluster control plane endpoint | `string` | n/a | yes |
| [cluster\_name](#input\_cluster\_name) | ES cluster name. | `string` | n/a | yes |
| [cluster\_token](#input\_cluster\_token) | Cluster master token, keep always secret! | `string` | n/a | yes |
| [custom\_init\_commands](#input\_custom\_init\_commands) | Any custom commands which should be run after bootstrapping the Elasticsearch cluster after starting Elasticsearch service | `string` | `""` | no |
| [custom\_pre\_start\_commands](#input\_custom\_pre\_start\_commands) | Any custom commands which should be run after bootstrapping the Elasticsearch cluster before starting Elasticsearch service | `string` | `""` | no |
| [data\_disk\_size](#input\_data\_disk\_size) | Persistent disk size specified in GB. | `string` | n/a | yes |
| [data\_disk\_type](#input\_data\_disk\_type) | Type of disk used as a persistent storage. | `string` | `"pd-ssd"` | no |
| [enable\_health\_check\_logging](#input\_enable\_health\_check\_logging) | Enable health check logging | `bool` | `false` | no |
| [es\_image\_creation\_timeout](#input\_es\_image\_creation\_timeout) | Timeout for creating ES image | `string` | `"10m"` | no |
| [es\_name\_suffix\_length](#input\_es\_name\_suffix\_length) | Length of random generated suffix for ES name | `number` | `8` | no |
| [health\_check\_healthy\_threshold](#input\_health\_check\_healthy\_threshold) | How many consecutive successes must occur to mark a VM instance healthy. | `number` | `2` | no |
| [health\_check\_interval\_sec](#input\_health\_check\_interval\_sec) | How often (in seconds) to send a health check. The default value is 5 seconds. | `number` | `5` | no |
| [health\_check\_timeout\_sec](#input\_health\_check\_timeout\_sec) | How long (in seconds) to wait before claiming failure. The default value is 5 seconds. It is invalid for timeoutSec to have greater value than checkIntervalSec. | `number` | `5` | no |
| [health\_check\_unhealthy\_threshold](#input\_health\_check\_unhealthy\_threshold) | How many consecutive failures must occur to mark a VM instance unhealthy. | `number` | `2` | no |
| [heap\_size](#input\_heap\_size) | Heap size setting for ES. | `string` | `"1800m"` | no |
| [instance\_name](#input\_instance\_name) | Base for GCE instances name. Must be unique within GCP project | `string` | n/a | yes |
| [load\_balancer\_subnetwork](#input\_load\_balancer\_subnetwork) | The subnetwork that the load balanced IP should belong to for this Forwarding Rule. If the network specified is in auto subnet mode, this field is optional. However, if the network is in custom subnet mode, a subnetwork must be specified. | `string` | `"10.64.0.0/26"` | no |
| [machine\_type](#input\_machine\_type) | The machine type to create | `string` | `"n1-standard-1"` | no |
| [namespace](#input\_namespace) | K8s namespace used to deploy endpoints and services. | `string` | `"production"` | no |
| [network](#input\_network) | GCE VPC used for compute instances | `string` | `"default"` | no |
| [node\_count](#input\_node\_count) | Number of ES nodes to deploy. | `number` | `1` | no |
| [node\_roles](#input\_node\_roles) | Map of node roles. Index is instance number, starting at zero. If not defined, node has all roles | `map(any)` | `{}` | no |
| [project](#input\_project) | Name of GCP project. | `string` | n/a | yes |
| [raw\_image\_source](#input\_raw\_image\_source) | URL of tar archive containing RAW source for ES image (you can use Packer image template to generate image, as mentioned above). | `string` | `"https://storage.googleapis.com/ackee-images/ackee-elasticsearch-7-disk-79.tar.gz"` | no |
| [region](#input\_region) | Region of GCP project. | `string` | n/a | yes |
| [root\_disk\_size](#input\_root\_disk\_size) | Persistent disk size specified in GB. | `string` | `"10"` | no |
| [zone](#input\_zone) | Zone of GCP project - optional parameter, if not set, the instances will be spread across the available zones. | `string` | `null` | no |## Outputs
| Name | Description |
|------|-------------|
| [ilb\_dns](#output\_ilb\_dns) | DNS name follows GCP internal rule SERVICE\_LABEL.FORWARDING\_RULE\_NAME.il4.REGION.lb.PROJECT\_ID.internal |
| [ip\_address](#output\_ip\_address) | The internal IP assigned to the regional forwarding rule. |