{"id":15291736,"url":"https://github.com/azure/helm-elasticstack","last_synced_at":"2025-10-07T04:31:37.813Z","repository":{"id":55889796,"uuid":"125362842","full_name":"Azure/helm-elasticstack","owner":"Azure","description":"Kubernetes Helm Charts and Tools to run Elastic Stack(ELK) on Azure Container Service(AKS)","archived":true,"fork":false,"pushed_at":"2020-12-09T08:30:14.000Z","size":212,"stargazers_count":73,"open_issues_count":0,"forks_count":40,"subscribers_count":12,"default_branch":"master","last_synced_at":"2024-12-22T18:05:06.520Z","etag":null,"topics":["acs","aks","aks-kubernetes-cluster","azure-container-service","elasticsearch","elasticsearch-cluster","helm","helm-charts","kibana","kibana-cluster","kubernetes","logstash"],"latest_commit_sha":null,"homepage":"","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/Azure.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2018-03-15T12:18:26.000Z","updated_at":"2024-10-06T20:41:35.000Z","dependencies_parsed_at":"2022-08-15T08:40:52.111Z","dependency_job_id":null,"html_url":"https://github.com/Azure/helm-elasticstack","commit_stats":null,"previous_names":[],"tags_count":2,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Azure%2Fhelm-elasticstack","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Azure%2Fhelm-elasticstack/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Azure%2Fhelm-elasticstack/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Azure%2Fhelm-elasticstack/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/Azure","download_url":"https://codeload.github.com/Azure/helm-elasticstack/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":235586525,"owners_count":19014035,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["acs","aks","aks-kubernetes-cluster","azure-container-service","elasticsearch","elasticsearch-cluster","helm","helm-charts","kibana","kibana-cluster","kubernetes","logstash"],"created_at":"2024-09-30T16:14:11.268Z","updated_at":"2025-10-07T04:31:32.451Z","avatar_url":"https://github.com/Azure.png","language":"Go","funding_links":[],"categories":[],"sub_categories":[],"readme":"# helm-elasticstack\n\n## Notice\n\nThis project is  read-only and [archive](https://docs.github.com/en/free-pro-team@latest/github/creating-cloning-and-archiving-repositories/archiving-repositories) mode. No active development anymore.\n\n## Versions\n\n| helm chart version | elastic stack version | tag                                                                      |\n|--------------------|-----------------------|--------------------------------------------------------------------------|\n| 1.1.0              | 6.4.3                 |[v1.1.0](https://github.com/Azure/helm-elasticstack/blob/v1.1.0/README.md)|\n| 1.0.0              | 6.2.4                 |[v1.0.0](https://github.com/Azure/helm-elasticstack/blob/v1.0.0/README.md)|\n\n## Introduction\n\n[![Build Status](https://travis-ci.org/Azure/helm-elasticstack.svg?branch=master)](https://travis-ci.org/Azure/helm-elasticstack)\n[![Go Report Card](https://goreportcard.com/badge/github.com/Azure/helm-elasticstack)](https://goreportcard.com/report/github.com/Azure/helm-elasticstack)\n\nThese [Helm](https://github.com/kubernetes/helm) charts bootstrap a production ready [Elastic Stack](https://www.elastic.co/products) service on a Kubernetes cluster managed by [Azure Container Service (AKS)](https://docs.microsoft.com/en-us/azure/aks/intro-kubernetes) and other Azure services.\n\nThe following features are included:\n\n* Deployment for [Elasticsearch](https://www.elastic.co/products/elasticsearch), [Kibana](https://www.elastic.co/products/kibana) and [Logstash](https://www.elastic.co/products/logstash) services\n* Deployment script which retrieves the secrets and certificates from [Azure Key Vault](https://azure.microsoft.com/en-us/services/key-vault/) and injects them into the Helm charts\n* TLS termination and load balancing for Kibana using [NGINX Ingress Controller](https://github.com/kubernetes/ingress-nginx)\n* [Azure Active Directory](https://docs.microsoft.com/en-us/azure/active-directory/develop/active-directory-authentication-scenarios) authentication for Kibana\n* Integration with [Azure Redis Cache](https://azure.microsoft.com/en-us/services/cache/) acting as middleware for log events between the Log Appenders and Logstash\n* TLS connection between Logstash and Redis Cache handled by [stunnel](https://www.stunnel.org/)\n* Support for [Multiple Data Pipelines](https://www.elastic.co/blog/logstash-multiple-pipelines) in Logstash allowing multiple Redis Caches as input (e.g one Redis cluster per environment)\n* Installation of a [Curator](https://github.com/elastic/curator) cron job that cleans up daily all indexes which are older than 30 days\n* Installation of [Elasticsearch Index Templates](https://www.elastic.co/guide/en/elasticsearch/reference/5.6/indices-templates.html) as a pre-deployment step\n* Installation of [Elasticsearch Watches](https://www.elastic.co/guide/en/elasticsearch/reference/5.6/watcher-api.html) as a post deployment step. The watches can be used for alerts and notifications over Microsoft Teams/Slack webhook or email\n* Installation of [Elasticsearch x-pack license](https://license.elastic.co/download) as a post deployment step\n\n\u003c!-- TOC --\u003e\n\n- [Introduction](#introduction)\n  - [Architecture](#architecture)\n  - [Azure Resources](#azure-resources)\n    - [Azure Key Vault](#azure-key-vault)\n    - [Public Static IP and DNS Domain](#public-static-ip-and-dns-domain)\n    - [Redis Cache](#redis-cache)\n    - [Application for Azure Active Directory](#application-for-azure-active-directory)\n    - [Microsoft Teams/Slack incoming Webhook](#microsoft-teamsslack-incoming-webhook)\n  - [Customize Logstash Configuration](#customize-logstash-configuration)\n    - [Multiple Data Pipelines](#multiple-data-pipelines)\n    - [Indexes Clean Up](#indexes-clean-up)\n    - [Index Templates](#index-templates)\n    - [Index Watches](#index-watches)\n    - [Elasticsearch License](#elasticsearch-license)\n  - [Installation](#installation)\n    - [NGINX Ingress Controller](#nginx-ingress-controller)\n    - [Elasticsearch Cluster](#elasticsearch-cluster)\n    - [Kibana and Logstash](#kibana-and-logstash)\n    - [Rolling Update](#rolling-update)\n  - [Contributing](#contributing)\n\n\u003c!-- /TOC --\u003e\n\n## Architecture\n\n![architecture](images/architecture.png?row=true)\n\n## Azure Resources\n\nA few Azure resources need to be provisioned before proceeding with the Helm charts installation.\n\n### Azure Key Vault\n\nAll secrets and certificates used by the charts are stored in an Azure Key Vault. The deployment script is able to fetch them and to inject them further into the charts.\n\nYou can create a new Key Vault with default permissions:\n\n```console\naz keyvault create --name \u003cKEYVAULT_NAME\u003e --resource-group \u003cRESOURCE_GROUP\u003e\n```\n\nIt is recommended that you use two different principals to operate the Key Vault:\n\n* A _Security Operator_ who has read/write access to secrets, keys and certificates. This principal should be only used for setting up the Key Vault or rotate the secrets.\n* A _Deployment Operator_ who is only able to read secrets. This principal should be used to perform the deployment.\n\nYou can configure the access policies for these principals as follows:\n\n```console\naz keyvault set-policy --upn \u003cSECURITY_OPERATOR_USER_PRINCIPAL\u003e --name \u003cKEYVAULT_NAME\u003e --resource-group \u003cRESOURCE_GROUP\u003e --certificate-permissions create delete get import list update --key-permissions create delete get import list update --secret-permissions get delete list set\n\naz keyvault set-policy --upn \u003cDEPLOYMENT_OPERATOR_USER_PRINCIPAL\u003e --name \u003cKEYVAULT_NAME\u003e --resource-group \u003cRESOURCE_GROP\u003e --secret-permissions get list\n```\n\n### Public Static IP and DNS Domain\n\nYou can allocate a public static IP in Azure. This IP will be used to expose Kibana to the world.\n\n```console\naz network public-ip create -n \u003cIP_NAME\u003e --resource-groug=\u003cRESOURCE_GROUP\u003e --allocation-method=static --dns-name=\u003cIP_NAME\u003e\n```\n\nIt is recommended to utilize the resource group where the ACS/AKS cluster is deployed.\n\nAt this point, you should register your DNS domain name using the public IP returned by Azure and purchase a TLS public certificate. The public certificate should be typically packaged along with the private key in the PKCS12 format in order to import it into Azure Key Vault. You can achieve this with the following command:\n\n```console\nexport DOMAIN=\u003cYOUR DNS DOMAIN\u003e\nopenssl pkcs12 -export -in ${DOMAIN}.cer -inkey ${DOMAIN}.key -out ${DOMAIN}.pfx\n```\n\nThe PKCS12 certificate can be now imported into Key Vault. You should login with your _Security Operator_ principal by executing the `az login` and then execute the import command. During the import, you have to provide a password that it will be used by Key Vault to encrypt the private key.\n\n```console\naz keyvault certificate import --name kibana-certificate --vault-name \u003cKEYVAULT_NAME\u003e -f ${DOMAIN}.pfx --password \u003cPASSWORD\u003e --tags domain=${DOMAIN}\n```\n\nThe private key password must be also stored in a different secret, such that it can be retrieved by the deployment script.\n\n```console\naz keyvault secret set --name kibana-certificate-key-password --vault-name \u003cKEYVAULT_NAME\u003e --value \u003cPASSWORD\u003e\n```\n\n### Redis Cache\n\nThe Azure Redis Cache is used as a middleware between the Log Appenders and Logstash service. This is quite scalable and it also decouples the Log Appenders from Elastic Stack service. You can use any Log Appender which is able to write log events into Redis.\n\n```console\naz redis create --name dev-logscache --location \u003cLOCATION\u003e --resrouce-group \u003cRESOURCE_GROUP\u003e --sku Standard --vm-size C1\n\n```\n\nYou have to store one of the Redis Keys in Key Vault.\n\n```console\naz keyvault secret set --vault-name \u003cKEYVAULT_NAME\u003e --name logstash-dev-redis-key --value=\u003cREDIS_KEY\u003e\n```\n\n### Application for Azure Active Directory\n\nAn Azure Active Directory application of type _Web app/API_ is required in order to use the AAD as an identity provider for Kibana. The authentication is provided by [oauth2_proxy](https://github.com/bitly/oauth2_proxy) reverse proxy which is deployed in the same POD as Kibana.\n\n```console\naz ad app create --display-name Kibana-App --homepage https://${DOMAIN} --reply-urls https://${DOMAIN}/oauth2/callback --identifier-uris https://${DOMAIN}/kibana-app --password \u003cAPPLICATION_SECRET\u003e\n```\n\n\u003e Note that you have to replace the _${DOMAIN}_ with the public DNS domain you registered.\n\nAfter the application was created, you should store the Application ID and secret in Key Vault:\n\n```console\naz keyvault secret set --name kibana-oauth-client-id --vault-name \u003cKEYVAULT_NAME\u003e --value \u003cAPPLICATION_ID\u003e\naz keyvault secret set --name kibana-oauth-client-secret --vault-name \u003cKEYVAULT_NAME\u003e --value \u003cAPPLICATION_SECRET\u003e\n```\n\nIn addition to the application credentials, the `oauth2_proxy` reverse proxy requires a random secret which is used to sign the single-sign-on cookie after login. You can generate and store these secret in Key Vault as follows:\n\n```console\ncookie_secret=$(openssl rand -hex 64)\naz keyvault secret set --name  kibana-oauth-cookie-secret --vault-name \u003cKEYVAULT_NAME\u003e --value ${cookie_secret}\n```\n\nYou should also update the access list with the emails of the users from your organization which are allowed to access Kibana. The white list is in [oauth2-proxy-config-secret.yaml](charts/kibana-logstash/templates/secrets/oauth2-proxy-config-secret.yaml) file.\n\n### Microsoft Teams/Slack incoming Webhook\n\nThe [Elasticsearch Watcher](https://www.elastic.co/guide/en/elasticsearch/reference/master/watcher-api.html) can post notifications into a webhook. For example, you can use a Microsoft Teams webhook, which can be created following these [instructions](https://docs.microsoft.com/en-us/microsoftteams/platform/concepts/connectors). \n\nAfter you configured the webhook, you should store its address in Key Vault:\n\n```console\naz keyvault secret set --vault-name \u003cKEYVAULT_NAME\u003e -n elasticsearch-watcher-webhook-teams --value \"/webhook/\u003cWEBHOOK_URIS\u003e\"\n```\n\nIf you want instead to use a [Slack Incoming Webhook](https://api.slack.com/incoming-webhooks), you can adjust the configuration in the [post-install-watches-secret.yaml](charts/kibana-logstash/templates/post-install-watches-secret.yaml) file.\n\n## Customize Logstash Configuration\n\n### Multiple Data Pipelines\n\nMultiple data pipelines can be defined in the [values.yaml](charts/kibana-logstash/environments/acs/values.yaml) file by creating multiple `stunnel` connections as follows:\n\n```yaml\nstunnel:\n  connections:\n    env1:\n      redis:\n        host: env1-logscache.redis.cache.windows.net\n        port: 6380\n        key:\n      local:\n        host: \"127.0.0.1\"\n        port: 6379\n    env2:\n      redis:\n        host: env2-logscache.redis.cache.windows.net\n        port: 6380\n        key:\n      local:\n        host: \"127.0.0.1\"\n        port: 6378\n```\n\n### Indexes Clean Up\n\nThe old indexes are cleaned up by the [Curator](https://github.com/elastic/curator) tool which is executed daily by a cron job. Its configuration is available in [curator-actions.yaml](charts/kibana-logstash/templates/config/curator-actions.yaml) file. You should adjust it according to your needs.\n\n### Index Templates\n\nThe [Elasticsearch Index Templates](https://www.elastic.co/guide/en/elasticsearch/reference/master/indices-templates.html) are installed automatically by a pre-install job. They are defined in the [pre-install-templates-config.yaml](charts/kibana-logstash/templates/pre-install-templates-config.yaml) file.\n\n### Index Watches\n\nThe [Elasticsearch Watches](https://www.elastic.co/guide/en/elasticsearch/reference/master/watcher-api.html) are also installed automatically by a post-install job. They can be used to trigger any alert or notification based on search queries. The watches configuration is available in [post-install-watches-secret.yaml](charts/kibana-logstash/templates/post-install-watches-secret.yaml) file. You should update this configuration according to your needs.\n\n### Elasticsearch License\n\nIn case you have an [Elasticsearch x-pack license](https://license.elastic.co/download), you can install it when [elasticsearch chart](charts/elasticsearch/README.md) is deployed.\n\n## Installation\n\n### NGINX Ingress Controller\n\nThe `nginx-ingress` will act as a frontend load balancer and it will provide TLS termination for the Kibana public endpoint. Get the latest version from [kubernetes/charts/stable/nginx-ingress](https://github.com/kubernetes/charts/tree/master/stable/nginx-ingress). Before starting the installation, updating e a few Helm values from `values.yaml` file is necessary.\n\nEnable the Kubernetes RBAC by setting:\n\n```console\nrbac.create=true\n```\n\nAnd set the static public IP allocated in Azure, as a load balancer frontend IP:\n\n```console\ncontroller.service.loadBalancerIP: \"\u003cYOUR PUBLIC IP\u003e\"\n```\n\nInstall now the helm package with the following commands:\n\n```console\ncd charts/stable/nginx-ingress\nhelm install -f values.yaml -n nginx-ingress .\n```\n\nAfter the installation is done, verify that the public IP is properly assigned to the controller.\n\n```console\n$\u003e kubectl get svc nginx-ingress-nginx-ingress-controller\n\nNAME                                     TYPE           CLUSTER-IP    EXTERNAL-IP        PORT(S)                      AGE\nnginx-ingress-nginx-ingress-controller   LoadBalancer   10.0.26.141   \u003cYOUR-PUBLIC-IP\u003e   80:32321/TCP,443:31990/TCP   10m\n```\n\n### Elasticsearch Cluster\n\nKibana requires an Elasticsearch cluster which can be installed using the [elasticsearch chart](charts/elasticsearch/README.md). Create a deployment using the `deploy.sh` script available in the chart. Check the [README](charts/elasticsearch/README.md) file for more details:\n\n```console\n./deploy.sh -e acs -n elk\n```\n\nThe command will install an Elasticsearch cluster in the `elk` namespace using the `acs` environment variables.\n\n### Kibana and Logstash\n\nYou can install now the [kibana-logstash](charts/kibana-logstash) chart using the `deploy.sh` script available in the chart. Check the [README](charts/kibana-logstash/README.md) file for more details.\n\n```console\n./deploy.sh -n elk -d \u003cDOMAIN\u003e -v \u003cKEYVAULT_NAME\u003e\n```\n\n\u003e Note to replace the `DOMAIN` with the Kibana DNS domain and the `KEYVAULT_NAME` with the Azure Key Vault name.\n\nThis command installs Kibana and Logstash in the `elk` namespace using the `acs` environment variables. If everything works well, the following output should be shown:\n\n```console\nChecking az command\nChecking helm command\nChecking kubectl command\nChecking openssl command\nRetrieving secrets from KeyVault:\n  Fetching Kibana certificate and private key\n  Fetching OAuth2 proxy secrets\n  Fetching Elasticsearch Watcher secrets\n  Fetching 'acs' environemnt specific secrets\nInstalling mse-elk helm chart\nDone\n```\n\nAnd the deployment should look like this:\n\n```console\n$\u003e kubectl get pods --namespace elk\n\nNAME                                                           READY     STATUS    RESTARTS   AGE\nes-client-7b8599646c-7qfpr                                     1/1       Running   0          1h\nes-client-7b8599646c-nkqjb                                     1/1       Running   0          1h\nes-client-7b8599646c-x2qjr                                     1/1       Running   0          1h\nes-data-0                                                      1/1       Running   0          1h\nes-data-1                                                      1/1       Running   0          1h\nes-data-2                                                      1/1       Running   0          1h\nes-data-3                                                      1/1       Running   0          1h\nes-data-4                                                      1/1       Running   0          1h\nes-data-5                                                      1/1       Running   0          1h\nes-master-57d94ff4f7-ncfzg                                     1/1       Running   0          1h\nes-master-57d94ff4f7-sfpk7                                     1/1       Running   0          1h\nes-master-57d94ff4f7-szrqb                                     1/1       Running   0          1h\nkibana-56c7bf5c46-mwxtl                                        2/2       Running   0          10m\nkibana-56c7bf5c46-t862s                                        2/2       Running   0          10m\nkibana-56c7bf5c46-tdmf8                                        2/2       Running   0          10m\nlogstash-0                                                     2/2       Running   0          10m\nlogstash-1                                                     2/2       Running   0          10m\nlogstash-2                                                     2/2       Running   0          10m\nnginx-ingress-nginx-ingress-controller-7f7488c7c7-wkx42        1/1       Running   0          1h\nnginx-ingress-nginx-ingress-default-backend-7c8bbc9879-cvl79   1/1       Running   0          1h\n```\n\n### Rolling Update\n\nYou can upgrade the charts after the initial installation whenever you have a change, by simply executing again the deployment scripts with the same arguments. Helm will create a new release for you.\n\n## Contributing\n\nThis project welcomes contributions and suggestions.  Most contributions require you to agree to a\nContributor License Agreement (CLA) declaring that you have the right to, and actually do, grant us\nthe rights to use your contribution. For details, visit [https://cla.microsoft.com](https://cla.microsoft.com).\n\nWhen you submit a pull request, a CLA-bot will automatically determine whether you need to provide\na CLA and decorate the PR appropriately (e.g., label, comment). Simply follow the instructions\nprovided by the bot. You will only need to do this once across all repos using our CLA.\n\nThis project has adopted the [Microsoft Open Source Code of Conduct](https://opensource.microsoft.com/codeofconduct/).\nFor more information see the [Code of Conduct FAQ](https://opensource.microsoft.com/codeofconduct/faq/) or\ncontact [opencode@microsoft.com](mailto:opencode@microsoft.com) with any additional questions or comments.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fazure%2Fhelm-elasticstack","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fazure%2Fhelm-elasticstack","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fazure%2Fhelm-elasticstack/lists"}