https://github.com/crowdstrike/logscale-azure-aks-deployment
LogScale Cluster Deployment in Azure AKS
https://github.com/crowdstrike/logscale-azure-aks-deployment
Last synced: 5 months ago
JSON representation
LogScale Cluster Deployment in Azure AKS
- Host: GitHub
- URL: https://github.com/crowdstrike/logscale-azure-aks-deployment
- Owner: CrowdStrike
- License: mit
- Created: 2023-06-28T17:13:57.000Z (about 3 years ago)
- Default Branch: main
- Last Pushed: 2024-04-09T15:41:36.000Z (about 2 years ago)
- Last Synced: 2025-09-10T06:21:52.105Z (10 months ago)
- Size: 434 KB
- Stars: 1
- Watchers: 4
- Forks: 0
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
- License: LICENSE
- Code of conduct: CODE_OF_CONDUCT.md
Awesome Lists containing this project
README
# LogScale Cluster Deployment in Azure AKS
## Overview :
This document is a guide to provision a self-hosted LogScale cluster on Azure Cloud using Azure AKS kubernetes, with Azure object store for event repositories.
The sequence of steps and associated tasks required to provision LogScale are split into sections for planning, deployment, and validation. Sections should be processed in top-down order to ensure that prerequisites are met as provisioning steps are executed.
Notes:
- This cluster deployment utilizes an independent Kafka service.
- This document assumes at least intermediate level knowledge of Azure Cloud.
## Architectural Diagrams:
**Deployment Overview:**
**Functional Overview:**
## Prerequisites :
- Falcon LogScale License Key
- LogScale Instance sizing guidelines:
- [Instance Sizing](https://library.humio.com/falcon-logscale-self-hosted/installation-provisioning-sizing.html)
- [Recommended Instance Sizing](https://library.humio.com/falcon-logscale-self-hosted/installation-prep-rec.html)
- Azure Portal account with an active subscription
- Azure Storage account
- [Create a storage account](https://learn.microsoft.com/en-us/azure/storage/common/storage-account-create?tabs=azure-portal)
- Azure Blob Container (i.e. storage bucket)
- [Create a blob container](https://learn.microsoft.com/en-us/azure/storage/blobs/storage-quickstart-blobs-portal#create-a-container)
- Good understanding of [Azure Resource groups](https://learn.microsoft.com/en-us/azure/azure-resource-manager/management/manage-resource-groups-portal)
- Good understanding Kubernetes
## Deployment:
**1. Create an AKS cluster using any of the following methods:**
- [Azure CLI](https://learn.microsoft.com/en-us/azure/aks/tutorial-kubernetes-deploy-cluster?tabs=azure-cli)
- [Azure Portal](https://learn.microsoft.com/en-us/azure/aks/learn/quick-kubernetes-deploy-portal?tabs=azure-cli#create-an-aks-cluster)
- [Terraform](https://learn.microsoft.com/en-us/azure/aks/learn/quick-kubernetes-deploy-terraform?tabs=azure-cli#implement-the-terraform-code)
Validate the cluster is up and running by checking kubernetes service status:
- kubectl get svc
**2. Deploy Kafka/ZooKeeper to the AKS cluster using the** [strimzi-kafka-operator](https://github.com/strimzi/strimzi-kafka-operator)
- Note: Default partition count is 24. For details, refer to the [LogScale Digest Rules](https://library.humio.com/falcon-logscale-self-hosted/cluster-management-digest-rules.html)
- Deploy the strimzi-kafka-operator (kafka/k8s orchestration facility - [strimzi key features](https://strimzi.io/docs/operators/latest/overview.html#key-features-product_str))
- Execute following commands in K8s environment
- helm repo add strimzi [https://strimzi.io/charts/](https://strimzi.io/charts/)
- helm repo update
- helm install strimzi-kafka strimzi/strimzi-kafka-operator
- Provision the Kafka+zookeeper cluster
- Configure Kafka/zookeeper parameters (see sample kafka-zookeeper.yaml)
- replicas
- offsets.topic.replication.factor
- transaction.state.log.replication.factor
- transaction.state.log.min.isr
- default.replication.factor
- min.insync.replicas
- Inter.broker.protocol.version
- storage.size
- Apply Kafka yaml
- Validate installation by checking pods and services
- kubectl get pods
- kubectl get svc
**3. Deploy S3Proxy**
- LogScale supports S3 compatible storage using S3Proxy service. Azure Blob Storage is one of them. For details, refer to the [S3Proxy documentation](https://github.com/gaul/s3proxy).
- Provision S3Proxy node(s)
- Configure parameters as required (see sample s3proxy.yaml)
| Name | Value |
| :-- | :-- |
| S3PROXY\_AUTHORIZATION | `none` |
| JCLOUDS\_PROVIDER | `azureblob` |
| JCLOUDS\_ENDPOINT | \ |
| JCLOUDS\_IDENTITY | \ |
| JCLOUDS\_CREDENTIAL | \ |
| JCLOUDS\_AZYREBLOB\_AUTH | `azureKey` |
| LOG\_LEVEL | `debug` |
- Apply S3Proxy & S3Proxy service (see sample s3proxy-service.yaml)
- Verify that s3proxy pod and service are running.
**4. Deploy LogScale**
- Note: This deployment uses NODE\_ROLES="all". For details, refer to the [NODE_ROLES documentation](https://library.humio.com/falcon-logscale-self-hosted/envar-node-roles.html).
- Deploy LogScale Operator (see setup\_logscale.sh)
- Obtain the latest stable version from [LogScale Operator Releases](https://github.com/humio/humio-operator/releases).
- Apply LogScale CRDs & Operator
- Configure LogScale
- Create a license key secret
- kubectl create secret generic example-humiocluster-license --from-literal=data=\
- Obtain the latest stable version from [LogScale Releases](https://library.humio.com/release-notes/release-notes-stable.html).
- In HumioCluster yaml, set the parameters as required (see sample logscale-cluster.yaml)
- Set the parameters under spec
- targetReplicationFactor
- storagePartitionsCount
- digestPartitionsCount
- Set the Env. variables under spec.environmentVariables which are required to enable the S3 compatible storage
| Name | Value |
| :-- | :-- |
| USING\_EPHEMERAL\_DISKS | `true` |
| NODE\_ROLES | `all\|ingestonly\|httponly`|
| S3\_STORAGE\_ENDPOINT\_BASE | \ |
| S3\_STORAGE\_ACCESSKEY | \ |
| S3\_STORAGE\_SECRETKEY | \ |
| LOCAL\_STORAGE\_PERCENTAGE | "80" |
| S3\_STORAGE\_PATH\_STYLE\_ACCESS | `true` |
| S3\_STORAGE\_IBM\_COMPAT | `true` |
| BUCKET\_STORAGE\_IGNORE\_ETAG\_UPLOAD | `true` |
| BUCKET\_STORAGE\_IGNORE\_ETAG\_AFTER\_UPLOAD | `false` |
| BUCKET\_STORAGE\_SSE\_COMPATIBLE | `true` |
| S3\_STORAGE\_ENCRYPTION\_KEY | `off` |
| S3\_STORAGE\_BUCKET | "\" |
| S3\_ARCHIVING\_PATH\_STYLE\_ACCESS | `true` |
| S3\_EXPORT\_PATH\_STYLE\_ACCESS | `true` |
| S3\_STORAGE\_PREFERRED\_COPY\_SOURCE | `true`
More information on environment variables : [Falcon LogScale - Configuration Parameters](https://library.humio.com/falcon-logscale-self-hosted/envar.html)
- Apply HumioCluster yaml
- Status of each container can be checked by running following command:
- kubectl get pods
- kubectl describe pod \
**5. Test and Verify**
- Check availability of pods and services
- kubectl get pods
- kubectl get svc
- Test connectivity and availability of LogScale cluster forwarding the port locally
- kubectl port-forward svc/example-humiocluster 8888:8080
**6. Deploy Application gateway and enable AGIC**
- Deploy Application Gateway Ingress Controller Add-on to access LogScale App from the Internet
- [Create and configure Application Gateway](https://learn.microsoft.com/en-us/azure/application-gateway/quick-create-portal)
- [Enable the ingress controller add-on for a new AKS cluster](https://learn.microsoft.com/en-us/azure/application-gateway/tutorial-ingress-controller-add-on-new)
**Useful Reference Links:**
- [LogScale K8s Reference Architecture](https://library.humio.com/falcon-logscale-self-hosted/installation-k8s-ref-arch.html)
- [K8s core concept for AKS](https://learn.microsoft.com/en-us/azure/aks/concepts-clusters-workloads)
WE STOP BREACHES