{"id":28188310,"url":"https://github.com/do-solutions/os-k8s","last_synced_at":"2026-04-24T12:33:52.901Z","repository":{"id":265661148,"uuid":"832650080","full_name":"DO-Solutions/os-k8s","owner":"DO-Solutions","description":"Collect and Forward DigitalOcean Kubernetes (DOKS) Logs to DigitalOcean Managed OpenSearch.","archived":false,"fork":false,"pushed_at":"2024-08-30T09:44:42.000Z","size":13,"stargazers_count":1,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-09-23T06:52:19.867Z","etag":null,"topics":["digitalocean","doks","kubernetes","logging","opensearch"],"latest_commit_sha":null,"homepage":"","language":"Shell","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/DO-Solutions.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-07-23T12:54:36.000Z","updated_at":"2024-08-30T09:45:27.000Z","dependencies_parsed_at":"2024-11-30T11:34:40.078Z","dependency_job_id":"b498dba1-2d5d-4a69-bed3-4f766e26664e","html_url":"https://github.com/DO-Solutions/os-k8s","commit_stats":null,"previous_names":["do-solutions/os-k8s"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/DO-Solutions/os-k8s","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/DO-Solutions%2Fos-k8s","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/DO-Solutions%2Fos-k8s/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/DO-Solutions%2Fos-k8s/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/DO-Solutions%2Fos-k8s/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/DO-Solutions","download_url":"https://codeload.github.com/DO-Solutions/os-k8s/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/DO-Solutions%2Fos-k8s/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":32224165,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-04-24T10:26:35.452Z","status":"ssl_error","status_checked_at":"2026-04-24T10:25:27.643Z","response_time":64,"last_error":"SSL_connect returned=1 errno=0 peeraddr=140.82.121.5:443 state=error: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["digitalocean","doks","kubernetes","logging","opensearch"],"created_at":"2025-05-16T08:12:33.326Z","updated_at":"2026-04-24T12:33:52.885Z","avatar_url":"https://github.com/DO-Solutions.png","language":"Shell","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Collect and Forward DigitalOcean Kubernetes (DOKS) Logs to DigitalOcean Managed OpenSearch.\n\n## Introduction\n\nThis project demonstrates how to collect and forward logs from a DigitalOcean Kubernetes (DOKS) cluster to a DigitalOcean Managed OpenSearch instance using AxoSyslog, a scalable security data processor. By following this guide, you'll learn how to set up a robust logging system that captures and analyzes logs from your Kubernetes applications, making it easier to monitor, troubleshoot, and secure your infrastructure.\n\nIn this guide we will use [AxoSyslog](https://axoflow.com/docs/AxoSyslog-core/intro/) to forward logs from a Kubernetes cluster to OpenSearch.\n\n## Prerequisites\n\nBefore getting started, ensure that you have the following prerequisites in place:\n\n1. **[DigitalOcean Account](https://www.digitalocean.com/):** You'll need access to a DigitalOcean account to create and manage your Kubernetes and OpenSearch resources.\n2. **[doctl CLI](https://docs.digitalocean.com/reference/doctl/how-to/install/):** The DigitalOcean Command Line Interface (CLI) tool, `doctl`, should be installed and configured on your local machine.\n3. **[Kubernetes Cluster](https://docs.digitalocean.com/products/kubernetes/):** A running DigitalOcean Kubernetes (DOKS) cluster.\n4. **[Helm](https://helm.sh/docs/intro/install/):** The Kubernetes package manager, Helm, should be installed to manage Kubernetes applications.\n5. **[Basic Knowledge](https://kubernetes.io/docs/concepts/):** Familiarity with Kubernetes, Helm, and DigitalOcean's managed services.\n\n## Use Case\n\nThis project is ideal for scenarios where you need a centralized logging solution to monitor and analyze logs from various applications running in a Kubernetes cluster. Whether you are managing a small set of applications or a large-scale infrastructure, collecting and forwarding logs to a dedicated OpenSearch cluster helps in:\n\n- **Security Monitoring:** Detect and respond to security incidents by analyzing logs in real time.\n- **Troubleshooting:** Quickly identify and resolve issues within your Kubernetes applications by accessing detailed logs.\n- **Compliance:** Maintain a log of events for compliance with industry regulations.\n\nBy integrating AxoSyslog with DigitalOcean Managed OpenSearch, you can efficiently process and store large volumes of logs, making it easier to extract valuable insights and maintain the health and security of your systems.\n\n## Step 1 - Create OpenSearch cluster\n\nIn this step, you’ll set up the core component of your logging system, the OpenSearch cluster. OpenSearch will serve as the destination for all the logs you collect from your Kubernetes cluster. By running the following command, you’ll create a new OpenSearch instance in your chosen region on DigitalOcean.\n\n```sh\ndoctl databases create opensearch-doks --engine opensearch --region lon1 --size db-s-1vcpu-2gb --num-nodes 1\n```\n\nReplace `lon1` with your desired region. For a list of available size slugs, visit our [API reference documentation.](https://docs.digitalocean.com/reference/api/api-reference/#tag/Databases)\n\n## Step 2 - Generate some random logs\n\nBefore you can forward logs to OpenSearch, you need some logs to work with. If you don’t have an application already generating logs within your Kubernetes cluster, this step will show you how to deploy a log generator. This log generator will produce a steady stream of sample logs that can be used to test and demonstrate your logging pipeline.\n\nFirst, add the log generator Helm chart repository and install the log generator:\n\n```sh\nhelm repo add kube-logging https://kube-logging.github.io/helm-charts\nhelm repo update\n```\n\nThen, install the log generator using Helm:\n\n```sh\nhelm install --generate-name --wait kube-logging/log-generator\n```\n\nYou can verify that the log generator is working by viewing the logs it produces:\n\n```sh\nkubectl logs -l app.kubernetes.io/name=log-generator\n```\n\n## Step 3 - Prepare AxoSyslog Collector for Installation\n\nIn this step, you’ll configure the AxoSyslog Collector, which is responsible for gathering logs from your Kubernetes cluster and forwarding them to OpenSearch. This involves providing the correct connection details for your OpenSearch cluster (hostname, user, and password).\n\nWe'll use helm to install AxoSyslog Collector and pass custom values.\n\nTo configure the AxoSyslog collector with the correct address, user, and password for your OpenSearch database, follow these steps:\n\n### Automated Script\n\nTo simplify the configuration, you can use an automated script that fetches the necessary OpenSearch connection details and updates your AxoSyslog configuration file.\n\nSave the following script as `update_axoflow_demo.sh`:\n\n```sh\n#!/bin/bash\n\n# Extract Database ID for opensearch-doks\nDB_ID=$(doctl databases list --format Name,ID --no-header | grep opensearch-doks | awk '{print $2}')\n\n# Get Hostname, Username, and Password\nOPENSEARCHHOSTNAME=$(doctl databases connection $DB_ID --no-header --format Host)\nOPENSEARCHUSERNAME=$(doctl databases connection $DB_ID --no-header --format User)\nOPENSEARCHPASSWORD=$(doctl databases connection $DB_ID --no-header --format Password)\n\n# Update axoflow-demo.yaml with extracted values using yq\nyq eval \".config.destinations.opensearch[0].address = \\\"$OPENSEARCHHOSTNAME\\\"\" -i axoflow-demo.yaml\nyq eval \".config.destinations.opensearch[0].user = \\\"$OPENSEARCHUSERNAME\\\"\" -i axoflow-demo.yaml\nyq eval \".config.destinations.opensearch[0].password = \\\"$OPENSEARCHPASSWORD\\\"\" -i axoflow-demo.yaml\n\necho \"axoflow-demo.yaml has been updated.\"\n```\n\nEnsure you have execute permission on your script before running it:\n\n```sh\nchmod +x update_axoflow_demo.sh \u0026\u0026 ./update_axoflow_demo.sh\n```\n\nThis script will fetch the necessary information from your DigitalOcean account using `doctl` and update your `axoflow-demo.yaml` file accordingly.\n\n### Manual Steps to Update `axoflow-demo.yaml`\n\nIf you prefer to manually configure your AxoSyslog Collector, follow these steps:\n\nRun the following command to extract database ID for `opensearch-doks`:\n\n```sh\ndoctl databases list --format Name,ID --no-header | grep opensearch-doks | awk '{print $2}'\n```\n\nTo retrieve hostname, username, and password, execute the following commands respectively:\n\n```sh\ndoctl databases connection \u003cid\u003e --no-header --format Host\ndoctl databases connection \u003cid\u003e --no-header --format User\ndoctl databases connection \u003cid\u003e --no-header --format Password\n```\n\nNow, you need to manually update the `axoflow-demo.yaml` file:\n\n   Open your `axoflow-demo.yaml` file in a text editor and replace the relevant fields with the extracted values:\n\n   ```yaml\n   config:\n     sources:\n       kubernetes:\n         # Collect kubernetes logs\n         enabled: true\n     destinations:\n       # Send logs to OpenSearch\n       opensearch:\n         - address: \"x.k.db.ondigitalocean.com\"\n           index: \"doks-demo\"\n           user: \"doadmin\"\n           password: \"AVNS_x\"\n           tls:\n             # Do not validate the server's TLS certificate.\n             peerVerify: false\n           # Send the syslog fields + the metadata from .k8s.* in JSON format\n           template: \"$(format-json --scope rfc5424 --exclude DATE --key ISODATE @timestamp=${ISODATE} k8s=$(format-json .k8s.* --shift-levels 2 --exclude .k8s.log))\"\n   ```\n\n## Step 4 - Install AxoSyslog-collector\n\nNow that the configuration is complete, the next step is to deploy the AxoSyslog Collector to your Kubernetes cluster. This will enable the collection and forwarding of logs to OpenSearch.\n\nAdd the AxoSyslog Helm repository and install the AxoSyslog Collector using the customized configuration file:\n\n```sh\nhelm repo add AxoSyslog https://axoflow.github.io/AxoSyslog-charts\nhelm repo update\n```\n\n```sh\nhelm install AxoSyslog -f axoflow-demo.yaml AxoSyslog/AxoSyslog-collector --wait\n```\n\nTo ensure that logs are being sent to the correct OpenSearch port, update the AxoSyslog Collector’s configuration by updating your `configmap`:\n\n```sh\nkubectl get configmap AxoSyslog-AxoSyslog-collector -o yaml | sed 's/9200\\/_bulk/25060\\/_bulk/' | kubectl apply -f -\n```\n\nFinally, delete the existing pods to apply the updated configuration:\n\n```sh\nkubectl delete pods -l app=AxoSyslog-AxoSyslog-collector\n```\n\n## Conclusion\n\nSetting up a logging pipeline from DigitalOcean Kubernetes to OpenSearch using AxoSyslog not only centralizes your logs but also enhances your ability to monitor, analyze, and secure your applications. With the steps provided in this guide, you can quickly deploy this solution, gaining deeper visibility into your Kubernetes environment and ensuring that your infrastructure remains resilient and compliant.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdo-solutions%2Fos-k8s","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fdo-solutions%2Fos-k8s","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdo-solutions%2Fos-k8s/lists"}