{"id":22026127,"url":"https://github.com/curiouslearner/kube-efk","last_synced_at":"2026-04-09T23:11:19.950Z","repository":{"id":81377437,"uuid":"436298834","full_name":"CuriousLearner/kube-efk","owner":"CuriousLearner","description":"Deploying EFK stack on Kubernetes","archived":false,"fork":false,"pushed_at":"2021-12-27T19:22:52.000Z","size":2002,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":2,"default_branch":"main","last_synced_at":"2025-07-12T06:27:07.257Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":null,"has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/CuriousLearner.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null}},"created_at":"2021-12-08T15:33:27.000Z","updated_at":"2021-12-27T19:22:54.000Z","dependencies_parsed_at":null,"dependency_job_id":"c599b2ed-c18b-49b3-870e-6c04cc330eed","html_url":"https://github.com/CuriousLearner/kube-efk","commit_stats":{"total_commits":7,"total_committers":2,"mean_commits":3.5,"dds":0.1428571428571429,"last_synced_commit":"853b16dc50457f5c7f7fec1ca24fcdf66f5908e4"},"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/CuriousLearner/kube-efk","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/CuriousLearner%2Fkube-efk","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/CuriousLearner%2Fkube-efk/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/CuriousLearner%2Fkube-efk/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/CuriousLearner%2Fkube-efk/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/CuriousLearner","download_url":"https://codeload.github.com/CuriousLearner/kube-efk/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/CuriousLearner%2Fkube-efk/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":265896119,"owners_count":23845426,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-30T07:25:21.848Z","updated_at":"2026-04-09T23:11:19.917Z","avatar_url":"https://github.com/CuriousLearner.png","language":null,"funding_links":[],"categories":[],"sub_categories":[],"readme":"# kube-efk\nDeploying EFK stack on Kubernetes\n\n__NOTE__: For deployment to Digital Ocean, I reduced the replicas of ES StatefulSet to 1 from 3.\n\nThis will deploy Elastic Search, Fluentd, and Kibana on a Kubernetes cluster.\n\nElastic Search (ES) is used to store the logs data and search it.\nKibana provides a dashboard to effectively search data on ES and make dashboards.\nFluentd is used instead of Logstash (in traditional ELK stack) for collecting, transforming, and shipping data to ES.\n\nElastic Search will be run as a StatefulSet and have a persistent volume attached. Currently, we'll have 3 pods running for ES. We'll also have a service to connect with Kibana.\n\nFluentd will be run as a daemon-set so that it is available on all worker nodes to collect log data from the containers and provide it to ES.\n\nKibana will be run as a deployment of 1 pod and service to interact with Elastic search for displaying data.\n\n## Steps\n\n### For deployment to DO\n\nCreate a Kubernetes cluster on Digital Ocean\n\nInstall `doctl` and save the config to kube so that kubectl can interact with it directly\n\n```bash\ndoctl kubernetes cluster kubeconfig save \u003ccluster-id-from-DO\u003e\n```\n\nNow we can continue rest of the steps. For deployment to consume less resources, I deployed this with 1 Elastic Search replica on DO, while tried out with 3 replicas locally using Minikube on Docker.\n\n### Steps for deploying EFK\n\nWe'll have all the pods/deployment/replica-set in `kube-logging` namespace.\n\n```bash\nkubectl create -f kube-logging.yaml\n```\n\nCreating ES service:\n\n```bash\nkubectl create -f elasticsearch_svc.yaml\n```\n\nCreating ES StatefulSet and connecting it to SVC:\n\n```bash\nkubectl create -f elasticsearch_statefulset.yaml\n```\n\n**Optional**: We can monitor StatefulSet as it is rolled out with 3 pods:\n\n```bash\nkubectl rollout status sts/es-cluster --namespace=kube-logging\n```\n\nFrom one terminal, we can forward port 9200 for ES-cluster-0 to the local machine using:\n\n```bash\nkubectl port-forward es-cluster-0 9200:9200 --namespace=kube-logging\n```\n\n**Optional**: We can test the connection with ES using the curl command:\n\n```bash\ncurl http://localhost:9200/_cluster/state?pretty\n```\n\nA response similar to this might be seen on successful connection:\n\n```bash\n{\n  \"name\": \"es-cluster-0\",\n  \"cluster_name\": \"k8s-logs\",\n  \"cluster_uuid\": \"PS3a_GbPQQ6LltHpUeQYSQ\",\n  \"version\": {\n    \"number\": \"7.2.0\",\n    \"build_flavor\": \"default\",\n    \"build_type\": \"docker\",\n    \"build_hash\": \"508c38a\",\n    \"build_date\": \"2019-06-20T15:54:18.811730Z\",\n    \"build_snapshot\": false,\n    \"lucene_version\": \"8.0.0\",\n    \"minimum_wire_compatibility_version\": \"6.8.0\",\n    \"minimum_index_compatibility_version\": \"6.0.0-beta1\"\n  },\n  \"tagline\": \"You Know, for Search\"\n}\n```\n\n![Response from Elastic Search](./images/es-response.png)\n\n\nNow, we'll have the Kibana deployment and service deployed:\n\n```bash\nkubectl create -f kibana.yaml\n```\n\n**Optional**: The rollout status can be viewed via:\n\n```bash\nkubectl rollout status deployment/kibana --namespace=kube-logging\n```\n\nIn a second terminal we can port forward 5601 of Kibana to local using:\n\n```bash\nkubectl port-forward \u003creplace-this-with-kibana-pod\u003e 5601:5601 --namespace=kube-logging\n```\n\nFinally, we'll spawn Fluentd as a Daemon-Set so that it is available on all the worker nodes using:\n\n```bash\nkubectl create -f fluentd.yaml\n```\n\nWe can verify that DaemonSet rolled out successfully using:\n\n```bash\nkubectl get ds --namespace=kube-logging\n```\n\nWe can now verify all the pods in the namespace kube-logging. It should have 3 pods of elastic search, and one each for fluentd and kibana, given that we're on a single node.\n\n```bash\nkubectl get pods --namespace=kube-logging\n```\n\n![Final pods](./images/kubectl-final-pods.png)\n\n\nNow we can access the Kibana dashboard on [http://localhost:5601](http://localhost:5601)\n\n\n![Kibana Dashboard Home](./images/kibana-dashboard-home.png)\n\nWe can create an index pattern:\n\n![Kibana Create Index Pattern](./images/kibana-creating-index-pattern.png)\n\nand then can see the analytics:\n\n![Kibana Dashboard Analytics for the index pattern](./images/kibana-analytics.png)\n\nThat's it. Now we can detect all the logs using fluentd and search ES through Kibana.\n\nFor deploying on Digital Ocean, the Stateful Set for Elastic Search was reduced to 1 pod. The final pods looks like this:\n\n![Digital Ocean Pods](./images/do-pods.png)\n\nand we can see everything deployed on Digital Ocean interface\n\n![Digital Ocean Deployed Kubernetes cluster resources](./images/do-deployed.png)\n\n-----------------\n\nThis is a sample project I did to learn `doctl` for managing resources on Digital Ocean and Kubernetes basics.\n\nI port-forwarded from DO kubernetes cluster to demo it locally.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcuriouslearner%2Fkube-efk","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fcuriouslearner%2Fkube-efk","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcuriouslearner%2Fkube-efk/lists"}