{"id":22319917,"url":"https://github.com/aldis-ameriks/affordable-kubernetes","last_synced_at":"2025-06-23T17:34:08.320Z","repository":{"id":108854525,"uuid":"156694800","full_name":"aldis-ameriks/affordable-kubernetes","owner":"aldis-ameriks","description":"Setting up an affordable Kubernetes cluster in GKE for personal projects","archived":false,"fork":false,"pushed_at":"2019-01-04T14:19:10.000Z","size":3555,"stargazers_count":4,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"master","last_synced_at":"2025-01-31T06:07:01.854Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/aldis-ameriks.png","metadata":{"files":{"readme":"readme.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2018-11-08T11:11:54.000Z","updated_at":"2022-01-05T09:39:02.000Z","dependencies_parsed_at":"2023-05-18T08:46:19.145Z","dependency_job_id":null,"html_url":"https://github.com/aldis-ameriks/affordable-kubernetes","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/aldis-ameriks%2Faffordable-kubernetes","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/aldis-ameriks%2Faffordable-kubernetes/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/aldis-ameriks%2Faffordable-kubernetes/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/aldis-ameriks%2Faffordable-kubernetes/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/aldis-ameriks","download_url":"https://codeload.github.com/aldis-ameriks/affordable-kubernetes/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":245588029,"owners_count":20640051,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-12-04T00:11:46.254Z","updated_at":"2025-03-26T04:27:48.088Z","avatar_url":"https://github.com/aldis-ameriks.png","language":"Go","funding_links":[],"categories":[],"sub_categories":[],"readme":"# The affordable GKE Kubernetes cluster\n\nMy goal was to set up cheap, fully functional and simple Kubernetes cluster for personal projects. But above all, to learn the ways of Kubernetes. The following setup will create a 3 node Kubernetes cluster that will cost ~5$ per month.\n\nMost of the steps and configuration taken from: https://www.doxsey.net/blog/kubernetes--the-surprisingly-affordable-platform-for-personal-projects.\n\nIf for some odd reason someone is reading this, I apologize for the ungenerous explanations. I'd urge you to go ahead and check the blog post, linked above.\n\n-----\n\n## Table of Contents\n1. [Create a new project](#section-1)\n2. [Create a new cluster](#section-2)\n3. [Install `gcloud` and `docker`](#section-3)\n4. [Open firewall](#section-4)\n5. [Nginx certificates](#section-5)\n6. [Build 'n' deploy](#section-6)\n7. [Get a domain](#section-7)\n8. [Automate updating DNS](#section-8)\n9. [Prometheus \u0026 Grafana](#section-9)\n\nStarting off with clean GCP account.\n\n\u003e Below I will go over the configuration options that I used to set up my cluster, if someone else is reading this, go wild and adapt it to your needs.\n\n### 1. Create a new project. I called mine the `the-affordable-project`.\u003ca name=\"section-1\"\u003e\u003c/a\u003e\n\n### 2.  Head over to `Kubernetes Engine` -\u003e `Clusters` and create a new Cluster with following parameters: \u003ca name=\"section-2\"\u003e\u003c/a\u003e\n    * Name: `affordable-cluster-1`\n    * Location type: `Zonal`\n    * Zone: `us-east1` (Always free nodes are available in following regions: us-west1, us-central1, us-east1. Source: https://cloud.google.com/compute/pricing)\n    * Master version: at this time, I'm using the latest one, `1.11.2-gke.9`\n    * Node pools\n        * Number of nodes: `3`\n        * Machine type: `f1-micro` (because we're cheap)\n        * Boot disk type: `SSD persistent disk`\n        * Boot disk size (GB): `10`\n        * Enable preemptible nodes (beta): `check` (the nodes will be cheaper)\n        * Enable auto-upgrade: `check`\n        * Enable auto-repair: `check`\n        * Enable autoscaling: `uncheck`\n    * Load balancing\n        * Enable HTTP load balancing: `uncheck` (gce provided load balancers are too pricey for our affordable cluster)\n    * Security\n        * Enable basic authentication: `check`\n        * Issue a client certificate: `check` \n    * Additional features\n        * Enable Stackdriver Logging service: `uncheck` (Stackdriver features are not free)\n        * Enable Stackdriver Monitoring service: `uncheck`\n        * Enable Kubernetes Dashboard: `uncheck` (Kubernetes Dashboard is deprecated and not really that useful. A better alternative could be Grafana with Prometheus)\n\nOur cluster:\n\n![alt text](img/1_cluster.png)\n\nWorkloads for a fresh cluster\n\u003e This is mostly for myself. Sometimes when running experiments, bloat in the form of unused resources might find it's way in the cluster. Good to know what were the original resources when cluster was first created.\n\n![alt text](img/2_workloads.png)\n\nServices for a fresh cluster\n\n![alt text](img/3_services.png)\n\nConfiguration for a fresh cluster\n\n![alt text](img/4_configuration.png)\n\n### 3. Install `gcloud` and `docker`. And then follow the steps to configure the clients:\u003ca name=\"section-3\"\u003e\u003c/a\u003e\n```\ngcloud auth login\ngcloud auth configure-docker\ngcloud init\n```\n\nIf a different project has already been configured previously, might have to fetch the cluster credentials.\n\n`gcloud container clusters get-credentials affordable-cluster-1 --project the-affordable-project`\n\n\n### 4. Since we're not going to use GCE load balancers, we will need to open our nodes to the public network. Before we can do that, we need a firewall opening. Head to https://console.cloud.google.com/networking/firewalls/list and create a new firewall rule.\u003ca name=\"section-4\"\u003e\u003c/a\u003e\n\n* Name: `http`\n* Targets: `All instances in the network`\n* Source IP ranges: `0.0.0.0/0`\n* Protocols and ports\n    * Specified protocols and ports:\n        * tcp: `80, 443`\n\n\u003e When doing the setup the first time, I missed the `Targets` option and left the default one which needs tags to be set. Without thinking, I added some value and afterwards spent a very long time debugging why access to the nodes was not working.\n\n\n### 5. Generate and upload self signed certs for nginx. Note that it's not safe and should not be used in a production environment. I did it to simplify the setup.\u003ca name=\"section-5\"\u003e\u003c/a\u003e\n\nGenerate certs: `sudo openssl req -x509 -nodes -days 365 -newkey rsa:2048 -keyout self-signed.key -out self-signed.crt`\n\nUpload them to GKE as a secret: `kubectl create secret generic nginx-certs --from-file self-signed.crt --from-file self-signed.key`\n\nThe nginx-certs secret should be visible under Configuration.\n\n### 6. Build and deploy app and nginx containers\u003ca name=\"section-6\"\u003e\u003c/a\u003e\n\nBuilding docker images:\n```\ndocker build -t gcr.io/the-affordable-project/publicgo:latest -f Dockerfile src/publicgo\ndocker build -t gcr.io/the-affordable-project/internalgo:latest -f Dockerfile src/internalgo\n\ndocker push gcr.io/the-affordable-project/publicgo:latest\ndocker push gcr.io/the-affordable-project/internalgo:latest\n```\n\nDeploy:\n```\nkubectl apply -f publicgo.yaml\nkubectl apply -f internalgo.yaml\nkubectl apply -f nginx.yaml\n```\n\nTest that it works. Find the external IPs for your nodes: `kubectl get nodes -o yaml | less` look for `ExternalIP`, should contain one entry for each node. Try them out:\n`http://\u003cExternalIP\u003e/go` should yield `Hello from publicgo` and `http://\u003cExternalIP\u003e/go/pinginternal` should yield `Here's the response from internalgo: Hello from internalgo`. Now see if the self signed certs are correctly set up: `https://\u003cExternalIP\u003e/go`.\n\n\n### 7. Get a domain and add `A` records that point to the 3 external ip addresses.\u003ca name=\"section-7\"\u003e\u003c/a\u003e\nIf all worked well, now we should have fully functional Kubernetes cluster on GKE.\n\u003e Ignore the Grafana entries, that was not part of this setup.\n\n![alt text](img/6_final_workloads.png)\n\n![alt text](img/7_final_services.png)\n\n### 8. Automate DNS updates.\u003ca name=\"section-8\"\u003e\u003c/a\u003e\nPreemtibles nodes live up to 24h after which they're destroyed and recreated. When that happens, they're assigned a new externalip. To avoid having to manually update `A` records everyday, there's a way to automate it. There's a great solution for that in the blog post I mentioned in the beginning that assumes using Cloudflare DNS services.\n\n\n### 9. Set up monitoring and alerting capabilities using Prometheus, Alertmanager, PushGateway and Grafana.\u003ca name=\"section-9\"\u003e\u003c/a\u003e\nSet up Tiller and Helm\nhttps://docs.helm.sh/using_helm/#installing-helm\n```\nbrew install kubernetes-helm\nhelm init\nkubectl create serviceaccount --namespace kube-system tiller\nkubectl create clusterrolebinding tiller-cluster-rule --clusterrole=cluster-admin --serviceaccount=kube-system:tiller\nkubectl patch deploy --namespace kube-system tiller-deploy -p '{\"spec\":{\"template\":{\"spec\":{\"serviceAccount\":\"tiller\"}}}}'\n```\n\nInstall Prometheus, Alertmanager and PushGateway\n```\nhelm install --name prometheus --namespace monitoring stable/prometheus\n```\n\nVerify that the setup is working.\nRun the following command and open localhost:9090 to boot up the Prometheus ui. \n```\n  export POD_NAME=$(kubectl get pods --namespace default -l \"app=prometheus,component=server\" -o jsonpath=\"{.items[0].metadata.name}\")\n  kubectl --namespace default port-forward $POD_NAME 9090\n```\n\nSame for Alertmanager on :9093\n```\n  export POD_NAME=$(kubectl get pods --namespace default -l \"app=prometheus,component=alertmanager\" -o jsonpath=\"{.items[0].metadata.name}\")\n  kubectl --namespace default port-forward $POD_NAME 9093\n```\n\nAnd PushGateway on :9091\n```\n  export POD_NAME=$(kubectl get pods --namespace default -l \"app=prometheus,component=pushgateway\" -o jsonpath=\"{.items[0].metadata.name}\")\n    kubectl --namespace default port-forward $POD_NAME 9091\n```\n\nInstall Grafana\n```\nhelm install --name grafana --namespace monitoring stable/grafana\n```\n\nGet password:\n```\nkubectl get secret --namespace default grafana -o jsonpath=\"{.data.admin-password}\" | base64 --decode ; echo\n```\n\nOpen tunnel\n```\nexport POD_NAME=$(kubectl get pods --namespace default -l \"app=grafana\" -o jsonpath=\"{.items[0].metadata.name}\")\nkubectl --namespace default port-forward $POD_NAME 3000\n```\n\nOpen localhost:3000 and login using `admin` and password you got from secrets.\nNext configure Prometheus datasource using internal dns (change the name to however you named your prometheus stack)\n```\nhistorical-rottweiler-prometheus-server.default.svc.cluster.local\n```\n\nFind a fancy dashboard here https://grafana.com/dashboards?search=Kubernetes and import it to your Grafana instance using the dashboard id.\n\n\n\n### Encountered issues\nWhen removing/adding services, they gain new cluster ips. We're using internal dns to resolve the ip address of the cluster (e.g. `publicgo.default.svc.cluster.local` might resolve to `10.3.xxx.xxx`) when routing traffic from nginx. Nginx caches the dns, so when deleting/creating services nginx could start returning 504 Bad Gateway. The quickest solution I found was to recreate the nginx `kubectl delete -f nginx.yaml \u0026\u0026 kubectl apply -f nginx.yaml`. I don't enjoy doing that, but at least it's quite fast.\n\n\n\n### Todo\nAdd NGINX Ingress controller and configure monitoring\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Faldis-ameriks%2Faffordable-kubernetes","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Faldis-ameriks%2Faffordable-kubernetes","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Faldis-ameriks%2Faffordable-kubernetes/lists"}