{"id":19766250,"url":"https://github.com/akaliutau/gke-pubsub","last_synced_at":"2026-05-08T06:16:52.367Z","repository":{"id":120056008,"uuid":"445376102","full_name":"akaliutau/gke-pubsub","owner":"akaliutau","description":"A simple research project to study HPA technology on Kubernetes cluster","archived":false,"fork":false,"pushed_at":"2022-01-10T09:06:51.000Z","size":321,"stargazers_count":0,"open_issues_count":0,"forks_count":1,"subscribers_count":1,"default_branch":"master","last_synced_at":"2025-01-11T00:13:32.836Z","etag":null,"topics":["autoscaling","gke-cluster","hpa","java11-spring-boot","kubernetes","pubsub","terraform"],"latest_commit_sha":null,"homepage":"","language":"Java","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/akaliutau.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2022-01-07T02:39:39.000Z","updated_at":"2022-01-10T09:06:55.000Z","dependencies_parsed_at":null,"dependency_job_id":"4d507dfc-f409-40b0-8620-96184f930cae","html_url":"https://github.com/akaliutau/gke-pubsub","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/akaliutau%2Fgke-pubsub","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/akaliutau%2Fgke-pubsub/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/akaliutau%2Fgke-pubsub/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/akaliutau%2Fgke-pubsub/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/akaliutau","download_url":"https://codeload.github.com/akaliutau/gke-pubsub/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":241099623,"owners_count":19909577,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["autoscaling","gke-cluster","hpa","java11-spring-boot","kubernetes","pubsub","terraform"],"created_at":"2024-11-12T04:23:32.496Z","updated_at":"2025-11-06T19:03:12.632Z","avatar_url":"https://github.com/akaliutau.png","language":"Java","funding_links":[],"categories":[],"sub_categories":[],"readme":"# About\r\n\r\nAlice and Bob are big friends, but they live in different cities (Alice in Amsterdam and Bob in Boston) and can't see each other in person.\r\nSo they communicate mostly via letters writing.\r\n\r\nAlice just loves receiving letters, but has little time to read them. \r\nShe decided to create a Reading Machine on the basis of Kubernetes cluster.\r\n\r\n\r\n# Implementation\r\n\r\nThis is a simple Spring Boot - based project to build a k8s cluster of scalable apps to process messages from PubSub topic. \r\n\r\n## Further details\r\n\r\nThere is a topic - `postbox` -  where Bob can publish messages to, and a cluster of listeners (letter-reader app) subscribed to this topic. \r\nThese applications can pickup messages from the topic and process them.\r\n\r\nEach message has the following format:\r\n\r\n```\r\n{\r\n  \"id\": \"\u003cunique uuid\u003e\",\r\n  \"resent_attemtps\": 1\r\n}\r\n```\r\n\r\n- Field `id` is required and must be unique across all letters\r\n- Field `resent_attemtps` is optional and indicates the number of re-send attempts made for this message.\r\n\r\nThere is a 2nd topic with name `read_letters`, which is used to count the total number of processed messages. \r\nThe dashboard app is basically just an interface to this topic, and collects other useful statistics along the way.\r\nDashboard's api is available via Swagger at `https://localhost:9000/swagger-ui/`\r\n\r\n# How to build\r\n\r\nHere the sequence of steps to perform (most of these steps will be covered in next sections)\r\n\r\n* Create Docker images for `dashboard` and `letter-reader` apps\r\n* Upload the Docker images to a private Docker registry at Google using `docker push` command\r\n* Use Terraform to define infrastructure-as-a-code which includes a Google Cloud k8s instances and PubSub topics/subscriptions.\r\n* Deploy a Kubernetes cluster using Terraform to Google Cloud\r\n* Deploy one instance of the `letter-reader` Docker image to the Kubernetes cluster\r\n* Deploy one instance of the `dashboard` Docker image to the Kubernetes cluster\r\n* Scale out the core processing part of the cluster to 10 instances of the `letter-reader` application without any changes \r\n  to production java code\r\n\r\n\r\n# Requirements\r\n\r\n* gcloud (see [2])\r\n* Java 11 SDK\r\n* terraform (see [4])\r\n* Kubectl (Kubectl aka Kube, is a command line tool for controlling Kubernetes clusters)\r\n* ssh\r\n\r\n\r\n# Environment settings\r\n\r\nThe Spring Cloud GCP Core Boot starter can be auto-configured using properties from the properties file (`src/main/resources/application.yaml`) \r\nwhich always have precedence over the Spring Boot configuration.\r\n\r\nThe GCP project ID is auto-configured from the `GOOGLE_CLOUD_PROJECT` environment variable, among several other sources. \r\n\r\nThe OAuth2 credentials are auto-configured from the `GOOGLE_APPLICATION_CREDENTIALS` environment variable.\r\n\r\nThis var can be set manually after auto-generating json with google account credentials:\r\n\r\n```\r\ngcloud auth application-default login\r\n```\r\n\r\n\r\nThe path to gcloud creds usually has the form:\r\n\r\n```\r\n/$HOME/.config/gcloud/legacy_credentials/$EMAIL/adc.json\r\n```\r\n\r\nwhere variable $EMAIL can be obtained via command:\r\n\r\n```\r\ngcloud config list account --format \"value(core.account)\"\r\n```\r\n\r\nAdd GOOGLE_APPLICATION_CREDENTIALS as permanent vars into the file:\r\n\r\n```\r\nsudo -H gedit /etc/environment\r\n```\r\n\r\nTo access cloud instances the SSH key is needed, it can be created using the following command:\r\n\r\n```\r\nssh-keygen -t rsa -f ~/.ssh/pubsub_rsa -C $USERNAME -b 2048\r\n```\r\n\r\n# Settings on GCP side\r\n\r\n(0) For convenience and generalization, set the env variable GOOGLE_CLOUD_PROJECT in file set_env.sh to your project id, f.e. \r\n`export GOOGLE_CLOUD_PROJECT=message-multi-processor`\r\n\r\n(1) Create a project:\r\n\r\n```\r\ngcloud projects create $GOOGLE_CLOUD_PROJECT\r\n```\r\n\r\nAfter successful creation project_id must be visible via command `gcloud projects list`\r\n\r\n(2) Activate billing account for project and enable PubSub and GKE services\r\n\r\n(3) Build an image with app and push it to GCloud private docker registry, f.e. :\r\n\r\n```\r\nsudo docker build -t gcp-dashboard:0.0.1 ./dashboard/\r\nsudo docker build -t gcp-letter-reader:0.0.1 ./letter-reader/\r\n```\r\n\r\nDocker image can be tested with the help of command (`ctrl+shift+c` to stop):\r\n```\r\nsudo docker run -p 9000:9000 \\\r\n    --env=GOOGLE_CLOUD_PROJECT=$GOOGLE_CLOUD_PROJECT \\\r\n    --env=GOOGLE_APPLICATION_CREDENTIALS=/secrets/adc.json \\\r\n    --volume=$GOOGLE_APPLICATION_CREDENTIALS:/secrets/adc.json \\\r\n    gcp-dashboard:0.0.1\r\n```\r\n\r\n(4) Authenticate the docker registry (after update, the following will be written to your Docker config file located at  \r\n`/root/.docker/config.json`), and push the image:\r\n\r\n```\r\ncurl -fsSL \"https://github.com/GoogleCloudPlatform/docker-credential-gcr/releases/download/v${VERSION}/docker-credential-gcr_${OS}_${ARCH}-${VERSION}.tar.gz\" \u003e helper.tar.gz\r\ntar xz -f ./helper.tar.gz\r\nsudo mv ./docker-credential-gcr  /usr/local/bin/docker-credential-gcr\r\nsudo chmod +x /usr/local/bin/docker-credential-gcr\r\nsudo docker-credential-gcr configure-docker\r\n```\r\n\r\nthen tag the image and push it to the registry:\r\n\r\n```\r\nsudo docker tag gcp-dashboard:0.0.1 gcr.io/$GOOGLE_CLOUD_PROJECT/gcp-dashboard:v1\r\nsudo docker push gcr.io/$GOOGLE_CLOUD_PROJECT/gcp-dashboard:v1\r\n```\r\n\r\n(5) Verify the pulling docker image from GCP registry: test the image with the following command, \r\nwhich will run a Docker container as a daemon on port 9000 from your newly created container image:\r\n\r\n```\r\nsudo docker run -ti --rm -p 9000:9000 \\\r\n  --env=GOOGLE_CLOUD_PROJECT=$GOOGLE_CLOUD_PROJECT \\\r\n  --env=GOOGLE_APPLICATION_CREDENTIALS=/secrets/adc.json \\\r\n  --volume=$GOOGLE_APPLICATION_CREDENTIALS:/secrets/adc.json \\\r\n  gcr.io/$GOOGLE_CLOUD_PROJECT/gcp-dashboard:v1\r\n```\r\n(6) Generate temporary SSL certificates with the help of `cerbot` \r\n\r\n```\r\npython3 -m venv venv\r\nsource ./venv/bin/activate\r\npip install -r requirements.txt\r\npython3 cerbot \u003cargs\u003e\r\n```\r\n\r\nOR as an alternative create self-signed certificates with the help of openssl utility:\r\n\r\n```\r\nopenssl req -x509 -nodes -days 365 -newkey rsa:2048 \\\r\n        -keyout ./terraform/secrets/dashboard-private.key \\\r\n        -out ./terraform/secrets/dashboard-certificate.crt\r\n```\r\n\r\n(7) Create infrastructure using Terraform:\r\n\r\nFor Terraform it's necessary to set in file set_env.sh the following variables:\r\n\r\n```\r\nexport TF_VAR_google_app_creds=$GOOGLE_APPLICATION_CREDENTIALS\r\nexport TF_VAR_project_id=$GOOGLE_CLOUD_PROJECT\r\n```\r\n(8) Run terraform init to download the latest version of the provider and build the .terraform directory\r\n\r\n```\r\nterraform init\r\nterraform plan\r\nterraform apply\r\n```\r\n\r\nInstances will be available via command:\r\n\r\n```\r\ngcloud compute instances list\r\n```\r\n\r\nOne can login to them and inspect using SSH access and then see the startup logs, check the java version, etc:\r\n\r\n```\r\ngcloud beta compute ssh --zone \"us-west1-a\" \"proc-vm-d1c76370a95aa91c\"  --project $GOOGLE_CLOUD_PROJECT\r\ncat /var/log/syslog \r\njava --version\r\n```\r\nNote, instance ID (here it's `proc-vm-d1c76370a95aa91c`) may change after each recreation.\r\n\r\n# Kubernetes\r\n\r\nThis step must be performed after creating all resources at GCP.\r\nAuthorise your installation of kubectl to work with GKE cluster:\r\n\r\n```\r\ngcloud container clusters get-credentials letter-processing-cluster --zone=europe-west2\r\n```\r\nAs a result, the configuration entry will be generated and persisted in file `$HOME/.kube/config`\r\n\r\nNote: Running `gcloud container clusters get-credentials` also changes the current context for kubectl to that cluster, i.e. \r\nfrom this moment all k8s commands will be tied with  letter-processing-cluster.\r\nFor example, one can look up existing namespaces and pods:\r\n\r\n\r\n```\r\nkubectl get pods --all-namespaces\r\nkubectl describe hpa\r\n```\r\n\r\n\r\n\r\n## Clean up\r\n\r\n(1) First, destroy the resources created by Terraform: \r\n\r\n```\r\nterraform apply -destroy\r\n```\r\n\r\n## Set up and tune up a scaling group\r\n\r\nHere and next a recipe from [5] is applied\r\n\r\nIn this project we are using Custom Metrics Adapter:\r\n\r\n```\r\nhttps://cloud.google.com/kubernetes-engine/docs/tutorials/autoscaling-metrics#step1\r\n```\r\nIt's running as a separate service in isolated container, and is defined as `kubernetes_service_account.metrics_driver` resource in terraform.\r\n\r\nThe name of metric can be picked up from [metrics-explorer](https://console.cloud.google.com/monitoring/metrics-explorer)\r\n\r\nHere we are using `pubsub.googleapis.com|subscription|num_undelivered_messages`, see \r\nthe file `terraform/k8s/autoscaler.tf` for details\r\n\r\nCreated metrics are available via GKE -\u003e Workloads -\u003e letter-reader -\u003e Autoscaler\r\n\r\n\r\n# Running application locally\r\n\r\nBuild app using command `mvn clean package`, then use the command f.e. `java -jar ./letter-reader/target/letter-reader-0.0.1.jar` to run an app in a separate window.\r\n \r\n# Testing \r\n\r\nGenerate 50 messages using script:\r\n\r\n```\r\nscripts/message_generator.sh 50\r\n```\r\nObserve autoscaling of cluster in action using f.e. command `kubectl describe hpa`\r\n\r\n\r\n# References\r\n\r\n[1] https://github.com/spring-guides/gs-messaging-gcp-pubsub\r\n\r\n[2] https://cloud.google.com/sdk/docs/install#deb\r\n\r\n[3] https://cloud.google.com/sdk/docs/components\r\n\r\n[4] https://cloud.google.com/docs/terraform (an open-source infrastructure as code software tool created by HashiCorp)\r\n\r\n[5] https://cloud.google.com/kubernetes-engine/docs/tutorials/autoscaling-metrics#pubsub\r\n\r\n[6] https://www.gcpweekly.com/gcp-resources\r\n\r\n[7] https://cloud.google.com/compute/docs/instances/access-overview\r\n\r\n[8] https://kubernetes.io/docs/tasks/tools/install-kubectl/ (Kubectl is a command line tool for controlling Kubernetes clusters)\r\n\r\n[9] https://github.com/steinim/gcp-terraform-workshop\r\n\r\n\r\n# Appendix 1. Terraform installation on Ubuntu 20.04 LTS\r\n\r\n```\r\nsudo apt-get update \u0026\u0026 sudo apt-get install -y gnupg software-properties-common curl\r\ncurl -fsSL https://apt.releases.hashicorp.com/gpg | sudo apt-key add -\r\nsudo apt-add-repository \"deb [arch=amd64] https://apt.releases.hashicorp.com $(lsb_release -cs) main\"\r\nsudo apt-get update \u0026\u0026 sudo apt-get install terraform\r\n```\r\n\r\nCheck the valid installation:\r\n\r\n```\r\nterraform -help\r\n```\r\n\r\n# Appendix 2. Terraform basics\r\n\r\n*Providers*: a provider is responsible for understanding API interactions and exposing resources. \r\nProviders generally are an IaaS (e.g. AWS, GCP, Microsoft Azure, OpenStack), PaaS (e.g. Heroku), or SaaS services\r\n\r\n*Resources*: resources are the most important element in the Terraform language. \r\nEach resource block describes one or more infrastructure objects, such as virtual networks, compute instances\r\n\r\n*Variables*: a variable can have a default value. In case of omitted default values, Terraform will ask to provide it \r\nwhen running a terraform command\r\n\r\n*Modules*: a module is just a folder which combines related terraform files\r\n\r\n*Outputs*: sometimes a variable is needed which is only known after terraform has done a change on a cloud provider — \r\nf.e. ip-addresses that are given to application. So output serves as an intermediate holding agent - \r\nit takes that value and exposes it to your variables\r\n\r\nTerraform automatically resolves dependencies, but sometimes cannot determine the order due to complex or circular dependencies.\r\nIn this case successful deployment can be achieved by applying command `terraform apply` twice \r\n\r\n# Appendix 3. Docker commands\r\n\r\nThe list of all available images can be accessed using `sudo docker image list` command.\r\n\r\nRemoving tagged images:\r\n\r\nFirst, untag it, f.e.:\r\n\r\n```\r\nsudo docker image rm gcr.io/message-multi-processor/gcp-pubsub:v1\r\nsudo docker image rm gcr.io/message-multi-processor/gcp-pubsub:latest\r\n```\r\n\r\n# Appendix 4. Troubleshooting\r\n\r\n(1) This command can turn on services:\r\n\r\n```\r\ngcloud services enable \u003cservice\u003e\r\nf.e.\r\ngcloud services enable container\r\n```\r\n\r\n(2) Error `Node pools of f1-micro machines are not supported due to insufficient memory`: \r\n\r\nhttps://cloud.google.com/kubernetes-engine/docs/concepts/cluster-architecture#memory_cpu\r\n\r\nhttps://cloud.google.com/compute/docs/general-purpose-machines#e2-shared-core\r\n\r\nNote, for Docker-based java apps its better to choose `e2-small` at least \r\n\r\n(3) Boot time optimisation:\r\n\r\n```\r\no.s.boot.StartupInfoLogger : InetAddress.getLocalHost().getHostName() took 264 milliseconds to respond. Please verify your network configuration.\r\n```\r\n\r\n```\r\n[Channel\u003c7\u003e: (pubsub.googleapis.com:443)] Failed to resolve name. status=Status{code=UNAVAILABLE, description=Unable to resolve host pubsub.googleapis.com, \r\ncause=java.lang.RuntimeException: java.net.UnknownHostException: pubsub.googleapis.com \r\nat io.grpc.internal.DnsNameResolver.resolveAddresses(DnsNameResolver.java:223) \r\nat io.grpc.internal.DnsNameResolver.doResolve(DnsNameResolver.java:282)\r\n```\r\n\r\nSuch messages are solid hints that some configuration parameters (such as timeouts, etc) need tuning\r\nFor example, for liveness_probe.timeout_seconds it's better to set a min value starting from 5s (the default value in 1s is too small)\r\n\r\n(4) Autoscaling related\r\n\r\ncheck custom metrics driver was deployed and running:\r\n\r\n```\r\nkubectl describe hpa\r\n```\r\n\r\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fakaliutau%2Fgke-pubsub","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fakaliutau%2Fgke-pubsub","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fakaliutau%2Fgke-pubsub/lists"}