{"id":15222128,"url":"https://github.com/googlecloudplatform/gke-security-scenarios-demo","last_synced_at":"2025-10-25T14:31:05.652Z","repository":{"id":40272812,"uuid":"142068424","full_name":"GoogleCloudPlatform/gke-security-scenarios-demo","owner":"GoogleCloudPlatform","description":"This project demonstrates a series of best practices for improving the security of containerized applications deployed to Kubernetes Engine. You will deploy multiple instances of the same container image with a variety of security settings to illustrate the use of RBAC, security contexts, and AppArmor policies.  ","archived":false,"fork":false,"pushed_at":"2024-08-15T20:10:40.000Z","size":194,"stargazers_count":94,"open_issues_count":15,"forks_count":55,"subscribers_count":28,"default_branch":"master","last_synced_at":"2024-12-18T08:40:51.537Z","etag":null,"topics":["containers","gke","gke-helmsman","google-cloud-platform","kubernetes","kubernetes-engine","security"],"latest_commit_sha":null,"homepage":"","language":"HCL","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/GoogleCloudPlatform.png","metadata":{"files":{"readme":"README-QWIKLABS.md","changelog":null,"contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2018-07-23T20:55:26.000Z","updated_at":"2024-09-24T19:04:09.000Z","dependencies_parsed_at":"2024-01-13T10:40:08.616Z","dependency_job_id":"2e694557-9761-46bc-8990-be3f009aa313","html_url":"https://github.com/GoogleCloudPlatform/gke-security-scenarios-demo","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/GoogleCloudPlatform%2Fgke-security-scenarios-demo","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/GoogleCloudPlatform%2Fgke-security-scenarios-demo/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/GoogleCloudPlatform%2Fgke-security-scenarios-demo/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/GoogleCloudPlatform%2Fgke-security-scenarios-demo/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/GoogleCloudPlatform","download_url":"https://codeload.github.com/GoogleCloudPlatform/gke-security-scenarios-demo/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":238161514,"owners_count":19426669,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["containers","gke","gke-helmsman","google-cloud-platform","kubernetes","kubernetes-engine","security"],"created_at":"2024-09-28T15:10:44.689Z","updated_at":"2025-10-25T14:31:05.291Z","avatar_url":"https://github.com/GoogleCloudPlatform.png","language":"HCL","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Three examples of securing applications in Kubernetes Engine\n\n## Table of Contents\n\n\u003c!-- TOC --\u003e\n\n  * [Introduction](#introduction)\n  * [Architecture](#architecture)\n     * [Hardened Web Server (nginx)](#hardened-web-server-nginx)\n     * [System Daemonset (AppArmor Loader)](#system-daemonset-apparmor-loader)\n     * [Simplified Kubernetes Controller (Pod Labeler)](#simplified-kubernetes-controller-pod-labeler)\n  * [Initialize GCP](#initialize-gcp)\n  * [Deployment steps](#deployment-steps)\n     * [Setup this project](#setup-this-project)\n     * [Provisioning the Kubernetes Engine Cluster](#provisioning-the-kubernetes-engine-cluster)\n  * [Validation](#validation)\n  * [Set up Nginx](#set-up-nginx)\n  * [Set up AppArmor-loader](#set-up-apparmor-loader)\n  * [Set up Pod-labeler](#set-up-pod-labeler)\n  * [Tear down](#tear-down)\n  * [Troubleshooting](#troubleshooting)\n     * [The install script fails with a Permission denied when running Terraform](#the-install-script-fails-with-a-permission-denied-when-running-terraform)\n     * [Error during scp](#error-during-scp)\n     * [Invalid fingerprint error during Terraform operations](#invalid-fingerprint-error-during-terraform-operations)\n  * [Relevant Material](#relevant-material)\n\n\u003c!-- TOC --\u003e\n\n## Introduction\n\nThis tutorial demonstrates how Kubernetes Engine security features can be used to grant varying levels of privilege to applications, based on their particular requirements.\n\nWhen configuring security, applications should be granted the smallest set of privileges that still allows them to operate correctly. When applications have more privileges than they need then they are more dangerous when compromised. In a Kubernetes cluster, these privileges can be grouped into the following broad levels:\n\n* **Host access:** describes what permissions an application has on it's host node, outside of its container. This is controlled via Pod and Container security contexts, as well as app armor profiles.\n* **Network access:** describes what other resources or workloads an application can access via the network. This is controlled with NetworkPolicies.\n* **Kubernetes API access:** describes which API calls an application is allowed to make against. API access is controlled using the Role Based Access Control (RBAC) model via Role and RoleBinding definitions.\n\nIn the sections below, you will explore three scenarios with distinct security\nrequirements at each of these levels:\n\n| Scenario | Host Access | Network Access  | Kubernetes API Access |\n|---|---|---|---|\n| Hardened Web Server (NGINX) | Minimal | Must be able to serve port 80 | None |\n| Kubernetes Controller (Demo) | Minimal | Must be able to access the API Server | List and Patch Pods |\n| Privileged Daemonset (AppArmor Loader) | Privileged  | None | None |\n\n## Architecture\n\nThe tutorial uses three applications to illustrate the scenarios described above:\n\n### Hardened Web Server (nginx)\n\nCreates an nginx deployment whose pods have their host-level access restricted by an AppArmor profile and whose network connectivity is restricted by a NetworkPolicy.\n\n### System Daemonset (AppArmor Loader)\n\nCreates a daemonset responsible for loading (installing) the app armor profile applied to the nginx pods on each node. Loading profiles requires more privileges than can be provided via Capabilities, so it's containers are given full privileges via their SecurityContexts.\n\n### Simplified Kubernetes Controller (Pod Labeler)\n\nThe pod-labeler deployment creates a single pod that watches all other pods in the default namespace and periodically labels them. This requires access to the Kubernetes API server - which is configured via RBAC using a ServiceAccount, Role, and RoleMapping.\n\n## Initialize GCP\n\nWhen using Cloud Shell execute the following command in order to setup gcloud cli. When executing this command please setup your region\nand zone.\n\n```console\ngcloud init\n```\n\n## Deployment steps\n\nThe steps below will walk you through using terraform to deploy a Kubernetes Engine cluster that you will then use for installing a pod that uses an AppArmor profile and another pod that securely interacts with the Kubernetes API.\n\n### Setup this project\n\nThis project requires the following Google Cloud Service APIs to be enabled:\n\n* `compute.googleapis.com`\n* `container.googleapis.com`\n* `cloudbuild.googleapis.com`\n\nIn addition, the terraform configuration takes three parameters to determine where the Kubernetes Engine cluster should be created:\n\n* `project`\n* `region`\n* `zone`\n\nFor simplicity, these parameters are to be specified in a file named `terraform.tfvars`, in the `terraform` directory. To ensure the appropriate APIs are enabled and to generate the `terraform/terraform.tfvars` file based on your gcloud defaults, run:\n\n```console\nmake setup-project\n```\n\nThis will enable the necessary Service APIs, and it will also generate a `terraform/terraform.tfvars` file with the following keys. The values themselves will match the output of `gcloud config list`:\n\n```console\ncat terraform/terraform.tfvars\n\nproject=\"YOUR_PROJECT\"\nregion=\"YOUR_REGION\"\nzone=\"YOUR_ZONE\"\n```\n\nIf you need to override any of the defaults, simply replace the desired value(s) to the right of the equals sign(s). Be sure your replacement values are still double-quoted.\n\n### Provisioning the Kubernetes Engine Cluster\n\nNext, apply the terraform configuration and create the environment with:\n\n```console\n# From within the project root, use make to apply the terraform\nmake create\n```\n\n## Validation\n\nOnce complete, terraform will output a message indicating successful creation of the cluster.\n\n```console\n...snip...\ngoogle_container_cluster.primary: Still creating... (3m20s elapsed)\ngoogle_container_cluster.primary: Still creating... (3m30s elapsed)\ngoogle_container_cluster.primary: Creation complete after 3m33s (ID: gke-security-demo)\n\nApply complete! Resources: 7 added, 0 changed, 0 destroyed.\n```\n\nOnce the apply completes, ssh to the bastion host:\n\n```command\ngcloud compute ssh gke-tutorial-bastion\n```\n\nand execute:\n\n```command\nkubectl get pods --all-namespaces\n\nNAMESPACE     NAME                                                          READY     STATUS        RESTARTS   AGE\nkube-system   calico-node-g7mlz                                             2/2       Running       0          56s\nkube-system   calico-node-rzbbc                                             2/2       Running       0          56s\nkube-system   calico-node-vertical-autoscaler-8b959b949-tnwhm               1/1       Running       0          2m\nkube-system   calico-node-vnw26                                             2/2       Running       0          56s\nkube-system   calico-typha-dbf7c7b6d-bpm2r                                  1/1       Running       0          1m\nkube-system   calico-typha-horizontal-autoscaler-5545fbd5d6-4wbgg           1/1       Running       0          2m\nkube-system   calico-typha-vertical-autoscaler-54d8f88b84-2pwdz             1/1       Running       0          2m\nkube-system   event-exporter-v0.2.1-5f5b89fcc8-vb5bc                        2/2       Running       0          2m\nkube-system   fluentd-gcp-scaler-7c5db745fc-hcpr5                           1/1       Running       0          1m\nkube-system   fluentd-gcp-v3.0.0-ks89p                                      2/2       Running       0          1m\nkube-system   fluentd-gcp-v3.0.0-mfbg4                                      2/2       Running       0          1m\nkube-system   heapster-v1.5.3-7ff6c8d4cc-tbf8d                              3/3       Running       0          1m\nkube-system   ip-masq-agent-f8kdp                                           1/1       Running       0          1m\nkube-system   ip-masq-agent-h7fbh                                           1/1       Running       0          1m\nkube-system   ip-masq-agent-nlkl9                                           1/1       Running       0          1m\nkube-system   kube-dns-788979dc8f-c6zv4                                     4/4       Running       0          1m\nkube-system   kube-dns-788979dc8f-xt97h                                     4/4       Running       0          2m\nkube-system   kube-dns-autoscaler-79b4b844b9-cmhns                          1/1       Running       0          1m\nkube-system   kube-proxy-gke-gke-security-demo-default-pool-a55f1c91-6lxr   1/1       Running       0          1m\nkube-system   kube-proxy-gke-gke-security-demo-default-pool-a55f1c91-7smw   1/1       Running       0          1m\nkube-system   kube-proxy-gke-gke-security-demo-default-pool-a55f1c91-9khs   1/1       Running       0          1m\nkube-system   l7-default-backend-5d5b9874d5-57mb5                           1/1       Running       0          2m\nkube-system   metrics-server-v0.2.1-7486f5bd67-f68bq                        2/2       Running       0          1m\n```\n\n## Set up Nginx\n\nAgain on the bastion host, execute:\n\n```command\nkubectl apply -f manifests/nginx.yaml\n\ndeployment.apps/nginx created\nservice/nginx-lb created\nnetworkpolicy.networking.k8s.io/nginx-from-external created\n```\n\nThen execute:\n\n```command\nkubectl get pods\n\nNAME                     READY     STATUS    RESTARTS   AGE\nnginx-775877bc7b-5hk65   0/1       Blocked   0          13s\nnginx-775877bc7b-jwcv9   0/1       Blocked   0          13s\nnginx-775877bc7b-xc7p2   0/1       Blocked   0          13s\n```\n\nYou should see that while the pods have been created, they're in a blocked state. The nginx pods are blocked because the manifest includes an AppArmor profile that doesn't exist on the nodes:\n\n```yaml\n...\n    metadata:\n      annotations:\n        container.apparmor.security.beta.kubernetes.io/nginx: localhost/k8s-nginx\n...\n```\n\nWe can confirm this by executing the following command showing the `Message` field:\n\n```command\nkubectl describe pod -l app=nginx\n\n...snip...\nLabels:         app=nginx\n                pod-template-hash=3314336736\nAnnotations:    container.apparmor.security.beta.kubernetes.io/nginx=localhost/k8s-nginx\n                kubernetes.io/limit-ranger=LimitRanger plugin set: cpu request for container nginx\nStatus:         Pending\nReason:         AppArmor\nMessage:        Cannot enforce AppArmor: profile \"k8s-nginx\" is not loaded\nIP:\nControlled By:  ReplicaSet/nginx-775877bc7b\nContainers:\n  nginx:\n    Container ID:\n...snip...\n```\n\n## Set up AppArmor-loader\n\nIn order to resolve this, the relevant AppArmor profile must be loaded. because we don't know on which nodes the nginx pods will be allocated, we must deploy the AppArmor profile to all nodes. The way you'll deploy this, ensuring all nodes are covered, is via a daemonset\n\n[https://kubernetes.io/docs/concepts/workloads/controllers/daemonset/#what-is-a-daemonset](https://kubernetes.io/docs/concepts/workloads/controllers/daemonset/#what-is-a-daemonset).\n\nThe included apparmor-loader.yaml is the manifest that is highly privileged, as it needs to write a file onto each node. The contents of that file are in the configmap included in the same; in order to deploy it, execute:\n\n```console\n# On the bastion host\nkubectl apply -f manifests/apparmor-loader.yaml\n\nnamespace/apparmor created\nconfigmap/apparmor-profiles created\ndaemonset.apps/apparmor-loader created\nnetworkpolicy.networking.k8s.io/deny-apparmor-communication created\n```\n\nThe nginx nodes are in backoff by this point, so the fastest way to resolve it is by deleting the existing nginx pods and allowing kubernetes to re-create them:\n\n```console\n# On the bastion host\nkubectl delete pods -l app=nginx\n\npod \"nginx-775877bc7b-5hk65\" deleted\npod \"nginx-775877bc7b-jwcv9\" deleted\npod \"nginx-775877bc7b-xc7p2\" deleted\n```\n\nAfter which, we can confirm that nginx is running successfully with:\n\n```console\nkubectl get pods\n\nNAME                     READY     STATUS    RESTARTS   AGE\nnginx-775877bc7b-n2rxx   1/1       Running   0          28s\nnginx-775877bc7b-q7qfb   1/1       Running   0          28s\nnginx-775877bc7b-zzbdl   1/1       Running   0          28s\n```\n\nAnother test that will show the successful deployment of nginx is by executing:\n\n```console\nkubectl get services\n\nNAME         TYPE           CLUSTER-IP      EXTERNAL-IP     PORT(S)        AGE\nkubernetes   ClusterIP      10.137.0.1      \u003cnone\u003e          443/TCP        7m\nnginx-lb     LoadBalancer   10.137.15.122   [NGINX_IP]      80:30033/TCP   54s\n```\n\nthe external IP listed by the nginx-lb is publicly accessible and can reached with curl, wget, or a web browser. Navigating to the IP displayed in place of [NGINX_IP] above, you will see:\n\n![nginx browser](./img/nginx_success_browser.png)\n\n## Set up Pod-labeler\n\nIn this scenario you'll go through the process of deploying an application that requires access to the Kubernetes API. This pod will update the metadata of other pods.\n\nFirst execute:\n\n```console\nkubectl get pods --show-labels\n\nNAME                          READY     STATUS              RESTARTS   AGE       LABELS\nnginx-d99fc9b6c-ff6hw         1/1       Running             0          4m        app=nginx,pod-template-hash=855975627\nnginx-d99fc9b6c-m7gc2         1/1       Running             0          4m        app=nginx,pod-template-hash=855975627\nnginx-d99fc9b6c-zbkf9         1/1       Running             0          4m        app=nginx,pod-template-hash=855975627\n```\n\nNow execute:\n\n```console\nkubectl apply -f manifests/pod-labeler.yaml\n\nrole.rbac.authorization.k8s.io/pod-labeler created\nserviceaccount/pod-labeler created\nrolebinding.rbac.authorization.k8s.io/pod-labeler created\ndeployment.apps/pod-labeler created\n```\n\nThis can take up to two minutes to finish the creation of the new pod, but once this has finished, re-execute:\n\n```console\nkubectl get pods --show-labels\n\nNAME                          READY     STATUS    RESTARTS   AGE       LABELS\nnginx-d99fc9b6c-ff6hw         1/1       Running   0          5m        app=nginx,pod-template-hash=855975627,updated=1530656545.33\nnginx-d99fc9b6c-m7gc2         1/1       Running   0          5m        app=nginx,pod-template-hash=855975627,updated=1530656545.35\nnginx-d99fc9b6c-zbkf9         1/1       Running   0          5m        app=nginx,pod-template-hash=855975627,updated=1530656545.36\npod-labeler-8845f6488-px9hl   1/1       Running   0          55s       app=pod-labeler,pod-template-hash=440192044,updated=1530656545.38\n```\n\nAnd you'll see that the pods have an additional \"updated=...\" label.\n\n## Tear down\n\n\nLog out of the bastion host and run the following to destroy the environment (type `yes` at the prompt to confirm)\n\n```command\nmake teardown\n\n...snip...\nmodule.network.google_compute_subnetwork.cluster-subnet: Destruction complete after 26s\nmodule.network.google_compute_network.gke-network: Destroying... (ID: kube-net)\nmodule.network.google_compute_network.gke-network: Still destroying... (ID: kube-net, 10s elapsed)\nmodule.network.google_compute_network.gke-network: Still destroying... (ID: kube-net, 20s elapsed)\nmodule.network.google_compute_network.gke-network: Destruction complete after 25s\n\nDestroy complete! Resources: 7 destroyed.\n```\n\n## Troubleshooting\n\n### The install script fails with a `Permission denied` when running Terraform\n\nThe credentials that Terraform is using do not provide the necessary permissions to create resources in the selected projects. Ensure that the account listed in `gcloud config list` has necessary permissions to create resources. If it does, regenerate the application default credentials using `gcloud auth application-default login`.\n\n### Error during scp\n\nOccasionally, the gke-tutorial-bastion host will not be ready for the subsequent `scp` command, in this case there will be an error that looks like: ![terraform scp error](./img/bastion_scp_error.png) If this happens, simply re-run `make create`.\n\n### Invalid fingerprint error during Terraform operations\n\nTerraform occasionally complains about an invalid fingerprint, when updating certain resources. If you see the error below, simply re-run the command. ![terraform fingerprint error](./img/terraform_fingerprint_error.png)\n\n## Relevant Material\n\n* [AppArmor](https://kubernetes.io/docs/tutorials/clusters/apparmor/)\n* [Terraform - Kubernetes Engine](https://www.terraform.io/docs/providers/google/r/container_cluster.html)\n* [Bastion Host](https://en.wikipedia.org/wiki/Bastion_host)\n* [RBAC](https://kubernetes.io/docs/reference/access-authn-authz/rbac/)\n* [Securing Kubernetes Engine](https://cloud.google.com/kubernetes-engine/docs/how-to/hardening-your-cluster)\n* [Security Contexts](https://kubernetes.io/docs/tasks/configure-pod-container/security-context/)\n\n**This is not an officially supported Google product**\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fgooglecloudplatform%2Fgke-security-scenarios-demo","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fgooglecloudplatform%2Fgke-security-scenarios-demo","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fgooglecloudplatform%2Fgke-security-scenarios-demo/lists"}