{"id":23234974,"url":"https://github.com/googlecloudplatform/gke-application-security-demo","last_synced_at":"2025-07-26T07:36:39.342Z","repository":{"id":33274307,"uuid":"142068626","full_name":"GoogleCloudPlatform/gke-application-security-demo","owner":"GoogleCloudPlatform","description":"This project demonstrates a series of best practices for improving the security of containerized applications deployed to Kubernetes Engine. You will deploy multiple instances of the same container image with a variety of security settings to illustrate the use of RBAC, security contexts, and AppArmor policies.  ","archived":false,"fork":false,"pushed_at":"2024-08-19T19:11:10.000Z","size":122,"stargazers_count":71,"open_issues_count":5,"forks_count":29,"subscribers_count":32,"default_branch":"master","last_synced_at":"2025-04-11T23:18:14.252Z","etag":null,"topics":["containers","gke","gke-helmsman","kubernetes","kubernetes-engine","security"],"latest_commit_sha":null,"homepage":"","language":"HCL","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/GoogleCloudPlatform.png","metadata":{"files":{"readme":"README-QWIKLABS.md","changelog":null,"contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2018-07-23T20:57:44.000Z","updated_at":"2025-03-15T17:10:34.000Z","dependencies_parsed_at":"2024-12-26T04:07:22.329Z","dependency_job_id":"e4138668-ca10-4ab6-bf28-bb163cf239e1","html_url":"https://github.com/GoogleCloudPlatform/gke-application-security-demo","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/GoogleCloudPlatform%2Fgke-application-security-demo","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/GoogleCloudPlatform%2Fgke-application-security-demo/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/GoogleCloudPlatform%2Fgke-application-security-demo/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/GoogleCloudPlatform%2Fgke-application-security-demo/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/GoogleCloudPlatform","download_url":"https://codeload.github.com/GoogleCloudPlatform/gke-application-security-demo/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248492885,"owners_count":21113163,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["containers","gke","gke-helmsman","kubernetes","kubernetes-engine","security"],"created_at":"2024-12-19T03:17:16.780Z","updated_at":"2025-04-11T23:18:19.549Z","avatar_url":"https://github.com/GoogleCloudPlatform.png","language":"HCL","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Best Practices for Securing Containerized Applications in Kubernetes Engine\n\n## Table of Contents\n\n\u003c!-- TOC --\u003e\n\n* [Introduction](#introduction)\n* [Architecture](#architecture)\n  * [Containers](#containers)\n  * [AppArmor](#apparmor)\n  * [The container configurations](#the-container-configurations)\n* [Initial Setup](#initial-setup)\n  * [Configure gcloud](#configure-gcloud)\n  * [Tools](#tools)\n* [Deployment](#deployment)\n  * [Authenticate gcloud](#authenticate-gcloud)\n  * [Configure gcloud settings](#configure-gcloud-settings)\n  * [Setup this project](#setup-this-project)\n  * [Provisioning the Kubernetes Engine Cluster](#provisioning-the-kubernetes-engine-cluster)\n* [Validation](#validation)\n* [Tear Down](#tear-down)\n* [Troubleshooting](#troubleshooting)\n* [Relevant Material](#relevant-material)\n\n\u003c!-- TOC --\u003e\n\n## Introduction\n\nThis guide demonstrates a series of best practices that will allow the user to improve the security of their containerized applications deployed to Kubernetes Engine.\n\nThe [principle of least privilege]((https://en.wikipedia.org/wiki/Principle_of_least_privilege))\nis widely recognized as an important design consideration in enhancing the protection of critical systems from faults and malicious behavior. It suggests that every component must be able to access **only** the information and resources that are necessary for its legitimate purpose. This guide will go about showing the user how to improve a container's security by providing a systematic approach to effectively remove unnecessary privileges.\n\n## Architecture\n\n### Containers\n\nAt their core, containers help make implementing security best practices easier by providing the user with an easy interface to run processes in a chroot environment as an unprivileged user and removing all but the kernel [capabilities](http://man7.org/linux/man-pages/man7/capabilities.7.html) needed to run the application. By default, all containers are run in the root user namespace so running containers as a non-root user is important.\n\n### AppArmor\n\nOn occasion, an application will need to access a kernel resource that requires special privileges normally granted only to the root user. However, running the application as a user with root privileges is a bad solution as it provides the application with access to the entire system. Instead, the kernel provides a set of capabilities that can be granted to a process to allow it coarse-grained access to only the kernel resources it needs and nothing more.\n\nUsing kernel modules such as AppArmor, Kubernetes provides an easy interface to both run the containerized application as a non-root user in the process namespace and restrict the set of capabilities granted to the process.\n\n### The container configurations\n\nThis demonstration will deploy five containers in a private cluster:\n\n1. A container run as the root user in the container in the Dockerfile\n1. A container run as a user created in the container in the Dockerfile\n1. A container that Kubernetes started as a non-root user despite the Dockerfile not specifying it be run as a non-root user\n1. A container with a lenient AppArmor profile that allows all non-root permissions.\n1. A container with an AppArmor profile applied to disallow the `/proc/cpuinfo` endpoint from being properly read\n\nEach container will be exposed outside the clusters as an internal load balancer.\n\nThe containers themselves are running a simple Go web server with five endpoints. The endpoints differ in terms of the privileges they need to complete the request. A non-root user cannot read a file owned by root. The `nobody` user cannot read `/proc/cpuinfo` when that privilege is being blocked by AppArmor.\n\n1. An endpoint to get the container's hostname\n1. An endpoint to get the username, UID, and GID of identity running the server\n1. An endpoint to read a file owned by the `root` user\n1. An endpoint to read a file owned by the `nobody` user\n1. An endpoint to read the `/proc/cpuinfo` file\n\n## Initial Setup\n\n### Configure gcloud\n\nAll the tools for the demo are installed. When using Cloud Shell execute the following\ncommand in order to setup gcloud cli. When executing this command please setup your region and zone.\n\n```console\ngcloud init\n```\n\n### Tools\n1. [Terraform \u003e= 0.11.7](https://www.terraform.io/downloads.html)\n2. [Google Cloud SDK version \u003e= 204.0.0](https://cloud.google.com/sdk/docs/downloads-versioned-archives)\n3. [kubectl matching the latest GKE version](https://kubernetes.io/docs/tasks/tools/install-kubectl/)\n4. bash or bash compatible shell\n5. [GNU Make 3.x or later](https://www.gnu.org/software/make/)\n6. A Google Cloud Platform project where you have permission to create\n   networks\n\n#### Install Cloud SDK\nThe Google Cloud SDK is used to interact with your GCP resources.\n[Installation instructions](https://cloud.google.com/sdk/downloads) for multiple platforms are available online.\n\n#### Install kubectl CLI\n\nThe kubectl CLI is used to interteract with both Kubernetes Engine and kubernetes in general.\n[Installation instructions](https://cloud.google.com/kubernetes-engine/docs/quickstart)\nfor multiple platforms are available online.\n\n#### Install Terraform\n\nTerraform is used to automate the manipulation of cloud infrastructure. Its\n[installation instructions](https://www.terraform.io/intro/getting-started/install.html) are also available online.\n\n## Deployment\n\nThe steps below will walk you through using terraform to deploy a Kubernetes Engine cluster that you will then use for exploring multiple types of container security configurations.\n\n### Authenticate gcloud\n\nPrior to running this demo, ensure you have authenticated your gcloud client by running the following command:\n\n```console\ngcloud auth application-default login\n```\n\n### Configure gcloud settings\n\nRun `gcloud config list` and make sure that `compute/zone`, `compute/region` and `core/project` are populated with values that work for you. You can set their values with the following commands:\n\n```console\n# Where the region is us-east1\ngcloud config set compute/region us-east1\n\nUpdated property [compute/region].\n```\n\n```console\n# Where the zone inside the region is us-east1-c\ngcloud config set compute/zone us-east1-c\n\nUpdated property [compute/zone].\n```\n\n```console\n# Where the project name is my-project-name\ngcloud config set project my-project-name\n\nUpdated property [core/project].\n```\n\n### Setup this project\n\nThis project requires the following Google Cloud Service APIs to be enabled:\n\n* `compute.googleapis.com`\n* `container.googleapis.com`\n* `cloudbuild.googleapis.com`\n\nIn addition, the terraform configuration takes three parameters to determine where the Kubernetes Engine cluster should be created:\n\n* `project`\n* `region`\n* `zone`\n\nFor simplicity, these parameters are to be specified in a file named `terraform.tfvars`, in the `terraform` directory. To ensure the appropriate APIs are enabled and to generate the `terraform/terraform.tfvars` file based on your gcloud defaults, run:\n\n```console\nmake setup-project\n```\n\nThis will enable the necessary Service APIs, and it will also generate a `terraform/terraform.tfvars` file with the following keys. The values themselves will match the output of `gcloud config list`:\n\n```console\n$ cat terraform/terraform.tfvars\n\nproject=\"YOUR_PROJECT\"\nregion=\"YOUR_REGION\"\nzone=\"YOUR_ZONE\"\n```\n\nIf you need to override any of the defaults, simply replace the desired value(s) to the right of the equals sign(s). Be sure your replacement values are still double-quoted.\n\n### Provisioning the Kubernetes Engine Cluster\n\nNext, apply the terraform configuration with:\n\n```console\n# From within the project root, use make to apply the terraform\nmake tf-apply\n```\n\nThis will take a few minutes to complete.  The following is the last few lines of successful output.\n\n```console\n...snip...\ngoogle_container_cluster.primary: Still creating... (2m20s elapsed)\ngoogle_container_cluster.primary: Still creating... (2m30s elapsed)\ngoogle_container_cluster.primary: Still creating... (2m40s elapsed)\ngoogle_container_cluster.primary: Still creating... (2m50s elapsed)\ngoogle_container_cluster.primary: Still creating... (3m0s elapsed)\ngoogle_container_cluster.primary: Still creating... (3m10s elapsed)\ngoogle_container_cluster.primary: Still creating... (3m20s elapsed)\ngoogle_container_cluster.primary: Still creating... (3m30s elapsed)\ngoogle_container_cluster.primary: Still creating... (3m40s elapsed)\ngoogle_container_cluster.primary: Creation complete after 3m44s (ID: gke-security-best-practices)\n\nApply complete! Resources: 7 added, 0 changed, 0 destroyed.\n```\n\nOnce that has completed, remote into the bastion instance using SSH:\n\n```console\ngcloud compute ssh gke-application-security-bastion\n```\n\nApply the manifests for the cluster using the deployment script:\n\n```console\n./scripts/deploy.sh\n```\n\nThis will take a minute or two to complete.  The final output should be similar to:\n\n```console\nnamespace/apparmor created\nconfigmap/apparmor-profiles created\ndaemonset.apps/apparmor-loader created\ndeployment.apps/armored-hello-user created\nservice/armored-hello-user created\ndeployment.apps/armored-hello-denied created\nservice/armored-hello-denied created\ndeployment.apps/hello-override created\nservice/hello-override created\ndeployment.apps/hello-root created\nservice/hello-root created\ndeployment.apps/hello-user created\nservice/hello-user created\n\n...snip...\n\nService hello-root has not allocated an IP yet.\nService hello-root has not allocated an IP yet.\nService hello-root IP has been allocated\nService hello-user has not allocated an IP yet.\nService hello-user has not allocated an IP yet.\nService hello-user has not allocated an IP yet.\nService hello-user has not allocated an IP yet.\nService hello-user IP has been allocated\nService hello-override IP has been allocated\nService armored-hello-user IP has been allocated\nService armored-hello-denied IP has been allocated\n```\n\nAt this point, the environment should be completely set up.\n\n## Validation\n\nTo test all of the services in one command, run the validation script from the scripts directory of the bastion host:\n\n```console\n./scripts/validate.sh\n```\n\nThis script queries each of the services to get:\n\n* the hostname of the pod being queried\n* the username, UID, and GID of the process the pod's web server is running as\n* the contents of a file owned by root\n* the contents of a file owned by a non-root user\n* the first 5 lines of content from `/proc/cpuinfo`\n\nThe first service, `hello-root`, has an output similar to:\n\n```console\nQuerying service running natively as root\nYou are querying host hello-root-54fdf49bf7-8bjmm\nUser: root\nUID: 0\nGID: 0\nYou have read the root.txt file.\nYou have read the user.txt file.\nprocessor : 0\nvendor_id : GenuineIntel\ncpu family  : 6\nmodel   : 63\nmodel name  : Intel(R) Xeon(R) CPU @ 2.30GHz\n```\n\nand it clearly shows that it is running as `root` and can perform all actions.\n\nThe third service, `hello-user`, has an output similar to:\n\n```console\nQuerying service containers running natively as user\nYou are querying host hello-user-76957b5645-hvfw2\nUser: nobody\nUID: 65534\nGID: 65534\nunable to open root.txt: open root.txt: permission denied\nYou have read the user.txt file.\n  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current\n                                 Dload  Upload   Total   Spent    Left  Speed\n100   804  100   804    0     0   156k      0 --:--:-- --:--:-- --:--:--  196k\nprocessor       : 0\nvendor_id       : GenuineIntel\ncpu family      : 6\nmodel           : 45\nmodel name      : Intel(R) Xeon(R) CPU @ 2.60GHz\n```\n\nwhich shows that it is running as `nobody` (65534) and therefore can read user.txt but not root.txt.\n\nThe third service, `hello-override`, has an output similar to:\n\n```console\nQuerying service containers normally running as root but overridden by Kubernetes\nYou are querying host hello-override-7c6c4b6c4-szmrh\nUser: nobody\nUID: 65534\nGID: 65534\nunable to open root.txt: open root.txt: permission denied\nYou have read the user.txt file.\n  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current\n                                 Dload  Upload   Total   Spent    Left  Speed\n100   804  100   804    0     0   144k      0 --:--:-- --:--:-- --:--:--  157k\nprocessor       : 0\nvendor_id       : GenuineIntel\ncpu family      : 6\nmodel           : 45\nmodel name      : Intel(R) Xeon(R) CPU @ 2.60GHz\n```\n\nand it shows that the container is running as the `nobody` user of id `65534`.  Therefore, it can again read the `user.txt` file and read from `/proc/cpuinfo`.\n\nThe fourth service, `armored-hello-user`, has an output similar to:\n\n```console\nQuerying service containers with an AppArmor profile allowing reading /proc/cpuinfo\nYou are querying host armored-hello-user-5645cd4496-qls6q\nUser: nobody\nUID: 65534\nGID: 65534\nunable to open root.txt: open root.txt: permission denied\nYou have read the user.txt file.\n  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current\n                                 Dload  Upload   Total   Spent    Left  Speed\n100   804  100   804    0     0   148k      0 --:--:-- --:--:-- --:--:--  157k\nprocessor       : 0\nvendor_id       : GenuineIntel\ncpu family      : 6\nmodel           : 45\nmodel name      : Intel(R) Xeon(R) CPU @ 2.60GHz\n```\n\nand it shows that the leniently armored container still has the default access of the\n`nobody` user.\n\nThe fifth and final service, `armored-hello-denied`, has an output similar to:\n\n```console\nQuerying service containers with an AppArmor profile blocking the reading of /proc/cpuinfo\nYou are querying host armored-hello-denied-6fccb988dd-sxhmz\nUser: nobody\nUID: 65534\nGID: 65534\nunable to open root.txt: open root.txt: permission denied\nYou have read the user.txt file.\n  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current\n                                 Dload  Upload   Total   Spent    Left  Speed\n100    63  100    63    0     0  12162      0 --:--:-- --:--:-- --:--:-- 12600\nunable to open root.txt: open /proc/cpuinfo: permission denied\n```\n\nand it shows that the container is prohibited by AppArmor policy from reading the `user.txt` and `/proc/cpuinfo`.\n\n## Tear Down\n\nTo tear down the environment, use :\n\n```console\n./scripts/teardown.sh\n```\n\nIt's output should look like the following:\n\n```console\ndaemonset.apps \"apparmor-loader\" deleted\nconfigmap \"apparmor-profiles\" deleted\nnamespace \"apparmor\" deleted\ndeployment.apps \"armored-hello-user\" deleted\nservice \"armored-hello-user\" deleted\ndeployment.apps \"armored-hello-denied\" deleted\nservice \"armored-hello-denied\" deleted\ndeployment.apps \"hello-override\" deleted\nservice \"hello-override\" deleted\ndeployment.apps \"hello-root\" deleted\nservice \"hello-root\" deleted\ndeployment.apps \"hello-user\" deleted\nservice \"hello-user\" deleted\n```\n\nAfter that script completes, log out of the bastion host and run the following to destroy the environment:\n\n```console\nmake tf-destroy\n```\n\nTerraform will destroy the environment and indicate when it has completed:\n\n```console\n...snip...\nmodule.network.google_compute_subnetwork.cluster-subnet: Destroying... (ID: us-east1/kube-net-subnet)\ngoogle_service_account.admin: Destruction complete after 0s\nmodule.network.google_compute_subnetwork.cluster-subnet: Still destroying... (ID: us-east1/kube-net-subnet, 10s elapsed)\nmodule.network.google_compute_subnetwork.cluster-subnet: Still destroying... (ID: us-east1/kube-net-subnet, 20s elapsed)\nmodule.network.google_compute_subnetwork.cluster-subnet: Destruction complete after 25s\nmodule.network.google_compute_network.gke-network: Destroying... (ID: kube-net)\nmodule.network.google_compute_network.gke-network: Still destroying... (ID: kube-net, 10s elapsed)\nmodule.network.google_compute_network.gke-network: Still destroying... (ID: kube-net, 20s elapsed)\nmodule.network.google_compute_network.gke-network: Destruction complete after 25s\n\nDestroy complete! Resources: 7 destroyed.\n```\n\n## Troubleshooting\n\n### Terraform destroy does not finish cleanly. The error will look something like\n\n```console\nError: Error applying plan:\n\n1 error(s) occurred:\n\n* module.network.google_compute_network.gke-network (destroy): 1 error(s) occurred:\n\n* google_compute_network.gke-network: The network resource 'projects/seymourd-sandbox/global/networks/kube-net' is already being used by 'projects/seymourd-sandbox/global/firewalls/k8s-29e43f3a2accf594-node-hc'\n\n\nTerraform does not automatically rollback in the face of errors. Instead, your Terraform state file has been partially updated with any resources that successfully completed. Please address the error above and apply again to incrementally change your infrastructure.\n\n```\n\nSolution: the cluster does not always cleanly remove all of the GCP resources associated with a service before the cluster is deleted. You will need to manually clean up the remaining resources using either the Cloud Console or gcloud.\n\n### The install script fails with a `Permission denied` when running Terraform\n\nThe credentials that Terraform is using do not provide the necessary permissions to create resources in the selected projects. Ensure that the account listed in `gcloud config list` has necessary permissions to create resources. If it does, regenerate the application default credentials using `gcloud auth application-default login`.\n\n### Invalid fingerprint error during Terraform operations\n\nTerraform occasionally complains about an invalid fingerprint, when updating certain resources. If you see the error below, simply re-run the command. ![terraform fingerprint error](./img/terraform_fingerprint_error.png)\n\n## Relevant Material\n\n* [Capabilities documentation](http://man7.org/linux/man-pages/man7/capabilities.7.html)\n* [Docker security documentation](https://docs.docker.com/engine/security/security/)\n* [AppArmor](https://wiki.ubuntu.com/AppArmor)\n* [Kubernetes Engine Release Notes](https://cloud.google.com/kubernetes-engine/release-notes)\n\n**This is not an officially supported Google product**\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fgooglecloudplatform%2Fgke-application-security-demo","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fgooglecloudplatform%2Fgke-application-security-demo","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fgooglecloudplatform%2Fgke-application-security-demo/lists"}