{"id":17232277,"url":"https://github.com/snigdhasambitak/cks","last_synced_at":"2025-02-24T12:31:09.729Z","repository":{"id":135581388,"uuid":"585956367","full_name":"snigdhasambitak/cks","owner":"snigdhasambitak","description":"Practice questions for Certified Kubernetes Security Specialist (CKS) exam","archived":false,"fork":false,"pushed_at":"2024-05-08T11:56:45.000Z","size":1422,"stargazers_count":50,"open_issues_count":1,"forks_count":36,"subscribers_count":4,"default_branch":"main","last_synced_at":"2024-10-15T05:01:52.719Z","etag":null,"topics":["apparmor","audit-log","cks","falco","kube-bench","kubernetes","opa","runsc","trivy"],"latest_commit_sha":null,"homepage":"","language":null,"has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/snigdhasambitak.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2023-01-06T14:50:05.000Z","updated_at":"2024-10-14T17:58:16.000Z","dependencies_parsed_at":null,"dependency_job_id":"3587c1e2-2747-4c62-8725-c3a377dffec4","html_url":"https://github.com/snigdhasambitak/cks","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/snigdhasambitak%2Fcks","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/snigdhasambitak%2Fcks/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/snigdhasambitak%2Fcks/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/snigdhasambitak%2Fcks/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/snigdhasambitak","download_url":"https://codeload.github.com/snigdhasambitak/cks/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":240477636,"owners_count":19807712,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["apparmor","audit-log","cks","falco","kube-bench","kubernetes","opa","runsc","trivy"],"created_at":"2024-10-15T05:00:36.136Z","updated_at":"2025-02-24T12:31:09.720Z","avatar_url":"https://github.com/snigdhasambitak.png","language":null,"funding_links":[],"categories":["Others"],"sub_categories":[],"readme":"- [CKS Simulator Kubernetes 1.25](#cks-simulator-kubernetes-125)\n * [Pre Setup](#pre-setup)\n * [Question 1 | Contexts](#question-1--contexts)\n * [Question 2 | Runtime Security with Falco](#question-2--runtime-security-with-falco)\n * [Question 3 | Apiserver Security](#question-3--apiserver-security)\n * [Question 4 | Pod Security Standard](#question-4--pod-security-standard)\n * [Question 5 | CIS Benchmark](#question-5--cis-benchmark)\n * [Question 6 | Verify Platform Binaries](#question-6--verify-platform-binaries)\n * [Question 7 | Open Policy Agent](#question-7--open-policy-agent)\n * [Question 8 | Secure Kubernetes Dashboard](#question-8--secure-kubernetes-dashboard)\n * [Question 9 | AppArmor Profile](#question-9--apparmor-profile)\n * [Question 10 | Container Runtime Sandbox gVisor](#question-10--container-runtime-sandbox-gvisor)\n * [Question 11 | Secrets in ETCD](#question-11--secrets-in-etcd)\n * [Question 12 | Hack Secrets](#question-12---hack-secrets)\n * [Question 13 | Restrict access to Metadata Server](#question-13--restrict-access-to-metadata-server)\n * [Question 14 | Syscall Activity](#question-14--syscall-activity)\n * [Question 15 | Configure TLS on Ingress](#question-15--configure-tls-on-ingress)\n * [Question 16 | Docker Image Attack Surface](#question-16--docker-image-attack-surface)\n * [Question 17 | Audit Log Policy](#question-17--audit-log-policy)\n * [Question 18 | Investigate Break-in via Audit Log](#question-18--investigate-break-in-via-audit-log)\n * [Question 19 | Immutable Root FileSystem](#question-19--immutable-root-filesystem)\n * [Question 20 | Update Kubernetes](#question-20--update-kubernetes)\n * [Question 21 | Image Vulnerability Scanning](#question-21--image-vulnerability-scanning)\n * [Question 22 | Manual Static Security Analysis](#question-22--manual-static-security-analysis)\n- [CKS Simulator Preview Kubernetes 1.25](#cks-simulator-preview-kubernetes-125)\n * [Preview Question 1](#preview-question-1)\n - [Answer:](#answer--20)\n * [Part 1 - check existing RBAC rules](#part-1---check-existing-rbac-rules)\n * [Part 2 - create additional RBAC rules](#part-2---create-additional-rbac-rules)\n * [Preview Question 2](#preview-question-2)\n - [Answer:](#answer--21)\n * [Preview Question 3](#preview-question-3)\n - [Answer:](#answer--22)\n- [CKS Tips Kubernetes 1.25](#cks-tips-kubernetes-125)\n * [Knowledge](#knowledge)\n + [Pre-Knowledge](#pre-knowledge)\n + [Knowledge](#knowledge-1)\n + [Approach](#approach)\n + [Content](#content)\n- [CKS Exam Info](#cks-exam-info)\n * [Read the Curriculum](#read-the-curriculum)\n * [Read the Handbook](#read-the-handbook)\n * [Read the important tips](#read-the-important-tips)\n * [Read the FAQ](#read-the-faq)\n- [Kubernetes documentation](#kubernetes-documentation)\n- [CKS clusters](#cks-clusters)\n- [The Test Environment / Browser Terminal](#the-test-environment---browser-terminal)\n * [Laggin](#laggin)\n * [Kubectl autocompletion and commands](#kubectl-autocompletion-and-commands)\n * [Copy \u0026 Paste](#copy---paste)\n * [Percentages and Score](#percentages-and-score)\n * [Notepad \u0026 Skipping Questions](#notepad---skipping-questions)\n * [Contexts](#contexts)\n- [PSI Bridge](#psi-bridge)\n- [Browser Terminal Setup](#browser-terminal-setup)\n * [Minimal Setup](#minimal-setup)\n + [Alias](#alias)\n + [Vim](#vim)\n * [Optional Setup](#optional-setup)\n + [Fast dry-run output](#fast-dry-run-output)\n + [Fast pod delete](#fast-pod-delete)\n + [Persist bash settings](#persist-bash-settings)\n + [Alias Namespace](#alias-namespace)\n * [Be fast](#be-fast)\n * [Vim](#vim-1)\n + [toggle vim line numbers](#toggle-vim-line-numbers)\n + [copy\u0026paste](#copy-paste)\n + [Indent multiple lines](#indent-multiple-lines)\n * [Split terminal screen](#split-terminal-screen)\n\n\n# CKS Simulator Kubernetes 1.25\nhttps://killer.sh\n\n \n\n## Pre Setup\n\nOnce you've gained access to your terminal it might be wise to spend ~1 minute to setup your environment. You could set these:\n\n```sh\nalias k=kubectl # will already be pre-configured\n\nexport do=\"--dry-run=client -o yaml\" # k create deploy nginx --image=nginx $do\n\nexport now=\"--force --grace-period 0\" # k delete pod x $now\n\n```\n\nVim\nThe following settings will already be configured in your real exam environment in ~/.vimrc. But it can never hurt to be able to type these down:\n\n```sh\nset tabstop=2\nset expandtab\nset shiftwidth=2\n```\n\nMore setup suggestions are in the tips section.\n \n\n## Question 1 | Contexts\n\n#### Task weight: 1%\n\n\nYou have access to multiple clusters from your main terminal through kubectl contexts. Write all context names into /opt/course/1/contexts, one per line.\n\nFrom the kubeconfig extract the certificate of user restricted@infra-prod and write it decoded to /opt/course/1/cert.\n\n \n\n#### Answer:\n\nMaybe the fastest way is just to run:\n\n```sh\nk config get-contexts # copy by hand\n\nk config get-contexts -o name \u003e /opt/course/1/contexts\n```\n\nOr using jsonpath:\n\n```sh\nk config view -o jsonpath=\"{.contexts[*].name}\"\nk config view -o jsonpath=\"{.contexts[*].name}\" | tr \" \" \"\\n\" # new lines\nk config view -o jsonpath=\"{.contexts[*].name}\" | tr \" \" \"\\n\" \u003e /opt/course/1/contexts\n```\nThe content could then look like:\n\n```sh\n#/opt/course/1/contexts\ngianna@infra-prod\ninfra-prod\nrestricted@infra-prod\nworkload-prod\nworkload-stage\n```\nFor the certificate we could just run\n\n```\nk config view --raw\n```\nAnd copy it manually. Or we do:\n```sh\nk config view --raw -ojsonpath=\"{.users[2].user.client-certificate-data}\" | base64 -d \u003e /opt/course/1/cert\n```\n\nOr even:\n\n```sh\nk config view --raw -ojsonpath=\"{.users[?(.name == 'restricted@infra-prod')].user.client-certificate-data}\" | base64 -d \u003e /opt/course/1/cert\n# /opt/course/1/cert\n-----BEGIN CERTIFICATE-----\nMIIDHzCCAgegAwIBAgIQN5Qe/Rj/PhaqckEI23LPnjANBgkqhkiG9w0BAQsFADAV\nMRMwEQYDVQQDEwprdWJlcm5ldGVzMB4XDTIwMDkyNjIwNTUwNFoXDTIxMDkyNjIw\nNTUwNFowKjETMBEGA1UEChMKcmVzdHJpY3RlZDETMBEGA1UEAxMKcmVzdHJpY3Rl\nZDCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBAL/Jaf/QQdijyJTWIDij\nqa5p4oAh+xDBX3jR9R0G5DkmPU/FgXjxej3rTwHJbuxg7qjTuqQbf9Fb2AHcVtwH\ngUjC12ODUDE+nVtap+hCe8OLHZwH7BGFWWscgInZOZW2IATK/YdqyQL5OKpQpFkx\niAknVZmPa2DTZ8FoyRESboFSTZj6y+JVA7ot0pM09jnxswstal9GZLeqioqfFGY6\nYBO/Dg4DDsbKhqfUwJVT6Ur3ELsktZIMTRS5By4Xz18798eBiFAHvgJGq1TTwuPM\nEhBfwYwgYbalL8DSHeFrelLBKgciwUKjr1lolnnuc1vhkX1peV1J3xrf6o2KkyMc\nlY0CAwEAAaNWMFQwDgYDVR0PAQH/BAQDAgWgMBMGA1UdJQQMMAoGCCsGAQUFBwMC\nMAwGA1UdEwEB/wQCMAAwHwYDVR0jBBgwFoAUPrspZIWR7YMN8vT5DF3s/LvpxPQw\nDQYJKoZIhvcNAQELBQADggEBAIDq0Zt77gXI1s+uW46zBw4mIWgAlBLl2QqCuwmV\nkd86eH5bD0FCtWlb6vGdcKPdFccHh8Z6z2LjjLu6UoiGUdIJaALhbYNJiXXi/7cf\nM7sqNOxpxQ5X5hyvOBYD1W7d/EzPHV/lcbXPUDYFHNqBYs842LWSTlPQioDpupXp\nFFUQPxsenNXDa4TbmaRvnK2jka0yXcqdiXuIteZZovp/IgNkfmx2Ld4/Q+Xlnscf\nCFtWbjRa/0W/3EW/ghQ7xtC7bgcOHJesoiTZPCZ+dfKuUfH6d1qxgj6Jwt0HtyEf\nQTQSc66BdMLnw5DMObs4lXDo2YE6LvMrySdXm/S7img5YzU=\n-----END CERTIFICATE-----\n```\n\n \n\n## Question 2 | Runtime Security with Falco\n\n#### Task weight: 4%\n\n \nUse context: `kubectl config use-context workload-prod`\n\n \n\nFalco is installed with default configuration on node `cluster1-node1`. Connect using `ssh cluster1-node1`. Use it to:\n\nFind a Pod running image nginx which creates unwanted package management processes inside its container.\nFind a Pod running image httpd which modifies `/etc/passwd`.\nSave the Falco logs for case 1 under `/opt/course/2/falco.log` in format:\n\n`time-with-nanosconds,container-id,container-name,user-name`\n\nNo other information should be in any line. Collect the logs for at least 30 seconds.\n\nAfterwards remove the threads (both 1 and 2) by scaling the replicas of the Deployments that control the offending Pods down to 0.\n\n \n\nAnswer:\nFalco, the open-source cloud-native runtime security project, is the de facto Kubernetes threat detection engine.\n\nNOTE: Other tools you might have to be familar with are sysdig or tracee\n\n \n\nUse Falco as service\n\nFirst we can investigate Falco config a little:\n\n```sh\n➜ ssh cluster1-node1\n\n➜ root@cluster1-node1:~# service falco status\n● falco.service - LSB: Falco syscall activity monitoring agent\n Loaded: loaded (/etc/init.d/falco; generated)\n Active: active (running) since Sat 2020-10-10 06:36:15 UTC; 2h 1min ago\n...\n\n➜ root@cluster1-node1:~# cd /etc/falco\n\n➜ root@cluster1-node1:/etc/falco# ls\n\n\nfalco.yaml falco_rules.local.yaml falco_rules.yaml k8s_audit_rules.yaml rules.available rules.d\n```\n\nThis is the default configuration, if we look into falco.yaml we can see:\n\n```sh\n\n# /etc/falco/falco.yaml\n\n...\n# Where security notifications should go.\n# Multiple outputs can be enabled.\n\nsyslog_output:\n enabled: true\n...\n```\n\nThis means that Falco is writing into syslog, hence we can do:\n\n```sh\n\n➜ root@cluster1-node1:~# cat /var/log/syslog | grep falco\nSep 15 08:44:04 ubuntu2004 falco: Falco version 0.29.1 (driver version 17f5df52a7d9ed6bb12d3b1768460def8439936d)\nSep 15 08:44:04 ubuntu2004 falco: Falco initialized with configuration file /etc/falco/falco.yaml\nSep 15 08:44:04 ubuntu2004 falco: Loading rules from file /etc/falco/falco_rules.yaml:\n...\n```\n\nYep, quite some action going on in there. Let's investigate the first offending Pod:\n\n```sh\n➜ root@cluster1-node1:~# cat /var/log/syslog | grep falco | grep nginx | grep process\nSep 16 06:23:47 ubuntu2004 falco: 06:23:47.376241377: Error Package management process launched in container (user=root user_loginuid=-1 command=apk container_id=7a5ea6a080d1 container_name=nginx image=docker.io/library/nginx:1.19.2-alpine)\n...\n\n➜ root@cluster1-node1:~# crictl ps -id 7a5ea6a080d1\nCONTAINER ID IMAGE NAME ... POD ID\n7a5ea6a080d1b 6f715d38cfe0e nginx ... 7a864406b9794\n\nroot@cluster1-node1:~# crictl pods -id 7a864406b9794\nPOD ID ... NAME NAMESPACE ...\n7a864406b9794 ... webapi-6cfddcd6f4-ftxg4 team-blue ...\n```\n\nFirst Pod is webapi-6cfddcd6f4-ftxg4 in Namespace team-blue.\n\n```sh\n➜ root@cluster1-node1:~# cat /var/log/syslog | grep falco | grep httpd | grep passwd\nSep 16 06:23:48 ubuntu2004 falco: 06:23:48.830962378: Error File below /etc opened for writing (user=root user_loginuid=-1 command=sed -i $d /etc/passwd parent=sh pcmdline=sh -c echo hacker \u003e\u003e /etc/passwd; sed -i '$d' /etc/passwd; true file=/etc/passwdngFmAl program=sed gparent=\u003cNA\u003e ggparent=\u003cNA\u003e gggparent=\u003cNA\u003e container_id=b1339d5cc2de image=docker.io/library/httpd)\n\n➜ root@cluster1-node1:~# crictl ps -id b1339d5cc2de\nCONTAINER ID IMAGE NAME ... POD ID\nb1339d5cc2dee f6b40f9f8ad71 httpd ... 595af943c3245\n\nroot@cluster1-node1:~# crictl pods -id 595af943c3245\nPOD ID ... NAME NAMESPACE ...\n595af943c3245 ... rating-service-68cbdf7b7-v2p6g team-purple ...\n```\n\nSecond Pod is rating-service-68cbdf7b7-v2p6g in Namespace team-purple.\n\n\nEliminate offending Pods\n\nThe logs from before should allow us to find and \"eliminate\" the offending Pods:\n\n```sh\n➜ k get pod -A | grep webapi\nteam-blue webapi-6cfddcd6f4-ftxg4 1/1 Running \n\n➜ k -n team-blue scale deploy webapi --replicas 0\ndeployment.apps/webapi scaled\n\n➜ k get pod -A | grep rating-service\nteam-purple rating-service-68cbdf7b7-v2p6g 1/1 Running\n\n➜ k -n team-purple scale deploy rating-service --replicas 0\ndeployment.apps/rating-service scaled\n```\n \n\n#### Use Falco from command line\n\nWe can also use Falco directly from command line, but only if the service is disabled:\n\n```sh\n\n➜ root@cluster1-node1:~# service falco stop\n\n➜ root@cluster1-node1:~# falco\nThu Sep 16 06:33:11 2021: Falco version 0.29.1 (driver version 17f5df52a7d9ed6bb12d3b1768460def8439936d)\nThu Sep 16 06:33:11 2021: Falco initialized with configuration file /etc/falco/falco.yaml\nThu Sep 16 06:33:11 2021: Loading rules from file /etc/falco/falco_rules.yaml:\nThu Sep 16 06:33:11 2021: Loading rules from file /etc/falco/falco_rules.local.yaml:\nThu Sep 16 06:33:11 2021: Loading rules from file /etc/falco/k8s_audit_rules.yaml:\nThu Sep 16 06:33:12 2021: Starting internal webserver, listening on port 8765\n06:33:17.382603204: Error Package management process launched in container (user=root user_loginuid=-1 command=apk container_id=7a5ea6a080d1 container_name=nginx image=docker.io/library/nginx:1.19.2-alpine)\n...\n```\n\nWe can see that rule files are loaded and logs printed afterwards.\n\n\n#### Create logs in correct format\n\nThe task requires us to store logs for \"unwanted package management processes\" in format time,container-id,container-name,user-name. The output from falco shows entries for \"Error Package management process launched\" in a default format. Let's find the proper file that contains the rule and change it:\n\n```sh\n➜ root@cluster1-node1:~# cd /etc/falco/\n\n➜ root@cluster1-node1:/etc/falco# grep -r \"Package management process launched\" .\n./falco_rules.yaml: Package management process launched in container (user=%user.name user_loginuid=%user.loginuid\n\n➜ root@cluster1-node1:/etc/falco# cp falco_rules.yaml falco_rules.yaml_ori\n\n➜ root@cluster1-node1:/etc/falco# vim falco_rules.yaml\n```\n\nFind the rule which looks like this:\n\n```yaml\n# Container is supposed to be immutable. Package management should be done in building the image.\n- rule: Launch Package Management Process in Container\n desc: Package management process ran inside container\n condition: \u003e\n spawned_process\n and container\n and user.name != \"_apt\"\n and package_mgmt_procs\n and not package_mgmt_ancestor_procs\n and not user_known_package_manager_in_container\n output: \u003e\n Package management process launched in container (user=%user.name user_loginuid=%user.loginuid\n command=%proc.cmdline container_id=%container.id container_name=%container.name image=%container.image.repository:%container.image.tag)\n priority: ERROR\n tags: [process, mitre_persistence]\n \n``` \nShould be changed into the required format:\n\n```yaml\n# Container is supposed to be immutable. Package management should be done in building the image.\n- rule: Launch Package Management Process in Container\n desc: Package management process ran inside container\n condition: \u003e\n spawned_process\n and container\n and user.name != \"_apt\"\n and package_mgmt_procs\n and not package_mgmt_ancestor_procs\n and not user_known_package_manager_in_container\n output: \u003e\n Package management process launched in container %evt.time,%container.id,%container.name,%user.name\n priority: ERROR\n tags: [process, mitre_persistence]\n```\n\nFor all available fields we can check https://falco.org/docs/rules/supported-fields, which should be allowed to open during the exam.\n\nNext we check the logs in our adjusted format:\n\n```sh\n➜ root@cluster1-node1:/etc/falco# falco | grep \"Package management\"\n\n06:38:28.077150666: Error Package management process launched in container 06:38:28.077150666,090aad374a0a,nginx,root\n06:38:33.058263010: Error Package management process launched in container 06:38:33.058263010,090aad374a0a,nginx,root\n06:38:38.068693625: Error Package management process launched in container 06:38:38.068693625,090aad374a0a,nginx,root\n06:38:43.066159360: Error Package management process launched in container 06:38:43.066159360,090aad374a0a,nginx,root\n06:38:48.059792139: Error Package management process launched in container 06:38:48.059792139,090aad374a0a,nginx,root\n06:38:53.063328933: Error Package management process launched in container 06:38:53.063328933,090aad374a0a,nginx,root\n```\n\nThis looks much better. Copy\u0026paste the output into file /opt/course/2/falco.log on your main terminal. The content should be cleaned like this:\n\n```sh\n# /opt/course/2/falco.log\n06:38:28.077150666,090aad374a0a,nginx,root\n06:38:33.058263010,090aad374a0a,nginx,root\n06:38:38.068693625,090aad374a0a,nginx,root\n06:38:43.066159360,090aad374a0a,nginx,root\n06:38:48.059792139,090aad374a0a,nginx,root\n06:38:53.063328933,090aad374a0a,nginx,root\n06:38:58.070912841,090aad374a0a,nginx,root\n06:39:03.069592140,090aad374a0a,nginx,root\n06:39:08.064805371,090aad374a0a,nginx,root\n06:39:13.078109098,090aad374a0a,nginx,root\n06:39:18.065077287,090aad374a0a,nginx,root\n06:39:23.061012151,090aad374a0a,nginx,root\n```\n\nFor a few entries it should be fast to just clean it up manually. If there are larger amounts of entries we could do:\n\n```sh\ncat /opt/course/2/falco.log.dirty | cut -d\" \" -f 9 \u003e /opt/course/2/falco.log\n```\nThe tool cut will split input into fields using space as the delimiter (-d\"\"). We then only select the 9th field using -f 9.\n\n\n#### Local falco rules\n\nThere is also a file /etc/falco/falco_rules.local.yaml in which we can override existing default rules. This is a much cleaner solution for production. Choose the faster way for you in the exam if nothing is specified in the task.\n\n \n\n## Question 3 | Apiserver Security\n\n#### Task weight: 3%\n\n \n\nUse context: `kubectl config use-context workload-prod`\n\n \n\nYou received a list from the DevSecOps team which performed a security investigation of the k8s cluster1 (workload-prod). The list states the following about the apiserver setup:\n\n* Accessible through a NodePort Service\n\nChange the apiserver setup so that:\n\n* Only accessible through a ClusterIP Service\n \n\n#### Answer:\n\nIn order to modify the parameters for the apiserver, we first ssh into the master node and check which parameters the apiserver process is running with:\n\n```sh\n➜ ssh cluster1-controlplane1\n\n➜ root@cluster1-controlplane1:~# ps aux | grep kube-apiserver\nroot 13534 8.6 18.1 1099208 370684 ? Ssl 19:55 8:40 kube-apiserver --advertise-address=192.168.100.11 --allow-privileged=true --anonymous-auth=true --authorization-mode=Node,RBAC --client-ca-file=/etc/kubernetes/pki/ca.crt --enable-admission-plugins=NodeRestriction --enable-bootstrap-token-auth=true --etcd-cafile=/etc/kubernetes/pki/etcd/ca.crt --etcd-certfile=/etc/kubernetes/pki/apiserver-etcd-client.crt --etcd-keyfile=/etc/kubernetes/pki/apiserver-etcd-client.key --etcd-servers=https://127.0.0.1:2379 --kubelet-client-certificate=/etc/kubernetes/pki/apiserver-kubelet-client.crt --kubelet-client-key=/etc/kubernetes/pki/apiserver-kubelet-client.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --kubernetes-service-node-port=31000 --proxy-client-cert-file=/etc/kubernetes/pki/front-proxy-client.crt --proxy-client-key-\n...\n```\nWe may notice the following argument:\n\n```sh\n--kubernetes-service-node-port=31000\n```\nWe can also check the Service and see its of type NodePort:\n\n```sh\n➜ root@cluster1-controlplane1:~# kubectl get svc\nNAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE\nkubernetes NodePort 10.96.0.1 \u003cnone\u003e 443:31000/TCP 5d2h\n```\nThe apiserver runs as a static Pod, so we can edit the manifest. But before we do this we also create a copy in case we mess things up:\n\n```sh\n➜ root@cluster1-controlplane1:~# cp /etc/kubernetes/manifests/kube-apiserver.yaml ~/3_kube-apiserver.yaml\n\n➜ root@cluster1-controlplane1:~# vim /etc/kubernetes/manifests/kube-apiserver.yaml\n```\n\nWe should remove the unsecure settings:\n\n```yaml\n# /etc/kubernetes/manifests/kube-apiserver.yaml\napiVersion: v1\nkind: Pod\nmetadata:\n annotations:\n kubeadm.kubernetes.io/kube-apiserver.advertise-address.endpoint: 192.168.100.11:6443\n creationTimestamp: null\n labels:\n component: kube-apiserver\n tier: control-plane\n name: kube-apiserver\n namespace: kube-system\nspec:\n containers:\n - command:\n - kube-apiserver\n - --advertise-address=192.168.100.11\n - --allow-privileged=true\n - --authorization-mode=Node,RBAC\n - --client-ca-file=/etc/kubernetes/pki/ca.crt\n - --enable-admission-plugins=NodeRestriction\n - --enable-bootstrap-token-auth=true\n - --etcd-cafile=/etc/kubernetes/pki/etcd/ca.crt\n - --etcd-certfile=/etc/kubernetes/pki/apiserver-etcd-client.crt\n - --etcd-keyfile=/etc/kubernetes/pki/apiserver-etcd-client.key\n - --etcd-servers=https://127.0.0.1:2379\n - --kubelet-client-certificate=/etc/kubernetes/pki/apiserver-kubelet-client.crt\n - --kubelet-client-key=/etc/kubernetes/pki/apiserver-kubelet-client.key\n - --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname\n# - --kubernetes-service-node-port=31000 # delete or set to 0\n - --proxy-client-cert-file=/etc/kubernetes/pki/front-proxy-client.crt\n - --proxy-client-key-file=/etc/kubernetes/pki/front-proxy-client.key\n...\n```\n\nOnce the changes are made, give the apiserver some time to start up again. Check the apiserver's Pod status and the process parameters:\n\n```sh\n➜ root@cluster1-controlplane1:~# kubectl -n kube-system get pod | grep apiserver\nkube-apiserver-cluster1-controlplane1 1/1 Running 0 38s\n\n➜ root@cluster1-controlplane1:~# ps aux | grep kube-apiserver | grep node-port\nThe apiserver got restarted without the unsecure settings. However, the Service kubernetes will still be of type NodePort:\n\n➜ root@cluster1-controlplane1:~# kubectl get svc\nNAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE\nkubernetes NodePort 10.96.0.1 \u003cnone\u003e 443:31000/TCP 5d3h\nWe need to delete the Service for the changes to take effect:\n\n➜ root@cluster1-controlplane1:~# kubectl delete svc kubernetes\nservice \"kubernetes\" deleted\n```\n\nAfter a few seconds:\n\n```sh\n➜ root@cluster1-controlplane1:~# kubectl get svc\nNAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE\nkubernetes ClusterIP 10.96.0.1 \u003cnone\u003e 443/TCP 6s\n```\n\nThis should satisfy the DevSecOps team.\n\n \n \n\n## Question 4 | Pod Security Standard\n\n#### Task weight: 8%\n\n\nUse context: `kubectl config use-context workload-prod`\n\n \n\nThere is Deployment `container-host-hacker` in Namespace `team-red` which mounts `/run/containerd` as a hostPath volume on the Node where its running. This means that the Pod can access various data about other containers running on the same Node.\n\nTo prevent this configure Namespace `team-red` to enforce the `baseline` Pod Security Standard. Once completed, delete the Pod of the Deployment mentioned above.\n\nCheck the ReplicaSet events and write the event/log lines containing the reason why the Pod isn't recreated into `/opt/course/4/logs`.\n\n \n\n#### Answer:\n\nMaking Namespaces use Pod Security Standards works via labels. We can simply edit it:\n```sh\nk edit ns team-red\n```\n\nNow we configure the requested label:\n\n```yaml\n# kubectl edit namespace team-red\napiVersion: v1\nkind: Namespace\nmetadata:\n labels:\n kubernetes.io/metadata.name: team-red\n pod-security.kubernetes.io/enforce: baseline # add\n name: team-red\n...\n```\n\nThis should already be enough for the default Pod Security Admission Controller to pick up on that change. Let's test it and delete the Pod to see if it'll be recreated or fails, it should fail!\n\n```sh\n➜ k -n team-red get pod\nNAME READY STATUS RESTARTS AGE\ncontainer-host-hacker-dbf989777-wm8fc 1/1 Running 0 115s\n\n➜ k -n team-red delete pod container-host-hacker-dbf989777-wm8fc \npod \"container-host-hacker-dbf989777-wm8fc\" deleted\n\n➜ k -n team-red get pod\nNo resources found in team-red namespace.\n```\n\nUsually the ReplicaSet of a Deployment would recreate the Pod if deleted, here we see this doesn't happen. Let's check why:\n\n```sh\n➜ k -n team-red get rs\nNAME DESIRED CURRENT READY AGE\ncontainer-host-hacker-dbf989777 1 0 0 5m25s\n\n➜ k -n team-red describe rs container-host-hacker-dbf989777\nName: container-host-hacker-dbf989777\nNamespace: team-red\n...\nEvents:\n Type Reason Age From Message\n ---- ------ ---- ---- -------\n...\n Warning FailedCreate 2m41s replicaset-controller Error creating: pods \"container-host-hacker-dbf989777-bjwgv\" is forbidden: violates PodSecurity \"baseline:latest\": hostPath volumes (volume \"containerdata\")\n Warning FailedCreate 2m2s (x9 over 2m40s) replicaset-controller (combined from similar events): Error creating: pods \"container-host-hacker-dbf989777-kjfpn\" is forbidden: violates PodSecurity \"baseline:latest\": hostPath volumes (volume \"containerdata\")\n ```\nThere we go! Finally we write the reason into the requested file so that Mr Scoring will be happy too!\n\n```sh\n# /opt/course/4/logs\nWarning FailedCreate 2m2s (x9 over 2m40s) replicaset-controller (combined from similar events): Error creating: pods \"container-host-hacker-dbf989777-kjfpn\" is forbidden: violates PodSecurity \"baseline:latest\": hostPath volumes (volume \"containerdata\")\nPod Security Standards can give a great base level of security! But when one finds themselves wanting to deeper adjust the levels like baseline or restricted... this isn't possible and 3rd party solutions like OPA could be looked at.\n```\n\n \n\n## Question 5 | CIS Benchmark\n\n#### Task weight: 3%\n\n \n\nUse context: `kubectl config use-context infra-prod`\n\n \n\nYou're ask to evaluate specific settings of `cluster2` against the CIS Benchmark recommendations. Use the tool kube-bench which is already installed on the nodes.\n\nConnect using `ssh cluster2-controlplane1` and `ssh cluster2-node1`.\n\nOn the master node ensure (correct if necessary) that the CIS recommendations are set for:\n\n* The --profiling argument of the kube-controller-manager\n\n* The ownership of directory /var/lib/etcd\n\nOn the worker node ensure (correct if necessary) that the CIS recommendations are set for:\n\n* The permissions of the kubelet configuration /var/lib/kubelet/config.yaml\n* The --client-ca-file argument of the kubelet\n \n\n#### Answer:\n\n##### Number 1\n\nFirst we ssh into the master node run kube-bench against the master components:\n\n```sh\n➜ ssh cluster2-controlplane1\n\n➜ root@cluster2-controlplane1:~# kube-bench run --targets=master\n...\n== Summary ==\n41 checks PASS\n13 checks FAIL\n11 checks WARN\n0 checks INFO\n```\n\nWe see some passes, fails and warnings. Let's check the required task (1) of the controller manager:\n\n```sh\n➜ root@cluster2-controlplane1:~# kube-bench run --targets=master | grep kube-controller -A 3\n1.3.1 Edit the Controller Manager pod specification file /etc/kubernetes/manifests/kube-controller-manager.yaml\non the master node and set the --terminated-pod-gc-threshold to an appropriate threshold,\nfor example:\n--terminated-pod-gc-threshold=10\n--\n1.3.2 Edit the Controller Manager pod specification file /etc/kubernetes/manifests/kube-controller-manager.yaml\non the master node and set the below parameter.\n--profiling=false\n\n1.3.6 Edit the Controller Manager pod specification file /etc/kubernetes/manifests/kube-controller-manager.yaml\non the master node and set the --feature-gates parameter to include RotateKubeletServerCertificate=true.\n--feature-gates=RotateKubeletServerCertificate=true\n```\n\nThere we see 1.3.2 which suggests to set --profiling=false, so we obey:\n\n```sh\n➜ root@cluster2-controlplane1:~# vim /etc/kubernetes/manifests/kube-controller-manager.yaml\n```\nEdit the corresponding line:\n\n```yaml\n# /etc/kubernetes/manifests/kube-controller-manager.yaml\napiVersion: v1\nkind: Pod\nmetadata:\n creationTimestamp: null\n labels:\n component: kube-controller-manager\n tier: control-plane\n name: kube-controller-manager\n namespace: kube-system\nspec:\n containers:\n - command:\n - kube-controller-manager\n - --allocate-node-cidrs=true\n - --authentication-kubeconfig=/etc/kubernetes/controller-manager.conf\n - --authorization-kubeconfig=/etc/kubernetes/controller-manager.conf\n - --bind-address=127.0.0.1\n - --client-ca-file=/etc/kubernetes/pki/ca.crt\n - --cluster-cidr=10.244.0.0/16\n - --cluster-name=kubernetes\n - --cluster-signing-cert-file=/etc/kubernetes/pki/ca.crt\n - --cluster-signing-key-file=/etc/kubernetes/pki/ca.key\n - --controllers=*,bootstrapsigner,tokencleaner\n - --kubeconfig=/etc/kubernetes/controller-manager.conf\n - --leader-elect=true\n - --node-cidr-mask-size=24\n - --port=0\n - --requestheader-client-ca-file=/etc/kubernetes/pki/front-proxy-ca.crt\n - --root-ca-file=/etc/kubernetes/pki/ca.crt\n - --service-account-private-key-file=/etc/kubernetes/pki/sa.key\n - --service-cluster-ip-range=10.96.0.0/12\n - --use-service-account-credentials=true\n - --profiling=false # add\n...\n```\n\nWe wait for the Pod to restart, then run kube-bench again to check if the problem was solved:\n\n```sh\n➜ root@cluster2-controlplane1:~# kube-bench run --targets=master | grep kube-controller -A 3\n1.3.1 Edit the Controller Manager pod specification file /etc/kubernetes/manifests/kube-controller-manager.yaml\non the master node and set the --terminated-pod-gc-threshold to an appropriate threshold,\nfor example:\n--terminated-pod-gc-threshold=10\n--\n1.3.6 Edit the Controller Manager pod specification file /etc/kubernetes/manifests/kube-controller-manager.yaml\non the master node and set the --feature-gates parameter to include RotateKubeletServerCertificate=true.\n--feature-gates=RotateKubeletServerCertificate=true\n```\n\nProblem solved and 1.3.2 is passing:\n\n```sh\nroot@cluster2-controlplane1:~# kube-bench run --targets=master | grep 1.3.2\n[PASS] 1.3.2 Ensure that the --profiling argument is set to false (Scored)\n \n```\n\n##### Number 2\n\nNext task (2) is to check the ownership of directory /var/lib/etcd, so we first have a look:\n\n```sh\n➜ root@cluster2-controlplane1:~# ls -lh /var/lib | grep etcd\ndrwx------ 3 root root 4.0K Sep 11 20:08 etcd\n```\n\nLooks like user root and group root. Also possible to check using:\n\n```sh\n➜ root@cluster2-controlplane1:~# stat -c %U:%G /var/lib/etcd\nroot:root\n```\nBut what has kube-bench to say about this?\n\n```sh\n➜ root@cluster2-controlplane1:~# kube-bench run --targets=master | grep \"/var/lib/etcd\" -B5\n\n1.1.12 On the etcd server node, get the etcd data directory, passed as an argument --data-dir,\nfrom the below command:\n\nps -ef | grep etcd\nRun the below command (based on the etcd data directory found above).\nFor example, chown etcd:etcd /var/lib/etcd\n```\n\nTo comply we run the following:\n\n```sh\n➜ root@cluster2-controlplane1:~# chown etcd:etcd /var/lib/etcd\n\n➜ root@cluster2-controlplane1:~# ls -lh /var/lib | grep etcd\ndrwx------ 3 etcd etcd 4.0K Sep 11 20:08 etcd\n```\n\nThis looks better. We run kube-bench again, and make sure test 1.1.12. is passing.\n\n```sh\n➜ root@cluster2-controlplane1:~# kube-bench run --targets=master | grep 1.1.12\n[PASS] 1.1.12 Ensure that the etcd data directory ownership is set to etcd:etcd (Scored)\n```\n\nDone.\n\n \n\n##### Number 3\n\nTo continue with number (3), we'll head to the worker node and ensure that the kubelet configuration file has the minimum necessary permissions as recommended:\n\n```sh\n\n➜ ssh cluster2-node1\n\n➜ root@cluster2-node1:~# kube-bench run --targets=node\n...\n== Summary ==\n13 checks PASS\n10 checks FAIL\n2 checks WARN\n0 checks INFO\n```\n\nAlso here some passes, fails and warnings. We check the permission level of the kubelet config file:\n\n```sh\n➜ root@cluster2-node1:~# stat -c %a /var/lib/kubelet/config.yaml\n777\n```\n\n777 is highly permissive access level and not recommended by the kube-bench guidelines:\n\n```sh\n➜ root@cluster2-node1:~# kube-bench run --targets=node | grep /var/lib/kubelet/config.yaml -B2\n\n4.1.9 Run the following command (using the config file location identified in the Audit step)\nchmod 644 /var/lib/kubelet/config.yaml\n```\n\nWe obey and set the recommended permissions:\n\n```sh\n➜ root@cluster2-node1:~# chmod 644 /var/lib/kubelet/config.yaml\n\n➜ root@cluster2-node1:~# stat -c %a /var/lib/kubelet/config.yaml\n644\n```\n\nAnd check if test 2.2.10 is passing:\n\n```sh\n➜ root@cluster2-node1:~# kube-bench run --targets=node | grep 4.1.9\n[PASS] 2.2.10 Ensure that the kubelet configuration file has permissions set to 644 or more restrictive (Scored)\n```\n\n\n##### Number 4\n\nFinally for number (4), let's check whether --client-ca-file argument for the kubelet is set properly according to kube-bench recommendations:\n\n```sh\n➜ root@cluster2-node1:~# kube-bench run --targets=node | grep client-ca-file\n[PASS] 4.2.3 Ensure that the --client-ca-file argument is set as appropriate (Automated)\n```\n\nThis looks passing with 4.2.3. The other ones are about the file that the parameter points to and can be ignored here.\n\nTo further investigate we run the following command to locate the kubelet config file, and open it:\n\n```sh\n➜ root@cluster2-node1:~# ps -ef | grep kubelet\nroot 5157 1 2 20:28 ? 00:03:22 /usr/bin/kubelet --bootstrap-kubeconfig=/etc/kubernetes/bootstrap-kubelet.conf --kubeconfig=/etc/kubernetes/kubelet.conf --config=/var/lib/kubelet/config.yaml --network-plugin=cni --pod-infra-container-image=k8s.gcr.io/pause:3.2\nroot 19940 11901 0 22:38 pts/0 00:00:00 grep --color=auto kubelet\n```\n\n```yaml\n➜ root@croot@cluster2-node1:~# vim /var/lib/kubelet/config.yaml\n# /var/lib/kubelet/config.yaml\napiVersion: kubelet.config.k8s.io/v1beta1\nauthentication:\n anonymous:\n enabled: false\n webhook:\n cacheTTL: 0s\n enabled: true\n x509:\n clientCAFile: /etc/kubernetes/pki/ca.crt\n...\n```\nThe clientCAFile points to the location of the certificate, which is correct.\n\n \n\n\n## Question 6 | Verify Platform Binaries\n\n#### Task weight: 2%\n\n\n(can be solved in any kubectl context)\n\n \n\nThere are four Kubernetes server binaries located at `/opt/course/6/binaries`. You're provided with the following verified sha512 values for these:\n\n###### kube-apiserver \n\n`f417c0555bc0167355589dd1afe23be9bf909bf98312b1025f12015d1b58a1c62c9908c0067a7764fa35efdac7016a9efa8711a44425dd6692906a7c283f032c`\n\n###### kube-controller-manager \n\n`60100cc725e91fe1a949e1b2d0474237844b5862556e25c2c655a33boa8225855ec5ee22fa4927e6c46a60d43a7c4403a27268f96fbb726307d1608b44f38a60`\n\n###### kube-proxy \n\n`52f9d8ad045f8eee1d689619ef8ceef2d86d50c75a6a332653240d7ba5b2a114aca056d9e513984ade24358c9662714973c1960c62a5cb37dd375631c8a614c6`\n\n###### kubelet \n\n`4be40f2440619e990897cf956c32800dc96c2c983bf64519854a3309fa5aa21827991559f9c44595098e27e6f2ee4d64a3fdec6baba8a177881f20e3ec61e26c`\n\nDelete those binaries that don't match with the sha512 values above.\n\n\n#### Answer:\n\nWe check the directory:\n\n```sh\n\n➜ cd /opt/course/6/binaries\n\n➜ ls\nkube-apiserver kube-controller-manager kube-proxy kubelet\n```\n\nTo generate the sha512 sum of a binary we do:\n\n```sh\n➜ sha512sum kube-apiserver \nf417c0555bc0167355589dd1afe23be9bf909bf98312b1025f12015d1b58a1c62c9908c0067a7764fa35efdac7016a9efa8711a44425dd6692906a7c283f032c kube-apiserver\n```\n\nLooking good, next:\n\n```\n➜ sha512sum kube-controller-manager\n60100cc725e91fe1a949e1b2d0474237844b5862556e25c2c655a33b0a8225855ec5ee22fa4927e6c46a60d43a7c4403a27268f96fbb726307d1608b44f38a60 kube-controller-manager\n```\n\nOkay, next:\n\n```sh\n➜ sha512sum kube-proxy\n52f9d8ad045f8eee1d689619ef8ceef2d86d50c75a6a332653240d7ba5b2a114aca056d9e513984ade24358c9662714973c1960c62a5cb37dd375631c8a614c6 kube-proxy\n```\n\nAlso good, and finally:\n\n```sh\n➜ sha512sum kubelet\n7b720598e6a3483b45c537b57d759e3e82bc5c53b3274f681792f62e941019cde3d51a7f9b55158abf3810d506146bc0aa7cf97b36f27f341028a54431b335be kubelet\n```\n\nCatch! Binary kubelet has a different hash!\n\n\nBut did we actually compare everything properly before? Let's have a closer look at kube-controller-manager again:\n\n```sh\n➜ sha512sum kube-controller-manager \u003e compare\n\n➜ vim compare \n```\n\nEdit to only have the provided hash and the generated one in one line each:\n\n```sh\n# ./compare\n60100cc725e91fe1a949e1b2d0474237844b5862556e25c2c655a33b0a8225855ec5ee22fa4927e6c46a60d43a7c4403a27268f96fbb726307d1608b44f38a60 \n60100cc725e91fe1a949e1b2d0474237844b5862556e25c2c655a33boa8225855ec5ee22fa4927e6c46a60d43a7c4403a27268f96fbb726307d1608b44f38a60\n```\n\nLooks right at a first glance, but if we do:\n\n```sh\n➜ cat compare | uniq\n60100cc725e91fe1a949e1b2d0474237844b5862556e25c2c655a33b0a8225855ec5ee22fa4927e6c46a60d43a7c4403a27268f96fbb726307d1608b44f38a60\n60100cc725e91fe1a949e1b2d0474237844b5862556e25c2c655a33boa8225855ec5ee22fa4927e6c46a60d43a7c4403a27268f96fbb726307d1608b44f38a60\n```\n\nThis shows they are different, by just one character actually.\n\nTo complete the task we do:\n\n```sh\nrm kubelet kube-controller-manager\n```\n \n \n\n## Question 7 | Open Policy Agent\n\n#### Task weight: 6%\n\n \n\nUse context: `kubectl config use-context infra-prod`\n\n \n\nThe Open Policy Agent and Gatekeeper have been installed to, among other things, enforce blacklisting of certain image registries. Alter the existing constraint and/or template to also blacklist images from very-bad-registry.com.\n\nTest it by creating a single Pod using image very-bad-registry.com/image in Namespace default, it shouldn't work.\n\nYou can also verify your changes by looking at the existing Deployment untrusted in Namespace default, it uses an image from the new untrusted source. The OPA contraint should throw violation messages for this one.\n\n \n\n#### Answer:\n\nWe look at existing OPA constraints, these are implemeted using CRDs by Gatekeeper:\n\n```sh\n➜ k get crd\nNAME CREATED AT\nblacklistimages.constraints.gatekeeper.sh 2020-09-14T19:29:31Z\nconfigs.config.gatekeeper.sh 2020-09-14T19:29:04Z\nconstraintpodstatuses.status.gatekeeper.sh 2020-09-14T19:29:05Z\nconstrainttemplatepodstatuses.status.gatekeeper.sh 2020-09-14T19:29:05Z\nconstrainttemplates.templates.gatekeeper.sh 2020-09-14T19:29:05Z\nrequiredlabels.constraints.gatekeeper.sh 2020-09-14T19:29:31Z\n```\n\nSo we can do:\n\n```sh\n➜ k get constraint\nNAME AGE\nblacklistimages.constraints.gatekeeper.sh/pod-trusted-images 10m\n\nNAME AGE\nrequiredlabels.constraints.gatekeeper.sh/namespace-mandatory-labels 10m\n```\n\nand then look at the one that is probably about blacklisting images:\n\n```sh\nk edit blacklistimages pod-trusted-images\n```\n```yaml\n# kubectl edit blacklistimages pod-trusted-images\napiVersion: constraints.gatekeeper.sh/v1beta1\nkind: BlacklistImages\nmetadata:\n...\nspec:\n match:\n kinds:\n - apiGroups:\n - \"\"\n kinds:\n - Pod\n```` \nIt looks like this constraint simply applies the template to all Pods, no arguments passed. So we edit the template:\n\n```sh\nk edit constrainttemplates blacklistimages\n```\n\n```yaml\n# kubectl edit constrainttemplates blacklistimages\napiVersion: templates.gatekeeper.sh/v1beta1\nkind: ConstraintTemplate\nmetadata:\n...\nspec:\n crd:\n spec:\n names:\n kind: BlacklistImages\n targets:\n - rego: |\n package k8strustedimages\n\n images {\n image := input.review.object.spec.containers[_].image\n not startswith(image, \"docker-fake.io/\")\n not startswith(image, \"google-gcr-fake.com/\")\n not startswith(image, \"very-bad-registry.com/\") # ADD THIS LINE\n }\n\n violation[{\"msg\": msg}] {\n not images\n msg := \"not trusted image!\"\n }\n target: admission.k8s.gatekeeper.sh\n```\n\nWe simply have to add another line. After editing we try to create a Pod of the bad image:\n\n```sh\n➜ k run opa-test --image=very-bad-registry.com/image\nError from server ([denied by pod-trusted-images] not trusted image!): admission webhook \"validation.gatekeeper.sh\" denied the request: [denied by pod-trusted-images] not trusted image!\n```\n\nNice! After some time we can also see that Pods of the existing Deployment \"untrusted\" will be listed as violators:\n\n```sh\n➜ k describe blacklistimages pod-trusted-images\n...\n Total Violations: 2\n Violations:\n Enforcement Action: deny\n Kind: Namespace\n Message: you must provide labels: {\"security-level\"}\n Name: sidecar-injector\n Enforcement Action: deny\n Kind: Pod\n Message: not trusted image!\n Name: untrusted-68c4944d48-tfsnb\n Namespace: default\nEvents: \u003cnone\u003e\n```\n\nGreat, OPA fights bad registries !\n\n \n\n \n\n## Question 8 | Secure Kubernetes Dashboard\n\n#### Task weight: 3%\n\n\nUse context: `kubectl config use-context workload-prod`\n\n \n\nThe Kubernetes Dashboard is installed in Namespace kubernetes-dashboard and is configured to:\n\n* Allow users to \"skip login\"\n* Allow insecure access (HTTP without authentication)\n* Allow basic authentication\n* Allow access from outside the cluster\n\n\nYou are asked to make it more secure by:\n\n* Deny users to \"skip login\"\n* Deny insecure access, enforce HTTPS (self signed certificates are ok for now)\n* Add the --auto-generate-certificates argument\n* Enforce authentication using a token (with possibility to use RBAC)\n* Allow only cluster internal access\n \n\n#### Answer:\n\nHead to https://github.com/kubernetes/dashboard/tree/master/docs to find documentation about the dashboard. This link is not on the allowed list of urls during the real exam. This means you should be provided will all information necessary in case of a task like this.\n\nFirst we have a look in Namespace kubernetes-dashboard:\n\n```sh\n➜ k -n kubernetes-dashboard get pod,svc\nNAME READY STATUS RESTARTS AGE\npod/dashboard-metrics-scraper-7b59f7d4df-fbpd9 1/1 Running 0 24m\npod/kubernetes-dashboard-6d8cd5dd84-w7wr2 1/1 Running 0 24m\n\nNAME TYPE ... PORT(S) AGE\nservice/dashboard-metrics-scraper ClusterIP ... 8000/TCP 24m\nservice/kubernetes-dashboard NodePort ... 9090:32520/TCP,443:31206/TCP 24m\n```\n\nWe can see one running Pod and a NodePort Service exposing it. Let's try to connect to it via a NodePort, we can use IP of any Node:\n\n(your port might be a different)\n\n```sh\n➜ k get node -o wide\nNAME STATUS ROLES AGE VERSION INTERNAL-IP ...\ncluster1-controlplane1 Ready master 37m v1.24.1 192.168.100.11 ...\ncluster1-node1 Ready \u003cnone\u003e 36m v1.24.1 192.168.100.12 ...\ncluster1-node2 Ready \u003cnone\u003e 34m v1.24.1 192.168.100.13 ...\n\n➜ curl http://192.168.100.11:32520\n\u003c!--\nCopyright 2017 The Kubernetes Authors.\n\nLicensed under the Apache License, Version 2.0 (the \"License\");\nyou may not use this file except in compliance with the License.\nYou may obtain a copy of the License at\n\n http://www.apache.org/licenses/LICENSE-2.0\n \n```\n\nThe dashboard is not secured because it allows unsecure HTTP access without authentication and is exposed externally. It's is loaded with a few parameter making it insecure, let's fix this.\n\nFirst we create a backup in case we need to undo something:\n\n```sh\nk -n kubernetes-dashboard get deploy kubernetes-dashboard -oyaml \u003e 8_deploy_kubernetes-dashboard.yaml\n```\n\nThen:\n\n```sh\nk -n kubernetes-dashboard edit deploy kubernetes-dashboard\n```\n\nThe changes to make are :\n\n```sh\n template:\n spec:\n containers:\n - args:\n - --namespace=kubernetes-dashboard \n - --authentication-mode=token # change or delete, \"token\" is default\n - --auto-generate-certificates # add\n #- --enable-skip-login=true # delete or set to false\n #- --enable-insecure-login # delete\n image: kubernetesui/dashboard:v2.0.3\n imagePullPolicy: Always\n name: kubernetes-dashboard\n```\n\nNext, we'll have to deal with the NodePort Service:\n\n```sh\n\nk -n kubernetes-dashboard get svc kubernetes-dashboard -o yaml \u003e 8_svc_kubernetes-dashboard.yaml # backup\n\nk -n kubernetes-dashboard edit svc kubernetes-dashboard\n```\n\nAnd make the following changes:\n\n```yaml\n spec:\n clusterIP: 10.107.176.19\n externalTrafficPolicy: Cluster # delete\n internalTrafficPolicy: Cluster\n ports:\n - name: http\n nodePort: 32513 # delete\n port: 9090\n protocol: TCP\n targetPort: 9090\n - name: https\n nodePort: 32441 # delete\n port: 443\n protocol: TCP\n targetPort: 8443\n selector:\n k8s-app: kubernetes-dashboard\n sessionAffinity: None\n type: ClusterIP # change or delete\nstatus:\n loadBalancer: {}\n```\n\nLet's confirm the changes, we can do that even without having a browser:\n\n```sh\n➜ k run tmp --image=nginx:1.19.2 --restart=Never --rm -it -- bash\nIf you don't see a command prompt, try pressing enter.\nroot@tmp:/# curl http://kubernetes-dashboard.kubernetes-dashboard:9090\ncurl: (7) Failed to connect to kubernetes-dashboard.kubernetes-dashboard port 9090: Connection refused\n\n➜ root@tmp:/# curl https://kubernetes-dashboard.kubernetes-dashboard\ncurl: (60) SSL certificate problem: self signed certificate\nMore details here: https://curl.haxx.se/docs/sslcerts.html\n\ncurl failed to verify the legitimacy of the server and therefore could not\nestablish a secure connection to it. To learn more about this situation and\nhow to fix it, please visit the web page mentioned above.\n\n➜ root@tmp:/# curl https://kubernetes-dashboard.kubernetes-dashboard -k\n\u003c!--\nCopyright 2017 The Kubernetes Authors.\n```\n\nWe see that insecure access is disabled and HTTPS works (using a self signed certificate for now). Let's also check the remote access:\n\n(your port might be a different)\n\n```sh\n➜ curl http://192.168.100.11:32520\ncurl: (7) Failed to connect to 192.168.100.11 port 32520: Connection refused\n\n➜ k -n kubernetes-dashboard get svc\nNAME TYPE CLUSTER-IP ... PORT(S)\ndashboard-metrics-scraper ClusterIP 10.111.171.247 ... 8000/TCP\nkubernetes-dashboard ClusterIP 10.100.118.128 ... 9090/TCP,443/TCP\n```\n\nMuch better.\n\n \n\n## Question 9 | AppArmor Profile\n\n#### Task weight: 3%\n\n \n\nUse context: `kubectl config use-context workload-prod`\n\n \n\nSome containers need to run more secure and restricted. There is an existing AppArmor profile located at `/opt/course/9/profile` for this.\n\n* Install the AppArmor profile on Node `cluster1-node1`. Connect using `ssh cluster1-node1`.\n\n* Add label `security=apparmor` to the Node\n\n* Create a Deployment named `apparmor` in Namespace `default` with:\n * One replica of image `nginx:1.19.2`\n * NodeSelector for `security=apparmor`\n * Single container named c1 with the AppArmor profile enabled\n\n\nThe Pod might not run properly with the profile enabled. Write the logs of the Pod into /opt/course/9/logs so another team can work on getting the application running.\n\n \n\n#### Answer:\nhttps://kubernetes.io/docs/tutorials/clusters/apparmor\n\n \n\n###### Part 1\n\nFirst we have a look at the provided profile:\n\n```sh\nvim /opt/course/9/profile\n```\n\n```yam\n# /opt/course/9/profile \n\n#include \u003ctunables/global\u003e\n \nprofile very-secure flags=(attach_disconnected) {\n #include \u003cabstractions/base\u003e\n\n file,\n\n # Deny all file writes.\n deny /** w,\n}\n```\n\nVery simple profile named very-secure which denies all file writes. Next we copy it onto the Node:\n\n```sh\n➜ scp /opt/course/9/profile cluster1-node1:~/\nWarning: Permanently added the ECDSA host key for IP address '192.168.100.12' to the list of known hosts.\nprofile 100% 161 329.9KB/s 00:00\n\n➜ ssh cluster1-node1\n\n➜ root@cluster1-node1:~# ls\nprofile\n```\n\nAnd install it:\n\n```sh\n➜ root@cluster1-node1:~# apparmor_parser -q ./profile\n```\n\nVerify it has been installed:\n\n```sh\n➜ root@cluster1-node1:~# apparmor_status\napparmor module is loaded.\n17 profiles are loaded.\n17 profiles are in enforce mode.\n /sbin/dhclient\n...\n man_filter\n man_groff\n very-secure\n0 profiles are in complain mode.\n56 processes have profiles defined.\n56 processes are in enforce mode.\n...\n0 processes are in complain mode.\n0 processes are unconfined but have a profile defined.\n```\n\nThere we see among many others the very-secure one, which is the name of the profile specified in /opt/course/9/profile.\n\n \n\n###### Part 2\n\nWe label the Node:\n\n```sh\nk label -h # show examples\n\nk label node cluster1-node1 security=apparmor\n```\n\n \n\n###### Part 3\nNow we can go ahead and create the Deployment which uses the profile.\n\n```sh\nk create deploy apparmor --image=nginx:1.19.2 $do \u003e 9_deploy.yaml\n```\n\n```yaml\nvim 9_deploy.yaml\n# 9_deploy.yaml\napiVersion: apps/v1\nkind: Deployment\nmetadata:\n creationTimestamp: null\n labels:\n app: apparmor\n name: apparmor\n namespace: default\nspec:\n replicas: 1\n selector:\n matchLabels:\n app: apparmor\n strategy: {}\n template:\n metadata:\n creationTimestamp: null\n labels:\n app: apparmor\n annotations: # add\n container.apparmor.security.beta.kubernetes.io/c1: localhost/very-secure # add\n spec:\n nodeSelector: # add\n security: apparmor # add\n containers:\n - image: nginx:1.19.2\n name: c1 # change\n resources: {}\n```` \n \n```sh \nk -f 9_deploy.yaml create\n```\n\nWhat the damage?\n\n```sh\n➜ k get pod -owide | grep apparmor\napparmor-85c65645dc-jbch8 0/1 CrashLoopBackOff ... cluster1-node1\n\n➜ k logs apparmor-85c65645dc-w852p\n/docker-entrypoint.sh: 13: /docker-entrypoint.sh: cannot create /dev/null: Permission denied\n/docker-entrypoint.sh: No files found in /docker-entrypoint.d/, skipping configuration\n2021/09/15 11:51:57 [emerg] 1#1: mkdir() \"/var/cache/nginx/client_temp\" failed (13: Permission denied)\nnginx: [emerg] mkdir() \"/var/cache/nginx/client_temp\" failed (13: Permission denied)\n```\n\nThis looks alright, the Pod is running on cluster1-node1 because of the nodeSelector. The AppArmor profile simply denies all filesystem writes, but Nginx needs to write into some locations to run, hence the errors.\n\nIt looks like our profile is running but we can confirm this as well by inspecting the container:\n\n```sh\n➜ ssh cluster1-node1\n\n➜ root@cluster1-node1:~# crictl pods | grep apparmor\nbe5c0aecee7c7 4 minutes ago Ready apparmor-85c65645dc-jbch8 ...\n\n➜ root@cluster1-node1:~# crictl ps -a | grep be5c0aecee7c7\ne4d91cbdf72fb ... Exited c1 6 be5c0aecee7c7\n\n➜ root@cluster1-node1:~# crictl inspect e4d91cbdf72fb | grep -i profile\n \"apparmor_profile\": \"localhost/very-secure\",\n \"apparmorProfile\": \"very-secure\",\n \n```\n \nFirst we find the Pod by it's name and get the pod-id. Next we use `crictl ps -a` to also show stopped containers. Then `crictl inspect` shows that the container is using our AppArmor profile. Notice to be fast between `ps` and `inspect` as K8s will restart the Pod periodically when in error state.\n\nTo complete the task we write the logs into the required location:\n\n```sh\nk logs apparmor-85c65645dc-jbch8 \u003e /opt/course/9/logs\n```\n\nFixing the errors is the job of another team, lucky us.\n\n \n\n## Question 10 | Container Runtime Sandbox gVisor\n\n#### Task weight: 4%\n\n \n\nUse context: `kubectl config use-context workload-prod`\n\n \n\nTeam purple wants to run some of their workloads more secure. Worker node cluster1-node2 has container engine containerd already installed and its configured to support the `runsc/gvisor runtime`.\n\nCreate a `RuntimeClass` named `gvisor` with handler `runsc`.\n\nCreate a Pod that uses the `RuntimeClass`. The Pod should be in `Namespace team-purple`, named `gvisor-test` and of image `nginx:1.19.2`. Make sure the Pod runs on `cluster1-node2`.\n\nWrite the `dmesg` output of the successfully started Pod into `/opt/course/10/gvisor-test-dmesg`.\n\n \n\n#### Answer:\n\nWe check the nodes and we can see that all are using containerd:\n\n```sh\n➜ k get node -o wide\nNAME STATUS ROLES ... CONTAINER-RUNTIME\ncluster1-controlplane1 Ready control-plane ... containerd://1.5.2\ncluster1-node1 Ready \u003cnone\u003e ... containerd://1.5.2\ncluster1-node2 Ready \u003cnone\u003e ... containerd://1.5.2\n```\n\nBut just one has containerd configured to work with runsc/gvisor runtime which is cluster1-node2.\n\n(Optionally) we ssh into the worker node and check if containerd+runsc is configured:\n\n```sh\n➜ ssh cluster1-node2\n\n➜ root@cluster1-node2:~# runsc --version\nrunsc version release-20201130.0\nspec: 1.0.1-dev\n\n➜ root@cluster1-node2:~# cat /etc/containerd/config.toml | grep runsc\n [plugins.\"io.containerd.grpc.v1.cri\".containerd.runtimes.runsc]\n runtime_type = \"io.containerd.runsc.v1\"\n```\n\nNow we best head to the k8s docs for RuntimeClasses https://kubernetes.io/docs/concepts/containers/runtime-class, steal an example and create the gvisor one:\n\n```sh\nvim 10_rtc.yaml\n# 10_rtc.yaml\napiVersion: node.k8s.io/v1\nkind: RuntimeClass\nmetadata:\n name: gvisor\nhandler: runsc\n```\n```sh\nk -f 10_rtc.yaml create\n```\n\nAnd the required Pod:\n\n```sh\nk -n team-purple run gvisor-test --image=nginx:1.19.2 $do \u003e 10_pod.yaml\n```\n```yaml\nvim 10_pod.yaml\n# 10_pod.yaml\napiVersion: v1\nkind: Pod\nmetadata:\n creationTimestamp: null\n labels:\n run: gvisor-test\n name: gvisor-test\n namespace: team-purple\nspec:\n nodeName: cluster1-node2 # add\n runtimeClassName: gvisor # add\n containers:\n - image: nginx:1.19.2\n name: gvisor-test\n resources: {}\n dnsPolicy: ClusterFirst\n restartPolicy: Always\nstatus: {}\n```\n\n```sh\nk -f 10_pod.yaml create\n```\n\nAfter creating the pod we should check if its running and if it uses the gvisor sandbox:\n\n```sh\n➜ k -n team-purple get pod gvisor-test\nNAME READY STATUS RESTARTS AGE\ngvisor-test 1/1 Running 0 30s\n\n➜ k -n team-purple exec gvisor-test -- dmesg\n[ 0.000000] Starting gVisor...\n[ 0.417740] Checking naughty and nice process list...\n[ 0.623721] Waiting for children...\n[ 0.902192] Gathering forks...\n[ 1.258087] Committing treasure map to memory...\n[ 1.653149] Generating random numbers by fair dice roll...\n[ 1.918386] Creating cloned children...\n[ 2.137450] Digging up root...\n[ 2.369841] Forking spaghetti code...\n[ 2.840216] Rewriting operating system in Javascript...\n[ 2.956226] Creating bureaucratic processes...\n[ 3.329981] Ready!\n\n```\nLooking good. And as required we finally write the dmesg output into the file:\n\n```sh\nk -n team-purple exec gvisor-test \u003e /opt/course/10/gvisor-test-dmesg -- dmesg\n```\n\n \n\n## Question 11 | Secrets in ETCD\n\n#### Task weight: 7%\n\n \n\nUse context: `kubectl config use-context workload-prod`\n\n\nThere is an existing Secret called `database-access` in Namespace `team-green`.\n\nRead the complete Secret content directly from ETCD (using `etcdctl`) and store it into `/opt/course/11/etcd-secret-content`. Write the plain and decoded Secret's value of key \"pass\" into `/opt/course/11/database-password`.\n\n \n\n#### Answer:\n\nLet's try to get the Secret value directly from ETCD, which will work since it isn't encrypted.\n\nFirst, we ssh into the master node where ETCD is running in this setup and check if etcdctl is installed and list its options:\n\n```sh\n➜ ssh cluster1-controlplane1\n\n➜ root@cluster1-controlplane1:~# etcdctl\nNAME:\n etcdctl - A simple command line client for etcd.\n\nWARNING:\n Environment variable ETCDCTL_API is not set; defaults to etcdctl v2.\n Set environment variable ETCDCTL_API=3 to use v3 API or ETCDCTL_API=2 to use v2 API.\n\nUSAGE:\n etcdctl [global options] command [command options] [arguments...]\n...\n --cert-file value identify HTTPS client using this SSL certificate file\n --key-file value identify HTTPS client using this SSL key file\n --ca-file value verify certificates of HTTPS-enabled servers using this CA bundle\n...\n```\n\nAmong others we see arguments to identify ourselves. The apiserver connects to ETCD, so we can run the following command to get the path of the necessary .crt and .key files:\n\n```sh\ncat /etc/kubernetes/manifests/kube-apiserver.yaml | grep etcd\n```\n\nThe output is as follows :\n\n```yaml\n - --etcd-cafile=/etc/kubernetes/pki/etcd/ca.crt\n - --etcd-certfile=/etc/kubernetes/pki/apiserver-etcd-client.crt\n - --etcd-keyfile=/etc/kubernetes/pki/apiserver-etcd-client.key\n - --etcd-servers=https://127.0.0.1:2379 # optional since we're on same node\n```\n\nWith this information we query ETCD for the secret value:\n\n```sh\n\n➜ root@cluster1-controlplane1:~# ETCDCTL_API=3 etcdctl \\\n--cert /etc/kubernetes/pki/apiserver-etcd-client.crt \\\n--key /etc/kubernetes/pki/apiserver-etcd-client.key \\\n--cacert /etc/kubernetes/pki/etcd/ca.crt get /registry/secrets/team-green/database-access\n```\n\nETCD in Kubernetes stores data under `/registry/{type}/{namespace}/{name}`. This is how we came to look for `/registry/secrets/team-green/database-access`. There is also an example on a page in the k8s documentation which you could save as a bookmark to access fast during the exam.\n\nThe tasks requires us to store the output on our terminal. For this we can simply copy\u0026paste the content into a new file on our terminal:\n\n```yaml\n# /opt/course/11/etcd-secret-content\n/registry/secrets/team-green/database-access\nk8s\n\n\nv1Secret\n\ndatabase-access\nteam-green\"*$3e0acd78-709d-4f07-bdac-d5193d0f2aa32bB\n0kubectl.kubernetes.io/last-applied-configuration{\"apiVersion\":\"v1\",\"data\":{\"pass\":\"Y29uZmlkZW50aWFs\"},\"kind\":\"Secret\",\"metadata\":{\"annotations\":{},\"name\":\"database-access\",\"namespace\":\"team-green\"}}\nz\nkubectl-client-side-applyUpdatevFieldsV1:\n{\"f:data\":{\".\":{},\"f:pass\":{}},\"f:metadata\":{\"f:annotations\":{\".\":{},\"f:kubectl.kubernetes.io/last-applied-configuration\":{}}},\"f:type\":{}}\npass\n confidentialOpaque\"\n \n```\n\nWe're also required to store the plain and \"decrypted\" database password. For this we can copy the base64-encoded value from the ETCD output and run on our terminal:\n\n```sh\n➜ echo Y29uZmlkZW50aWFs | base64 -d \u003e /opt/course/11/database-password\n\n➜ cat /opt/course/11/database-password\nconfidential\n ```\n\n \n\n## Question 12 | Hack Secrets\n\n#### Task weight: 8%\n\n \n\nUse context: `kubectl config use-context restricted@infra-prod`\n\n \n\nYou're asked to investigate a possible permission escape in Namespace `restricted`. The context authenticates as user `restricted` which has only limited permissions and shouldn't be able to read Secret values.\n\nTry to find the password-key values of the Secrets `secret1`, `secret2` and `secret3` in Namespace `restricted`. Write the decoded plaintext values into files `/opt/course/12/secret1`, `/opt/course/12/secret2` and `/opt/course/12/secret3`.\n\n \n\n#### Answer:\n\nFirst we should explore the boundaries, we can try:\n\n```sh\n➜ k -n restricted get role,rolebinding,clusterrole,clusterrolebinding\nError from server (Forbidden): roles.rbac.authorization.k8s.io is forbidden: User \"restricted\" cannot list resource \"roles\" in API group \"rbac.authorization.k8s.io\" in the namespace \"restricted\"\nError from server (Forbidden): rolebindings.rbac.authorization.k8s.io is forbidden: User \"restricted\" cannot list resource \"rolebindings\" in API group \"rbac.authorization.k8s.io\" in the namespace \"restricted\"\nError from server (Forbidden): clusterroles.rbac.authorization.k8s.io is forbidden: User \"restricted\" cannot list resource \"clusterroles\" in API group \"rbac.authorization.k8s.io\" at the cluster scope\nError from server (Forbidden): clusterrolebindings.rbac.authorization.k8s.io is forbidden: User \"restricted\" cannot list resource \"clusterrolebindings\" in API group \"rbac.authorization.k8s.io\" at the cluster scope\n```\n\nBut no permissions to view RBAC resources. So we try the obvious:\n\n```sh\n➜ k -n restricted get secret\nError from server (Forbidden): secrets is forbidden: User \"restricted\" cannot list resource \"secrets\" in API group \"\" in the namespace \"restricted\"\n\n➜ k -n restricted get secret -o yaml\napiVersion: v1\nitems: []\nkind: List\nmetadata:\n resourceVersion: \"\"\n selfLink: \"\"\nError from server (Forbidden): secrets is forbidden: User \"restricted\" cannot list resource \"secrets\" in API group \"\" in the namespace \"restricted\"\n```\n\nWe're not allowed to get or list any Secrets. What can we see though?\n\n```sh\n➜ k -n restricted get all\nNAME READY STATUS RESTARTS AGE\npod1-fd5d64b9c-pcx6q 1/1 Running 0 37s\npod2-6494f7699b-4hks5 1/1 Running 0 37s\npod3-748b48594-24s76 1/1 Running 0 37s\nError from server (Forbidden): replicationcontrollers is forbidden: User \"restricted\" cannot list resource \"replicationcontrollers\" in API group \"\" in the namespace \"restricted\"\nError from server (Forbidden): services is forbidden: User \"restricted\" cannot list resource \"services\" in API group \"\" in the namespace \"restricted\"\n...\n```\n\nThere are some Pods, lets check these out regarding Secret access:\n\n```sh\nk -n restricted get pod -o yaml | grep -i secret\n```\n\nThis output provides us with enough information to do:\n\n```sh\n➜ k -n restricted exec pod1-fd5d64b9c-pcx6q -- cat /etc/secret-volume/password\nyou-are\n\n➜ echo you-are \u003e /opt/course/12/secret1\n```\n\nAnd for the second Secret:\n\n```sh\n➜ k -n restricted exec pod2-6494f7699b-4hks5 -- env | grep PASS\nPASSWORD=an-amazing\n\n➜ echo an-amazing \u003e /opt/course/12/secret2\n```\n\nNone of the Pods seem to mount secret3 though. Can we create or edit existing Pods to mount secret3?\n\n```sh\n➜ k -n restricted run test --image=nginx\nError from server (Forbidden): pods is forbidden: User \"restricted\" cannot create resource \"pods\" in API group \"\" in the namespace \"restricted\"\n\n➜ k -n restricted delete pod pod1\nError from server (Forbidden): pods \"pod1\" is forbidden: User \"restricted\" cannot delete resource \"pods\" in API group \"\" in the namespace \"restricted\"\n```\n\nDoesn't look like it.\n\nBut the Pods seem to be able to access the Secrets, we can try to use a Pod's ServiceAccount to access the third Secret. We can actually see (like using `k -n restricted get pod -o yaml | grep automountServiceAccountToken`) that only Pod `pod3-*` has the `ServiceAccount` token mounted:\n\n```sh\n➜ k -n restricted exec -it pod3-748b48594-24s76 -- sh\n\n/ # mount | grep serviceaccount\ntmpfs on /run/secrets/kubernetes.io/serviceaccount type tmpfs (ro,relatime)\n\n/ # ls /run/secrets/kubernetes.io/serviceaccount\nca.crt namespace token\n```\n\n\n###### NOTE: You should have knowledge about ServiceAccounts and how they work with Pods like described in the docs\n\n\nWe can see all necessary information to contact the apiserver manually:\n\n```json\n/ # curl https://kubernetes.default/api/v1/namespaces/restricted/secrets -H \"Authorization: Bearer $(cat /run/secrets/kubernetes.io/serviceaccount/token)\" -k\n...\n {\n \"metadata\": {\n \"name\": \"secret3\",\n \"namespace\": \"restricted\",\n...\n }\n ]\n },\n \"data\": {\n \"password\": \"cEVuRXRSYVRpT24tdEVzVGVSCg==\"\n },\n \"type\": \"Opaque\"\n }\n...\n```\n\nLet's encode it and write it into the requested location:\n\n```sh\n➜ echo cEVuRXRSYVRpT24tdEVzVGVSCg== | base64 -d\npEnEtRaTiOn-tEsTeR\n\n➜ echo cEVuRXRSYVRpT24tdEVzVGVSCg== | base64 -d \u003e /opt/course/12/secret3\n```\n\nThis will give us:\n```yaml\n# /opt/course/12/secret1\nyou-are\n```\n\n```yaml\n# /opt/course/12/secret2\nan-amazing\n```\n\n```yaml\n# /opt/course/12/secret3\npEnEtRaTiOn-tEsTeR\n```\n\nWe hacked all Secrets! It can be tricky to get RBAC right and secure.\n\n\n###### NOTE: One thing to consider is that giving the permission to \"list\" Secrets, will also allow the user to read the Secret values like using kubectl get secrets -o yaml even without the \"get\" permission set.\n\n \n\n \n\n## Question 13 | Restrict access to Metadata Server\n\n#### Task weight: 7%\n\n \n\nUse context: `kubectl config use-context infra-prod`\n\n \n\nThere is a metadata service available at `http://192.168.100.21:32000` on which Nodes can reach sensitive data, like cloud credentials for initialisation. By default, all Pods in the cluster also have access to this endpoint. The DevSecOps team has asked you to restrict access to this metadata server.\n\nIn Namespace `metadata-access`:\n\n* Create a NetworkPolicy named `metadata-deny` which prevents egress to `192.168.100.21` for all Pods but still allows access to everything else\n* Create a NetworkPolicy named `metadata-allow` which allows Pods having label `role: metadata-accessor` to access endpoint `192.168.100.21`\n\nThere are existing Pods in the target Namespace with which you can test your policies, but don't change their labels.\n\n \n\n#### Answer:\n \n\nCheck the Pods in the Namespace `metadata-access` and their labels:\n\n```sh\n➜ k -n metadata-access get pods --show-labels\nNAME ... LABELS\npod1-7d67b4ff9-xrcd7 ... app=pod1,pod-template-hash=7d67b4ff9\npod2-7b6fc66944-2hc7n ... app=pod2,pod-template-hash=7b6fc66944\npod3-7dc879bd59-hkgrr ... app=pod3,role=metadata-accessor,pod-template-hash=7dc879bd59\n```\n\nThere are three Pods in the Namespace and one of them has the label role=metadata-accessor.\n\nCheck access to the metadata server from the Pods:\n\n```sh\n➜ k exec -it -n metadata-access pod1-7d67b4ff9-xrcd7 -- curl http://192.168.100.21:32000\nmetadata server\n\n➜ k exec -it -n metadata-access pod2-7b6fc66944-2hc7n -- curl http://192.168.100.21:32000\nmetadata server\n\n➜ k exec -it -n metadata-access pod3-7dc879bd59-hkgrr -- curl http://192.168.100.21:32000\nmetadata server\n```\n\nAll three are able to access the metadata server.\n\nTo restrict the access, we create a NetworkPolicy to deny access to the specific IP.\n\n```sh\nvim 13_metadata-deny.yaml\n```\n```yaml\n# 13_metadata-deny.yaml\napiVersion: networking.k8s.io/v1\nkind: NetworkPolicy\nmetadata:\n name: metadata-deny\n namespace: metadata-access\nspec:\n podSelector: {}\n policyTypes:\n - Egress\n egress:\n - to:\n - ipBlock:\n cidr: 0.0.0.0/0\n except:\n - 192.168.100.21/32\n```\n\n```sh\nk -f 13_metadata-deny.yaml apply\n``` \n\n###### NOTE: You should know about general default-deny K8s NetworkPolcies.\n\n\nVerify that access to the metadata server has been blocked, but other endpoints are still accessible:\n\n```sh\n➜ k exec -it -n metadata-access pod1-7d67b4ff9-xrcd7 -- curl http://192.168.100.21:32000\ncurl: (28) Failed to connect to 192.168.100.21 port 32000: Operation timed out\ncommand terminated with exit code 28\n\n➜ kubectl exec -it -n metadata-access pod1-7d67b4ff9-xrcd7 -- curl -I https://kubernetes.io\nHTTP/2 200\ncache-control: public, max-age=0, must-revalidate\ncontent-type: text/html; charset=UTF-8\ndate: Mon, 14 Sep 2020 15:39:39 GMT\netag: \"b46e429397e5f1fecf48c10a533f5cd8-ssl\"\nstrict-transport-security: max-age=31536000\nage: 13\ncontent-length: 22252\nserver: Netlify\nx-nf-request-id: 1d94a1d1-6bac-4a98-b065-346f661f1db1-393998290\n```\n\nSimilarly, verify for the other two Pods.\n\nNow create another NetworkPolicy that allows access to the metadata server from Pods with label role=metadata-accessor.\n\n```sh\nvim 13_metadata-allow.yaml\n```\n\n```yaml\n# 13_metadata-allow.yaml\napiVersion: networking.k8s.io/v1\nkind: NetworkPolicy\nmetadata:\n name: metadata-allow\n namespace: metadata-access\nspec:\n podSelector:\n matchLabels:\n role: metadata-accessor\n policyTypes:\n - Egress\n egress:\n - to:\n - ipBlock:\n cidr: 192.168.100.21/32\n```\n\n```sh\nk -f 13_metadata-allow.yaml apply\n```\n\nVerify that required Pod has access to metadata endpoint and others do not:\n\n```sh\n➜ k -n metadata-access exec pod3-7dc879bd59-hkgrr -- curl http://192.168.100.21:32000\nmetadata server\n\n➜ k -n metadata-access exec pod2-7b6fc66944-9ngzr -- curl http://192.168.100.21:32000\n^Ccommand terminated with exit code 130\n```\n\nIt only works for the Pod having the label. With this we implemented the required security restrictions.\n\nIf a Pod doesn't have a matching NetworkPolicy then all traffic is allowed from and to it. Once a Pod has a matching NP then the contained rules are additive. This means that for Pods having label `metadata-accessor` the rules will be combined to:\n\n```yaml\n# merged policies into one for pods with label metadata-accessor\nspec:\n podSelector: {}\n policyTypes:\n - Egress\n egress:\n - to: # first rule\n - ipBlock: # condition 1\n cidr: 0.0.0.0/0\n except:\n - 192.168.100.21/32\n - to: # second rule\n - ipBlock: # condition 1\n cidr: 192.168.100.21/32\n```\n\nWe can see that the merged NP contains two separate rules with one condition each. We could read it as:\n\n```yaml\nAllow outgoing traffic if:\n(destination is 0.0.0.0/0 but not 192.168.100.21/32) OR (destination is 192.168.100.21/32)\n```\n\nHence it allows Pods with label metadata-accessor to access everything.\n\n \n\n## Question 14 | Syscall Activity\n\n#### Task weight: 4%\n\n \n\nUse context: `kubectl config use-context workload-prod`\n\n \n\nThere are Pods in Namespace `team-yellow`. A security investigation noticed that some processes running in these Pods are using the Syscall `kill`, which is forbidden by a Team Yellow internal policy.\n\nFind the offending Pod(s) and remove these by reducing the replicas of the parent Deployment to 0.\n\n \n\n#### Answer:\n\nSyscalls are used by processes running in Userspace to communicate with the Linux Kernel. There are many available syscalls: https://man7.org/linux/man-pages/man2/syscalls.2.html. It makes sense to restrict these for container processes and Docker/Containerd already restrict some by default, like the reboot Syscall. Restricting even more is possible for example using Seccomp or AppArmor.\n\nBut for this task we should simply find out which binary process executes a specific Syscall. Processes in containers are simply run on the same Linux operating system, but isolated. That's why we first check on which nodes the Pods are running:\n\n```sh\n➜ k -n team-yellow get pod -owide\nNAME ... NODE NOMINATED NODE ...\ncollector1-7585cc58cb-n5rtd 1/1 ... cluster1-node1 \u003cnone\u003e ...\ncollector1-7585cc58cb-vdlp9 1/1 ... cluster1-node1 \u003cnone\u003e ...\ncollector2-8556679d96-z7g7c 1/1 ... cluster1-node1 \u003cnone\u003e ...\ncollector3-8b58fdc88-pjg24 1/1 ... cluster1-node1 \u003cnone\u003e ...\ncollector3-8b58fdc88-s9ltc 1/1 ... cluster1-node1 \u003cnone\u003e ...\n```\n\nAll on `cluster1-node1`, hence we ssh into it and find the processes for the first Deployment collector1 .\n\n```sh\n➜ ssh cluster1-node1\n\n➜ root@cluster1-node1:~# crictl pods --name collector1\nPOD ID CREATED STATE NAME ...\n21aacb8f4ca8d 17 minutes ago Ready collector1-7585cc58cb-vdlp9 ...\n186631e40104d 17 minutes ago Ready collector1-7585cc58cb-n5rtd ...\n\n➜ root@cluster1-node1:~# crictl ps --pod 21aacb8f4ca8d\nCONTAINER ID IMAGE CREATED ... POD ID\n9ea02422f8660 5d867958e04e1 12 minutes ago ... 21aacb8f4ca8d\n\n➜ root@cluster1-node1:~# crictl inspect 9ea02422f8660 | grep args -A1\n \"args\": [\n \"./collector1-process\"\n```\n\n* Using crictl pods we first searched for the Pods of Deployment collector1, which has two replicas\n* We then took one pod-id to find it's containers using crictl ps\n* And finally we used crictl inspect to find the process name, which is collector1-process\n\nWe can find the process PIDs (two because there are two Pods):\n\n```\n➜ root@cluster1-node1:~# ps aux | grep collector1-process\nroot 35039 0.0 0.1 702208 1044 ? Ssl 13:37 0:00 ./collector1-process\nroot 35059 0.0 0.1 702208 1044 ? Ssl 13:37 0:00 ./collector1-process\n```\n\nUsing the PIDs we can call strace to find Sycalls:\n\n```sh\n➜ root@cluster1-node1:~# strace -p 35039\nstrace: Process 35039 attached\nfutex(0x4d7e68, FUTEX_WAIT_PRIVATE, 0, NULL) = 0\nkill(666, SIGTERM) = -1 ESRCH (No such process)\nepoll_pwait(3, [], 128, 999, NULL, 1) = 0\nkill(666, SIGTERM) = -1 ESRCH (No such process)\nepoll_pwait(3, [], 128, 999, NULL, 1) = 0\nkill(666, SIGTERM) = -1 ESRCH (No such process)\nepoll_pwait(3, ^Cstrace: Process 35039 detached\n \u003cdetached ...\u003e\n...\n```\n\nFirst try and already a catch! We see it uses the forbidden Syscall by calling kill(666, SIGTERM).\n\nNext let's check the Deployment collector2 processes:\n\n```sh\n➜ root@cluster1-node1:~# ps aux | grep collector2-process\nroot 35375 0.0 0.0 702216 604 ? Ssl 13:37 0:00 ./collector2-process\n\n➜ root@cluster1-node1:~# strace -p 35375\nstrace: Process 35375 attached\nfutex(0x4d9e68, FUTEX_WAIT_PRIVATE, 0, NULL) = 0\nfutex(0x4d9e68, FUTEX_WAIT_PRIVATE, 0, NULL) = 0\nfutex(0x4d9e68, FUTEX_WAIT_PRIVATE, 0, NULL) = 0\nfutex(0x4d9e68, FUTEX_WAIT_PRIVATE, 0, NULL) = 0\n...\n```\n\nLooks alright. What about collector3:\n\n```sh\n➜ root@cluster1-node1:~# ps aux | grep collector3-process\nroot 35155 0.0 0.1 702472 1040 ? Ssl 13:37 0:00 ./collector3-process\nroot 35241 0.0 0.1 702472 1044 ? Ssl 13:37 0:00 ./collector3-process\n\n➜ root@cluster1-node1:~# strace -p 35155\nstrace: Process 35155 attached\nfutex(0x4d9e68, FUTEX_WAIT_PRIVATE, 0, NULL) = 0\nfutex(0x4d9e68, FUTEX_WAIT_PRIVATE, 0, NULL) = 0\nfutex(0x4d9e68, FUTEX_WAIT_PRIVATE, 0, NULL) = 0\nepoll_pwait(3, [], 128, 999, NULL, 1) = 0\nepoll_pwait(3, [], 128, 999, NULL, 1) = 0\n...\n```\n\nAlso nothing about the forbidden Syscall. So we finalise the task:\n\n```sh\nk -n team-yellow scale deploy collector1 --replicas 0\n```\n\nAnd the world is a bit safer again.\n\n \n\n## Question 15 | Configure TLS on Ingress\n#### Task weight: 4%\n\n \n\nUse context: `kubectl config use-context workload-prod`\n\n \n\nIn Namespace `team-pink` there is an existing Nginx Ingress resources named `secure` which accepts two paths `/app` and `/api` which point to different ClusterIP Services.\n\nFrom your main terminal you can connect to it using for example:\n\n* HTTP: `curl -v http://secure-ingress.test:31080/app`\n* HTTPS: `curl -kv https://secure-ingress.test:31443/app`\n\nRight now it uses a default generated TLS certificate by the Nginx Ingress Controller.\n\nYou're asked to instead use the key and certificate provided at `/opt/course/15/tls.key` and `/opt/course/15/tls.crt`. As it's a self-signed certificate you need to use `curl -k` when connecting to it.\n\n \n\n#### Answer:\n\nInvestigate\nWe can get the IP address of the Ingress and we see it's the same one to which `secure-ingress.test` is pointing to:\n\n```sh\n➜ k -n team-pink get ing secure \nNAME CLASS HOSTS ADDRESS PORTS AGE\nsecure \u003cnone\u003e secure-ingress.test 192.168.100.12 80 7m11s\n\n➜ ping secure-ingress.test\nPING cluster1-node1 (192.168.100.12) 56(84) bytes of data.\n64 bytes from cluster1-node1 (192.168.100.12): icmp_seq=1 ttl=64 time=0.316 ms\n```\n\nNow, let's try to access the paths /app and /api via HTTP:\n\n```sh\n➜ curl http://secure-ingress.test:31080/app\nThis is the backend APP!\n\n➜ curl http://secure-ingress.test:31080/api\nThis is the API Server!\n```\n\nWhat about HTTPS?\n\n```sh\n➜ curl https://secure-ingress.test:31443/api\ncurl: (60) SSL certificate problem: unable to get local issuer certificate\nMore details here: https://curl.haxx.se/docs/sslcerts.html\n\ncurl failed to verify the legitimacy of the server and therefore could not\nestablish a secure connection to it. To learn more about this situation and\nhow to fix it, please visit the web page mentioned above.\n\n➜ curl -k https://secure-ingress.test:31443/api\nThis is the API Server!\n```\n\nHTTPS seems to be already working if we accept self-signed certificated using -k. But what kind of certificate is used by the server?\n\n```sh\n➜ curl -kv https://secure-ingress.test:31443/api\n...\n* Server certificate:\n* subject: O=Acme Co; CN=Kubernetes Ingress Controller Fake Certificate\n* start date: Sep 28 12:28:35 2020 GMT\n* expire date: Sep 28 12:28:35 2021 GMT\n* issuer: O=Acme Co; CN=Kubernetes Ingress Controller Fake Certificate\n* SSL certificate verify result: unable to get local issuer certificate (20), continuing anyway.\n...\n```\n\nIt seems to be \"Kubernetes Ingress Controller Fake Certificate\".\n\n\n#### Implement own TLS certificate\n\nFirst, let us generate a Secret using the provided key and certificate:\n\n```sh\n➜ cd /opt/course/15\n\n➜ :/opt/course/15$ ls\ntls.crt tls.key\n\n➜ :/opt/course/15$ k -n team-pink create secret tls tls-secret --key tls.key --cert tls.crt\nsecret/tls-secret created\n```\n\nNow, we configure the Ingress to make use of this Secret:\n\n```sh\n➜ k -n team-pink get ing secure -oyaml \u003e 15_ing_bak.yaml\n\n➜ k -n team-pink edit ing secure\n```\n```yaml\n# kubectl -n team-pink edit ing secure\napiVersion: networking.k8s.io/v1\nkind: Ingress\nmetadata:\n annotations:\n...\n generation: 1\n name: secure\n namespace: team-pink\n...\nspec:\n tls: # add\n - hosts: # add\n - secure-ingress.test # add\n secretName: tls-secret # add\n rules:\n - host: secure-ingress.test\n http:\n paths:\n - backend:\n service:\n name: secure-app\n port: 80\n path: /app\n pathType: ImplementationSpecific\n - backend:\n service:\n name: secure-api\n port: 80\n path: /api\n pathType: ImplementationSpecific\n...\n```\n\nAfter adding the changes we check the Ingress resource again:\n\n```sh\n➜ k -n team-pink get ing\nNAME CLASS HOSTS ADDRESS PORTS AGE\nsecure \u003cnone\u003e secure-ingress.test 192.168.100.12 80, 443 25m\n```\n\nIt now actually lists port 443 for HTTPS. To verify:\n\n```sh\n➜ curl -k https://secure-ingress.test:31443/api\nThis is the API Server!\n\n➜ curl -kv https://secure-ingress.test:31443/api\n...\n* Server certificate:\n* subject: CN=secure-ingress.test; O=secure-ingress.test\n* start date: Sep 25 18:22:10 2020 GMT\n* expire date: Sep 20 18:22:10 2040 GMT\n* issuer: CN=secure-ingress.test; O=secure-ingress.test\n* SSL certificate verify result: self signed certificate (18), continuing anyway.\n...\n```\n\nWe can see that the provided certificate is now being used by the Ingress for TLS termination.\n\n \n\n## Question 16 | Docker Image Attack Surface\n\n#### Task weight: 7%\n\n \nUse context: `kubectl config use-context workload-prod`\n\n \n\nThere is a Deployment `image-verify` in Namespace `team-blue` which runs image `registry.killer.sh:5000/image-verify:v1`. DevSecOps has asked you to improve this image by:\n\n* Changing the base image to alpine:3.12\n* Not installing curl\n* Updating nginx to use the version constraint \u003e=1.18.0\n* Running the main process as user myuser\n\n**Do not** add any new lines to the Dockerfile, just edit existing ones. The file is located at `/opt/course/16/image/Dockerfile`.\n\nTag your version as `v2`. You can build, tag and push using:\n\n```sh\ncd /opt/course/16/image\npodman build -t registry.killer.sh:5000/image-verify:v2 .\npodman run registry.killer.sh:5000/image-verify:v2 # to test your changes\npodman push registry.killer.sh:5000/image-verify:v2\n```\n\nMake the Deployment use your updated image tag v2.\n\n \n\n#### Answer:\n\nWe should have a look at the Docker Image at first:\n\n```sh\ncd /opt/course/16/image\n\ncp Dockerfile Dockerfile.bak\n\nvim Dockerfile\n```\n\n```yaml\n# /opt/course/16/image/Dockerfile\nFROM alpine:3.4\nRUN apk update \u0026\u0026 apk add vim curl nginx=1.10.3-r0\nRUN addgroup -S myuser \u0026\u0026 adduser -S myuser -G myuser\nCOPY ./run.sh run.sh\nRUN [\"chmod\", \"+x\", \"./run.sh\"]\nUSER root\nENTRYPOINT [\"/bin/sh\", \"./run.sh\"]\n```\n\nVery simple Dockerfile which seems to execute a script `run.sh` :\n\n```sh\n# /opt/course/16/image/run.sh\nwhile true; do date; id; echo; sleep 1; done\n```\n\nSo it only outputs current date and credential information in a loop. We can see that output in the existing Deployment image-verify:\n\n```sh\n➜ k -n team-blue logs -f -l id=image-verify\nFri Sep 25 20:59:12 UTC 2020\nuid=0(root) gid=0(root) groups=0(root),1(bin),2(daemon),3(sys),4(adm),6(disk),10(wheel),11(floppy),20(dialout),26(tape),27(video)\n```\n\nWe see its running as root.\n\nNext we update the Dockerfile according to the requirements:\n\n```yaml\n# /opt/course/16/image/Dockerfile\n\n# change\nFROM alpine:3.12\n\n# change\nRUN apk update \u0026\u0026 apk add vim nginx\u003e=1.18.0\n\nRUN addgroup -S myuser \u0026\u0026 adduser -S myuser -G myuser\nCOPY ./run.sh run.sh\nRUN [\"chmod\", \"+x\", \"./run.sh\"]\n\n# change\nUSER myuser\n\nENTRYPOINT [\"/bin/sh\", \"./run.sh\"]\n```\n\nThen we build the new image:\n\n```sh\n➜ :/opt/course/16/image$ podman build -t registry.killer.sh:5000/image-verify:v2 .\n...\nSTEP 7/7: ENTRYPOINT [\"/bin/sh\", \"./run.sh\"]\nCOMMIT registry.killer.sh:5000/image-verify:v2\n--\u003e ceb8989101b\nSuccessfully tagged registry.killer.sh:5000/image-verify:v2\nceb8989101bccd9f6b9c3b4c6c75f6c3561f19a5b784edd1f1a36fa0fb34a9df\n```\n\nWe can then test our changes by running the container locally:\n\n```sh\n➜ :/opt/course/16/image$ podman run registry.killer.sh:5000/image-verify:v2 \nThu Sep 16 06:01:47 UTC 2021\nuid=101(myuser) gid=102(myuser) groups=102(myuser)\n\nThu Sep 16 06:01:48 UTC 2021\nuid=101(myuser) gid=102(myuser) groups=102(myuser)\n\nThu Sep 16 06:01:49 UTC 2021\nuid=101(myuser) gid=102(myuser) groups=102(myuser)\n```\n\nLooking good, so we push:\n\n```sh\n➜ :/opt/course/16/image$ podman push registry.killer.sh:5000/image-verify:v2\nGetting image source signatures\nCopying blob cd0853834d88 done \nCopying blob 5298d0709c3e skipped: already exists \nCopying blob e6688e911f15 done \nCopying blob dbc406096645 skipped: already exists \nCopying blob 98895ed393d9 done \nCopying config ceb8989101 done \nWriting manifest to image destination\nStoring signatures\n```\n\nAnd we update the Deployment to use the new image:\n\n```sh\nk -n team-blue edit deploy image-verify\n```\n\n```yaml\n# kubectl -n team-blue edit deploy image-verify\napiVersion: apps/v1\nkind: Deployment\nmetadata:\n...\nspec:\n...\n template:\n...\n spec:\n containers:\n - image: registry.killer.sh:5000/image-verify:v2 # change\n```\n\nAnd afterwards we can verify our changes by looking at the Pod logs:\n\n```sh\n➜ k -n team-blue logs -f -l id=image-verify\nFri Sep 25 21:06:55 UTC 2020\nuid=101(myuser) gid=102(myuser) groups=102(myuser)\n```\n\nAlso to verify our changes even further:\n\n```sh\n➜ k -n team-blue exec image-verify-55fbcd4c9b-x2flc -- curl\nOCI runtime exec failed: exec failed: container_linux.go:349: starting container process caused \"exec: \\\"curl\\\": executable file not found in $PATH\": unknown\ncommand terminated with exit code 126\n\n➜ k -n team-blue exec image-verify-55fbcd4c9b-x2flc -- nginx -v\nnginx version: nginx/1.18.0\n```\n\nAnother task solved.\n\n\n \n\n## Question 17 | Audit Log Policy\n\n#### Task weight: 7%\n\n \n\nUse context: `kubectl config use-context infra-prod`\n\n \n\nAudit Logging has been enabled in the cluster with an Audit Policy located at `/etc/kubernetes/audit/policy.yaml` on `cluster2-controlplane1`.\n\nChange the configuration so that only one backup of the logs is stored.\n\nAlter the Policy in a way that it only stores logs:\n\n* From Secret resources, level Metadata\n* From \"system:nodes\" userGroups, level RequestResponse\n\nAfter you altered the Policy make sure to empty the log file so it only contains entries according to your changes, like using `truncate -s 0 /etc/kubernetes/audit/logs/audit.log`.\n\n \n\n###### NOTE: You can use jq to render json more readable. cat data.json | jq\n \n\n#### Answer:\n\nFirst we check the apiserver configuration and change as requested:\n\n```sh\n➜ ssh cluster2-controlplane1\n\n➜ root@cluster2-controlplane1:~# cp /etc/kubernetes/manifests/kube-apiserver.yaml ~/17_kube-apiserver.yaml # backup\n\n➜ root@cluster2-controlplane1:~# vim /etc/kubernetes/manifests/kube-apiserver.yaml \n```\n```yaml\n# /etc/kubernetes/manifests/kube-apiserver.yaml \napiVersion: v1\nkind: Pod\nmetadata:\n annotations:\n kubeadm.kubernetes.io/kube-apiserver.advertise-address.endpoint: 192.168.100.21:6443\n creationTimestamp: null\n labels:\n component: kube-apiserver\n tier: control-plane\n name: kube-apiserver\n namespace: kube-system\nspec:\n containers:\n - command:\n - kube-apiserver\n - --audit-policy-file=/etc/kubernetes/audit/policy.yaml\n - --audit-log-path=/etc/kubernetes/audit/logs/audit.log\n - --audit-log-maxsize=5\n - --audit-log-maxbackup=1 # CHANGE\n - --advertise-address=192.168.100.21\n - --allow-privileged=true\n...\n```\n\n\n###### NOTE: You should know how to enable Audit Logging completely yourself as described in the docs. Feel free to try this in another cluster in this environment.\n\n \n\nNow we look at the existing Policy:\n\n```sh\n➜ root@cluster2-controlplane1:~# vim /etc/kubernetes/audit/policy.yaml\n```\n```yaml\n# /etc/kubernetes/audit/policy.yaml\napiVersion: audit.k8s.io/v1\nkind: Policy\nrules:\n- level: Metadata\n```\n\nWe can see that this simple Policy logs everything on Metadata level. So we change it to the requirements:\n```yaml\n# /etc/kubernetes/audit/policy.yaml\napiVersion: audit.k8s.io/v1\nkind: Policy\nrules:\n\n# log Secret resources audits, level Metadata\n- level: Metadata\n resources:\n - group: \"\"\n resources: [\"secrets\"]\n\n# log node related audits, level RequestResponse\n- level: RequestResponse\n userGroups: [\"system:nodes\"]\n\n# for everything else don't log anything\n- level: None\n```\n\nAfter saving the changes we have to restart the apiserver:\n\n```sh\n➜ root@cluster2-controlplane1:~# cd /etc/kubernetes/manifests/\n\n➜ root@cluster2-controlplane1:/etc/kubernetes/manifests# mv kube-apiserver.yaml ..\n\n➜ root@cluster2-controlplane1:/etc/kubernetes/manifests# watch crictl ps # wait for apiserver gone\n\n➜ root@cluster2-controlplane1:/etc/kubernetes/manifests# truncate -s 0 /etc/kubernetes/audit/logs/audit.log\n\n➜ root@cluster2-controlplane1:/etc/kubernetes/manifests# mv ../kube-apiserver.yaml .\n```\n\nOnce the apiserver is running again we can check the new logs and scroll through some entries:\n\n```sh\ncat audit.log | tail | jq\n```\n\n```json\n{\n \"kind\": \"Event\",\n \"apiVersion\": \"audit.k8s.io/v1\",\n \"level\": \"Metadata\",\n \"auditID\": \"e598dc9e-fc8b-4213-aee3-0719499ab1bd\",\n \"stage\": \"RequestReceived\",\n \"requestURI\": \"...\",\n \"verb\": \"watch\",\n \"user\": {\n \"username\": \"system:serviceaccount:gatekeeper-system:gatekeeper-admin\",\n \"uid\": \"79870838-75a8-479b-ad42-4b7b75bd17a3\",\n \"groups\": [\n \"system:serviceaccounts\",\n \"system:serviceaccounts:gatekeeper-system\",\n \"system:authenticated\"\n ]\n },\n \"sourceIPs\": [\n \"192.168.102.21\"\n ],\n \"userAgent\": \"manager/v0.0.0 (linux/amd64) kubernetes/$Format\",\n \"objectRef\": {\n \"resource\": \"secrets\",\n \"apiVersion\": \"v1\"\n },\n \"requestReceivedTimestamp\": \"2020-09-27T20:01:36.238911Z\",\n \"stageTimestamp\": \"2020-09-27T20:01:36.238911Z\",\n \"annotations\": {\n \"authentication.k8s.io/legacy-token\": \"...\"\n }\n}\n```\n\nAbove we logged a watch action by OPA Gatekeeper for Secrets, level Metadata.\n\n```json\n{\n \"kind\": \"Event\",\n \"apiVersion\": \"audit.k8s.io/v1\",\n \"level\": \"RequestResponse\",\n \"auditID\": \"c90e53ed-b0cf-4cc4-889a-f1204dd39267\",\n \"stage\": \"ResponseComplete\",\n \"requestURI\": \"...\",\n \"verb\": \"list\",\n \"user\": {\n \"username\": \"system:node:cluster2-controlplane1\",\n \"groups\": [\n \"system:nodes\",\n \"system:authenticated\"\n ]\n },\n \"sourceIPs\": [\n \"192.168.100.21\"\n ],\n \"userAgent\": \"kubelet/v1.19.1 (linux/amd64) kubernetes/206bcad\",\n \"objectRef\": {\n \"resource\": \"configmaps\",\n \"namespace\": \"kube-system\",\n \"name\": \"kube-proxy\",\n \"apiVersion\": \"v1\"\n },\n \"responseStatus\": {\n \"metadata\": {},\n \"code\": 200\n },\n \"responseObject\": {\n \"kind\": \"ConfigMapList\",\n \"apiVersion\": \"v1\",\n \"metadata\": {\n \"selfLink\": \"/api/v1/namespaces/kube-system/configmaps\",\n \"resourceVersion\": \"83409\"\n },\n \"items\": [\n {\n \"metadata\": {\n \"name\": \"kube-proxy\",\n \"namespace\": \"kube-system\",\n \"selfLink\": \"/api/v1/namespaces/kube-system/configmaps/kube-proxy\",\n \"uid\": \"0f1c3950-430a-4543-83e4-3f9c87a478b8\",\n \"resourceVersion\": \"232\",\n \"creationTimestamp\": \"2020-09-26T20:59:50Z\",\n \"labels\": {\n \"app\": \"kube-proxy\"\n },\n \"annotations\": {\n \"kubeadm.kubernetes.io/component-config.hash\": \"...\"\n },\n \"managedFields\": [\n {\n...\n }\n ]\n },\n...\n }\n ]\n },\n \"requestReceivedTimestamp\": \"2020-09-27T20:01:36.223781Z\",\n \"stageTimestamp\": \"2020-09-27T20:01:36.225470Z\",\n \"annotations\": {\n \"authorization.k8s.io/decision\": \"allow\",\n \"authorization.k8s.io/reason\": \"\"\n }\n}\n```\n\nAnd in the one above we logged a list action by system:nodes for a ConfigMaps, level RequestResponse.\n\nBecause all JSON entries are written in a single line in the file we could also run some simple verifications on our Policy:\n\n```yaml\n# shows Secret entries\ncat audit.log | grep '\"resource\":\"secrets\"' | wc -l\n\n# confirms Secret entries are only of level Metadata\ncat audit.log | grep '\"resource\":\"secrets\"' | grep -v '\"level\":\"Metadata\"' | wc -l\n\n# shows RequestResponse level entries\ncat audit.log | grep -v '\"level\":\"RequestResponse\"' | wc -l\n\n# shows RequestResponse level entries are only for system:nodes\ncat audit.log | grep '\"level\":\"RequestResponse\"' | grep -v \"system:nodes\" | wc -l\n```\n\nLooks like our job is done.\n\n\n \n\n## Question 18 | Investigate Break-in via Audit Log\n\n#### Task weight: 4%\n\n \n\nUse context: `kubectl config use-context infra-prod`\n\n \n\nNamespace security contains five Secrets of type Opaque which can be considered highly confidential. The latest Incident-Prevention-Investigation revealed that ServiceAccount p.auster had too broad access to the cluster for some time. This SA should've never had access to any Secrets in that Namespace.\n\nFind out which Secrets in Namespace security this SA did access by looking at the Audit Logs under /opt/course/18/audit.log.\n\nChange the password to any new string of only those Secrets that were accessed by this SA.\n\n \n\nNOTE: You can use jq to render json more readable. cat data.json | jq\n\n \n\n \n\nAnswer:\nFirst we look at the Secrets this is about:\n\n➜ k -n security get secret | grep Opaque\nkubeadmin-token Opaque 1 37m\nmysql-admin Opaque 1 37m\npostgres001 Opaque 1 37m\npostgres002 Opaque 1 37m\nvault-token Opaque 1 37m\nNext we investigate the Audit Log file:\n\n➜ cd /opt/course/18\n\n➜ :/opt/course/18$ ls -lh\ntotal 7.1M\n-rw-r--r-- 1 k8s k8s 7.5M Sep 24 21:31 audit.log\n\n➜ :/opt/course/18$ cat audit.log | wc -l\n4451\nAudit Logs can be huge and it's common to limit the amount by creating an Audit Policy and to transfer the data in systems like Elasticsearch. In this case we have a simple JSON export, but it already contains 4451 lines.\n\nWe should try to filter the file down to relevant information:\n\n➜ :/opt/course/18$ cat audit.log | grep \"p.auster\" | wc -l\n28\nNot too bad, only 28 logs for ServiceAccount p.auster.\n\n➜ :/opt/course/18$ cat audit.log | grep \"p.auster\" | grep Secret | wc -l\n2\nAnd only 2 logs related to Secrets...\n\n➜ :/opt/course/18$ cat audit.log | grep \"p.auster\" | grep Secret | grep list | wc -l\n0\n\n➜ :/opt/course/18$ cat audit.log | grep \"p.auster\" | grep Secret | grep get | wc -l\n2\nNo list actions, which is good, but 2 get actions, so we check these out:\n\ncat audit.log | grep \"p.auster\" | grep Secret | grep get | jq\n{\n \"kind\": \"Event\",\n \"apiVersion\": \"audit.k8s.io/v1\",\n \"level\": \"RequestResponse\",\n \"auditID\": \"74fd9e03-abea-4df1-b3d0-9cfeff9ad97a\",\n \"stage\": \"ResponseComplete\",\n \"requestURI\": \"/api/v1/namespaces/security/secrets/vault-token\",\n \"verb\": \"get\",\n \"user\": {\n \"username\": \"system:serviceaccount:security:p.auster\",\n \"uid\": \"29ecb107-c0e8-4f2d-816a-b16f4391999c\",\n \"groups\": [\n \"system:serviceaccounts\",\n \"system:serviceaccounts:security\",\n \"system:authenticated\"\n ]\n },\n...\n \"userAgent\": \"curl/7.64.0\",\n \"objectRef\": {\n \"resource\": \"secrets\",\n \"namespace\": \"security\",\n \"name\": \"vault-token\",\n \"apiVersion\": \"v1\"\n },\n ...\n}\n{\n \"kind\": \"Event\",\n \"apiVersion\": \"audit.k8s.io/v1\",\n \"level\": \"RequestResponse\",\n \"auditID\": \"aed6caf9-5af0-4872-8f09-ad55974bb5e0\",\n \"stage\": \"ResponseComplete\",\n \"requestURI\": \"/api/v1/namespaces/security/secrets/mysql-admin\",\n \"verb\": \"get\",\n \"user\": {\n \"username\": \"system:serviceaccount:security:p.auster\",\n \"uid\": \"29ecb107-c0e8-4f2d-816a-b16f4391999c\",\n \"groups\": [\n \"system:serviceaccounts\",\n \"system:serviceaccounts:security\",\n \"system:authenticated\"\n ]\n },\n...\n \"userAgent\": \"curl/7.64.0\",\n \"objectRef\": {\n \"resource\": \"secrets\",\n \"namespace\": \"security\",\n \"name\": \"mysql-admin\",\n \"apiVersion\": \"v1\"\n },\n...\n}\nThere we see that Secrets vault-token and mysql-admin were accessed by p.auster. Hence we change the passwords for those.\n\n➜ echo new-vault-pass | base64\nbmV3LXZhdWx0LXBhc3MK\n\n➜ k -n security edit secret vault-token\n\n➜ echo new-mysql-pass | base64\nbmV3LW15c3FsLXBhc3MK\n\n➜ k -n security edit secret mysql-admin\nAudit Logs ftw.\n\nBy running cat audit.log | grep \"p.auster\" | grep Secret | grep password we can see that passwords are stored in the Audit Logs, because they store the complete content of Secrets. It's never a good idea to reveal passwords in logs. In this case it would probably be sufficient to only store Metadata level information of Secrets which can be controlled via a Audit Policy.\n\n \n\n \n\n## Question 19 | Immutable Root FileSystem\n\n#### Task weight: 2%\n \n\nUse context: `kubectl config use-context workload-prod`\n\n \n\nThe Deployment `immutable-deployment` in Namespace `team-purple` should run immutable, it's created from file `/opt/course/19/immutable-deployment.yaml`. Even after a successful break-in, it shouldn't be possible for an attacker to modify the filesystem of the running container.\n\nModify the Deployment in a way that no processes inside the container can modify the local filesystem, only `/tmp` directory should be writeable. Don't modify the Docker image.\n\nSave the updated YAML under `/opt/course/19/immutable-deployment-new.yaml` and update the running Deployment.\n\n \n\n#### Answer:\n\nProcesses in containers can write to the local filesystem by default. This increases the attack surface when a non-malicious process gets hijacked. Preventing applications to write to disk or only allowing to certain directories can mitigate the risk. If there is for example a bug in Nginx which allows an attacker to override any file inside the container, then this only works if the Nginx process itself can write to the filesystem in the first place.\n\nMaking the root filesystem readonly can be done in the Docker image itself or in a Pod declaration.\n\nLet us first check the Deployment `immutable-deployment` in Namespace `team-purple`:\n\n```sh\n➜ k -n team-purple edit deploy -o yaml\n```\n\n```yaml\n# kubectl -n team-purple edit deploy -o yaml\napiVersion: apps/v1\nkind: Deployment\nmetadata:\n namespace: team-purple\n name: immutable-deployment\n labels:\n app: immutable-deployment\n ...\nspec:\n replicas: 1\n selector:\n matchLabels:\n app: immutable-deployment\n template:\n metadata:\n labels:\n app: immutable-deployment\n spec:\n containers:\n - image: busybox:1.32.0\n command: ['sh', '-c', 'tail -f /dev/null']\n imagePullPolicy: IfNotPresent\n name: busybox\n restartPolicy: Always\n...\n```\n\nThe container has write access to the Root File System, as there are no restrictions defined for the Pods or containers by an existing SecurityContext. And based on the task we're not allowed to alter the Docker image.\n\nSo we modify the YAML manifest to include the required changes:\n\n```sh\ncp /opt/course/19/immutable-deployment.yaml /opt/course/19/immutable-deployment-new.yaml\n\nvim /opt/course/19/immutable-deployment-new.yaml\n```\n\n```yaml\n# /opt/course/19/immutable-deployment-new.yaml\napiVersion: apps/v1\nkind: Deployment\nmetadata:\n namespace: team-purple\n name: immutable-deployment\n labels:\n app: immutable-deployment\nspec:\n replicas: 1\n selector:\n matchLabels:\n app: immutable-deployment\n template:\n metadata:\n labels:\n app: immutable-deployment\n spec:\n containers:\n - image: busybox:1.32.0\n command: ['sh', '-c', 'tail -f /dev/null']\n imagePullPolicy: IfNotPresent\n name: busybox\n securityContext: # add\n readOnlyRootFilesystem: true # add\n volumeMounts: # add\n - mountPath: /tmp # add\n name: temp-vol # add\n volumes: # add\n - name: temp-vol # add\n emptyDir: {} # add\n restartPolicy: Always\n \n```\n\nSecurityContexts can be set on Pod or container level, here the latter was asked. Enforcing readOnlyRootFilesystem: true will render the root filesystem readonly. We can then allow some directories to be writable by using an emptyDir volume.\n\nOnce the changes are made, let us update the Deployment:\n\n```sh\n➜ k delete -f /opt/course/19/immutable-deployment-new.yaml\ndeployment.apps \"immutable-deployment\" deleted\n\n➜ k create -f /opt/course/19/immutable-deployment-new.yaml\ndeployment.apps/immutable-deployment created\n```\n\nWe can verify if the required changes are propagated:\n\n```sh\n➜ k -n team-purple exec immutable-deployment-5b7ff8d464-j2nrj -- touch /abc.txt\ntouch: /abc.txt: Read-only file system\ncommand terminated with exit code 1\n\n➜ k -n team-purple exec immutable-deployment-5b7ff8d464-j2nrj -- touch /var/abc.txt\ntouch: /var/abc.txt: Read-only file system\ncommand terminated with exit code 1\n\n➜ k -n team-purple exec immutable-deployment-5b7ff8d464-j2nrj -- touch /etc/abc.txt\ntouch: /etc/abc.txt: Read-only file system\ncommand terminated with exit code 1\n\n➜ k -n team-purple exec immutable-deployment-5b7ff8d464-j2nrj -- touch /tmp/abc.txt\n\n➜ k -n team-purple exec immutable-deployment-5b7ff8d464-j2nrj -- ls /tmp\nabc.txt\n```\n\nThe Deployment has been updated so that the container's file system is read-only, and the updated YAML has been placed under the required location. Sweet!\n\n\n \n\n## Question 20 | Update Kubernetes\n\n\n#### Task weight: 8%\n\n \n\nUse context: `kubectl config use-context workload-stage`\n\n \n\nThe cluster is running Kubernetes `1.24.7`, update it to `1.25.2`.\n\nUse `apt` package manager and `kubeadm` for this.\n\nUse `ssh cluster3-controlplane1` and `ssh cluster3-node1` to connect to the instances.\n\n \n\n#### Answer:\n\nLet's have a look at the current versions:\n\n```sh\n➜ k get node\nNAME STATUS ROLES AGE VERSION\ncluster3-controlplane1 Ready control-plane 21d v1.24.7\ncluster3-node1 Ready \u003cnone\u003e 21d v1.24.7\n``` \n\n###### Control Plane Master Components\n\nFirst we should update the control plane components running on the master node, so we drain it:\n\n```sh\n➜ k drain cluster3-controlplane1 --ignore-daemonsets\nNext we ssh into it and check versions:\n\n➜ ssh cluster3-controlplane1\n\n➜ root@cluster3-controlplane1:~# kubeadm version\nkubeadm version: \u0026version.Info{Major:\"1\", Minor:\"24\", GitVersion:\"v1.24.1\", GitCommit:\"3ddd0f45aa91e2f30c70734b175631bec5b5825a\", GitTreeState:\"clean\", BuildDate:\"2022-05-24T12:24:38Z\", GoVersion:\"go1.18.2\", Compiler:\"gc\", Platform:\"linux/amd64\"}\n\n➜ root@cluster3-controlplane1:~# kubelet --version\nKubernetes v1.23.1\n```\n\nWe see `kubeadm` is already installed in the required version. Else we would need to install it:\n\n```sh\n# not necessary because here kubeadm is already installed in correct version\napt-mark unhold kubeadm\napt-mark hold kubectl kubelet\napt install kubeadm=1.24.1-00\napt-mark hold kubeadm\n```\n\nCheck what kubeadm has available as an upgrade plan:\n\n```sh\n➜ root@cluster3-controlplane1:~# kubeadm upgrade plan\n...\n[upgrade/config] Making sure the configuration is correct:\n[upgr","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsnigdhasambitak%2Fcks","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fsnigdhasambitak%2Fcks","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsnigdhasambitak%2Fcks/lists"}