{"id":28168961,"url":"https://github.com/ruzickap/cks-notes","last_synced_at":"2025-10-15T07:43:24.970Z","repository":{"id":45640795,"uuid":"415237279","full_name":"ruzickap/cks-notes","owner":"ruzickap","description":"CKS Notes","archived":false,"fork":false,"pushed_at":"2025-05-04T05:56:20.000Z","size":39,"stargazers_count":2,"open_issues_count":1,"forks_count":2,"subscribers_count":2,"default_branch":"main","last_synced_at":"2025-05-04T06:18:33.083Z","etag":null,"topics":["cks","cks-exam","cks-exam-preparation","k8s","k8s-security","kubernetes","public","security"],"latest_commit_sha":null,"homepage":"","language":null,"has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/ruzickap.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2021-10-09T07:33:00.000Z","updated_at":"2025-01-27T05:44:05.000Z","dependencies_parsed_at":"2022-09-20T21:50:38.680Z","dependency_job_id":null,"html_url":"https://github.com/ruzickap/cks-notes","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ruzickap%2Fcks-notes","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ruzickap%2Fcks-notes/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ruzickap%2Fcks-notes/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ruzickap%2Fcks-notes/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/ruzickap","download_url":"https://codeload.github.com/ruzickap/cks-notes/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":254364222,"owners_count":22058880,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["cks","cks-exam","cks-exam-preparation","k8s","k8s-security","kubernetes","public","security"],"created_at":"2025-05-15T15:14:59.708Z","updated_at":"2025-10-15T07:43:24.959Z","avatar_url":"https://github.com/ruzickap.png","language":null,"funding_links":[],"categories":[],"sub_categories":[],"readme":"# CKS Notes\n\nHandy notes can also be found here: [https://github.com/dragon7-fc/misc/tree/1385c4a2e4719b9aa914c3b274c2877f7305d11e](https://github.com/dragon7-fc/misc/tree/1385c4a2e4719b9aa914c3b274c2877f7305d11e)\n\n## Test k8s cluster using Vagrant\n\nPrepare the test environment – a Kubernetes cluster with one master node and one\nworker node using Vagrant + VirtualBox:\n\n```bash\ngit clone git@github.com:kodekloudhub/certified-kubernetes-administrator-course.git\n\ncd certified-kubernetes-administrator-course || exit\nsed -i 's/NUM_WORKER_NODE = 2/NUM_WORKER_NODE = 1/' Vagrantfile\n\nvagrant up\n```\n\nThe installation steps are based on: [install_master.sh](https://github.com/killer-sh/cks-course-environment/blob/master/cluster-setup/latest/install_master.sh)\n\nExecute the following commands on both Kubernetes nodes (`kubemaster` and `kubenode01`):\n\n```bash\n# vagrant ssh kubemaster\n# vagrant ssh kubenode01\n\nKUBE_VERSION=1.21.5\n\nsudo curl -fsSLo /usr/share/keyrings/kubernetes-archive-keyring.gpg \\\n  https://packages.cloud.google.com/apt/doc/apt-key.gpg\necho \"deb [signed-by=/usr/share/keyrings/kubernetes-archive-keyring.gpg] \\\n  https://apt.kubernetes.io/ kubernetes-xenial main\" |\n  sudo tee /etc/apt/sources.list.d/kubernetes.list\n\nsudo apt-get update\nsudo apt-get install -y apparmor-utils apt-transport-https ca-certificates \\\n  containerd curl docker.io etcd-client jq lsb-release mc strace tree\n\ncat \u003c\u003c EOF | sudo tee /etc/modules-load.d/containerd.conf\noverlay\nbr_netfilter\nEOF\nsudo modprobe overlay\nsudo modprobe br_netfilter\ncat \u003c\u003c EOF | sudo tee /etc/sysctl.d/k8s.conf\nnet.bridge.bridge-nf-call-iptables  = 1\nnet.ipv4.ip_forward                 = 1\nnet.bridge.bridge-nf-call-ip6tables = 1\nEOF\n\nsudo sysctl --system\n\nsudo mkdir -p /etc/containerd\ncontainerd config default |\n  sed 's/SystemdCgroup = false/SystemdCgroup = true/' |\n  sudo tee /etc/containerd/config.toml\nsudo systemctl restart containerd\n\ncat \u003c\u003c EOF | sudo tee /etc/crictl.yaml\nruntime-endpoint: unix:///run/containerd/containerd.sock\nEOF\n\nsudo usermod -aG docker \"${USER}\"\n\nsudo systemctl daemon-reload\nsudo systemctl enable containerd docker\nsudo systemctl restart containerd\n\nsudo apt-get install -y \\\n  kubelet=\"${KUBE_VERSION}-00\" \\\n  kubeadm=\"${KUBE_VERSION}-00\" \\\n  kubectl=\"${KUBE_VERSION}-00\"\nsudo apt-mark hold kubelet kubeadm kubectl\n\ncat \u003e\u003e ~/.bashrc \u003c\u003c EOF\nsource \u003c(kubectl completion bash)\nalias k=kubectl\ncomplete -F __start_kubectl k\nEOF\n```\n\nExecute the following commands on the **master** node (`kubemaster`) only:\n\n```bash\n# vagrant ssh kubemaster\n\nsudo kubeadm init \\\n  --cri-socket /run/containerd/containerd.sock \\\n  --kubernetes-version=\"${KUBE_VERSION}\" \\\n  --pod-network-cidr=10.224.0.0/16 \\\n  --apiserver-advertise-address=192.168.56.2 \\\n  --skip-token-print\n\nmkdir -p \"${HOME}/.kube\"\nsudo cp -i /etc/kubernetes/admin.conf \"${HOME}/.kube/config\"\nsudo chown \"$(id -u):$(id -g)\" \"${HOME}/.kube/config\"\n\ncurl -LO https://github.com/cilium/cilium-cli/releases/latest/download/cilium-linux-amd64.tar.gz \u0026\u0026\n  sudo tar xzvfC cilium-linux-amd64.tar.gz /usr/local/bin \u0026\u0026\n  rm cilium-linux-amd64.tar.gz\ncilium install\n\nkubeadm token create --print-join-command --ttl 0\n```\n\nExecute the following commands on the **worker** node (`kubenode01`) only:\n\n```bash\n# vagrant ssh kubenode01\nsudo kubeadm join --cri-socket /run/containerd/containerd.sock \\\n  --kubernetes-version=${KUBE_VERSION} 192.168.56.2:6443 \\\n  --token i0sn6a.jnvsbw73yi03nre7 \\\n  --discovery-token-ca-cert-hash sha256:ffa3c5c3cd8ee55bd9497e8d6d9556d3bcef7b0879f871a088819f232c4673e0\n```\n\n## Kubernetes certificates\n\n* `/etc/kubernetes/pki`\n* `/var/lib/kubelet/pki`\n\n```console\n$ sudo tree /etc/kubernetes/pki\n/etc/kubernetes/pki\n├── apiserver-etcd-client.crt\n├── apiserver-etcd-client.key\n├── apiserver-kubelet-client.crt\n├── apiserver-kubelet-client.key\n├── apiserver.crt\n├── apiserver.key\n├── ca.crt\n├── ca.key\n├── etcd\n│   ├── ca.crt\n│   ├── ca.key\n│   ├── healthcheck-client.crt\n│   ├── healthcheck-client.key\n│   ├── peer.crt\n│   ├── peer.key\n│   ├── server.crt\n│   └── server.key\n├── front-proxy-ca.crt\n├── front-proxy-ca.key\n├── front-proxy-client.crt\n├── front-proxy-client.key\n├── sa.key\n└── sa.pub\n\n$ sudo tree /var/lib/kubelet/pki\n/var/lib/kubelet/pki\n├── kubelet-client-2021-10-09-07-44-38.pem\n├── kubelet-client-current.pem -\u003e /var/lib/kubelet/pki/kubelet-client-2021-10-09-07-44-38.pem\n├── kubelet.crt\n└── kubelet.key\n```\n\n## Docker namespace\n\n```bash\ndocker run --name cont1 -d ubuntu sh -c \"sleep 1d\"\ndocker run --name cont2 --pid=container:cont1 -d ubuntu sh -c \"sleep 111d\"\n```\n\n```console\n$ docker exec cont2 ps -elf\nF S UID        PID  PPID  C PRI  NI ADDR SZ WCHAN  STIME TTY          TIME CMD\n4 S root         1     0  0  80   0 -   653 do_wai 09:59 ?        00:00:00 sh -c sleep 1d\n0 S root         7     1  0  80   0 -   628 hrtime 09:59 ?        00:00:00 sleep 1d\n4 S root         8     0  0  80   0 -   653 do_wai 09:59 ?        00:00:00 sh -c sleep 111d\n0 S root        15     8  0  80   0 -   628 hrtime 09:59 ?        00:00:00 sleep 111d\n4 R root        16     0  0  80   0 -  1475 -      10:00 ?        00:00:00 ps -elf\n\n$ docker exec cont1 ps -elf\nF S UID        PID  PPID  C PRI  NI ADDR SZ WCHAN  STIME TTY          TIME CMD\n4 S root         1     0  0  80   0 -   653 do_wai 09:59 ?        00:00:00 sh -c sleep 1d\n0 S root         7     1  0  80   0 -   628 hrtime 09:59 ?        00:00:00 sleep 1d\n4 S root         8     0  0  80   0 -   653 do_wai 09:59 ?        00:00:00 sh -c sleep 111d\n0 S root        15     8  0  80   0 -   628 hrtime 09:59 ?        00:00:00 sleep 111d\n4 R root        22     0  0  80   0 -  1475 -      10:00 ?        00:00:00 ps -elf\n```\n\n## Network Policies\n\n* [Network Policies](https://kubernetes.io/docs/concepts/services-networking/network-policies/)\n\nRun two test pods + services:\n\n```bash\nkubectl run frontend --image=nginx --port=80 --expose=true\nkubectl run backend --image=nginx --port=80 --expose=true\nkubectl create namespace database\nkubectl label namespace database ns=database\nkubectl run -n database database --image=nginx --port=80 --expose=true\n```\n\nCheck connectivity:\n\n```bash\nkubectl exec frontend -- curl -s backend\nkubectl exec backend -- curl -s frontend\nkubectl exec backend -- curl -s database.database.svc.cluster.local\n```\n\nDeny all policy:\n\n```bash\nkubectl apply -f - \u003c\u003c EOF\napiVersion: networking.k8s.io/v1\nkind: NetworkPolicy\nmetadata:\n  name: default-deny\n  namespace: default\nspec:\n  podSelector: {}\n  policyTypes:\n  - Ingress\n  - Egress\nEOF\n```\n\nAllow connection between `frontend` -\u003e `backend` (and DNS):\n\n```bash\nkubectl apply -f - \u003c\u003c EOF\napiVersion: networking.k8s.io/v1\nkind: NetworkPolicy\nmetadata:\n  name: frontend\n  namespace: default\nspec:\n  podSelector:\n    matchLabels:\n      run: frontend\n  policyTypes:\n  - Egress\n  egress:\n  - to:\n    - podSelector:\n        matchLabels:\n          run: backend\n  # DNS - allow by default\n  - to:\n    ports:\n      - port: 53\n        protocol: UDP\n      - port: 53\n        protocol: TCP\n---\napiVersion: networking.k8s.io/v1\nkind: NetworkPolicy\nmetadata:\n  name: backend\n  namespace: default\nspec:\n  podSelector:\n    matchLabels:\n      run: backend\n  policyTypes:\n  - Ingress\n  - Egress\n  ingress:\n  - from:\n    - podSelector:\n        matchLabels:\n          run: frontend\n  egress:\n  - to:\n    - namespaceSelector:\n        matchLabels:\n          ns: database\n  # DNS - allow by default\n  - to:\n    ports:\n      - port: 53\n        protocol: UDP\n      - port: 53\n        protocol: TCP\nEOF\n```\n\nThe following command will work because there are no NetworkPolicies in the\ndatabase namespace.\n\n```bash\nkubectl exec backend -- curl -s database.database.svc.cluster.local\n```\n\n## Kubernetes Dashboard\n\n* [Dashboard arguments](https://github.com/kubernetes/dashboard/blob/0dc322e21320f8678271a117812f3922f4965e23/docs/common/dashboard-arguments.md)\n\n\u003e Do not do this in your production environment.\n\nInstall:\n\n```bash\nkubectl apply -f https://raw.githubusercontent.com/kubernetes/dashboard/v2.3.1/aio/deploy/recommended.yaml\n```\n\nCreate `ClusterRoleBinding`:\n\n```bash\nkubectl create clusterrolebinding kubernetes-dashboard-view-all --serviceaccount kubernetes-dashboard:kubernetes-dashboard --clusterrole view\n```\n\nConfigure `NodePort` \"access\":\n\n```bash\nkubectl patch service -n kubernetes-dashboard kubernetes-dashboard --type='json' -p '[{\"op\":\"replace\",\"path\":\"/spec/type\",\"value\":\"NodePort\"}]'\n```\n\nEdit configuration by running\n`kubectl edit deploy -n kubernetes-dashboard kubernetes-dashboard`:\n\n```yaml\napiVersion: apps/v1\nkind: Deployment\nmetadata:\n  name: kubernetes-dashboard\n  namespace: kubernetes-dashboard\n...\nspec:\n...\n  template:\n    spec:\n      containers:\n      - args:\n        # - --auto-generate-certificates     # Allow use port 9090 for insecure HTTP connections\n        - --namespace=kubernetes-dashboard\n        - --authentication-mode=basic        # Enable basic authentication\n        - --enable-skip-login=true           # Enable \"skip button\" on the login page will be shown\n        - --enable-insecure-login            # Enable Dashboard login when using HTTP\n...\n```\n\nYou should be able to access the Kubernetes Dashboard at [http://192.168.56.2:32645](http://192.168.56.2:32645):\n\n```bash\ncurl -k http://192.168.56.2:32645\n```\n\n## Ingress\n\nInstall `ingress-nginx` and delete \"NetworkPolicies\" + \"Services\" + \"Pods\":\n\n```bash\nkubectl apply -f \\\n  https://raw.githubusercontent.com/kubernetes/ingress-nginx/controller-v1.0.3/deploy/static/provider/baremetal/deploy.yaml\nkubectl delete networkpolicies,service,pods --all\n```\n\n```console\n$ kubectl get pods,service -n ingress-nginx\nNAME                                            READY   STATUS              RESTARTS   AGE\npod/ingress-nginx-admission-create--1-hb24s     0/1     Completed           0          13s\npod/ingress-nginx-admission-patch--1-znw8z      0/1     Completed           0          13s\npod/ingress-nginx-controller-6c68f5b657-wzbx8   0/1     ContainerCreating   0          13s\n\nNAME                                         TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)                      AGE\nservice/ingress-nginx-controller             NodePort    10.96.253.251   \u003cnone\u003e        80:32550/TCP,443:31606/TCP   13s\nservice/ingress-nginx-controller-admission   ClusterIP   10.100.6.131    \u003cnone\u003e        443/TCP                      13s\n```\n\nVerify that `ingress-nginx` is running by executing the following command\n(a 404 error is expected at this stage):\n\n```bash\ncurl -kv http://192.168.56.2:32550 https://192.168.56.2:31606\n```\n\nStart two applications:\n\n```bash\nkubectl run app1 --image=ghcr.io/stefanprodan/podinfo:6.0.0 --port=9898 \\\n  --expose=true --env=\"PODINFO_UI_MESSAGE=app1\"\nkubectl run app2 --image=ghcr.io/stefanprodan/podinfo:6.0.0 --port=9898 \\\n  --expose=true --env=\"PODINFO_UI_MESSAGE=app2\"\n```\n\n```bash\nkubectl apply -f - \u003c\u003c EOF\napiVersion: networking.k8s.io/v1\nkind: Ingress\nmetadata:\n  name: my-ingress\n  annotations:\n    nginx.ingress.kubernetes.io/rewrite-target: /\nspec:\n  ingressClassName: \"nginx\"\n  rules:\n  - http:\n      paths:\n      - path: /app1\n        pathType: Prefix\n        backend:\n          service:\n            name: app1\n            port:\n              number: 9898\n      - path: /app2\n        pathType: Prefix\n        backend:\n          service:\n            name: app2\n            port:\n              number: 9898\nEOF\n```\n\nYou should be able to reach the Ingress and the backend services, which will\nprovide different responses:\n\n```console\n$ curl -sk https://192.168.56.2:31606/app1 | jq\n{\n  \"hostname\": \"app1\",\n  \"version\": \"6.0.0\",\n  \"revision\": \"\",\n  \"color\": \"#34577c\",\n  \"logo\": \"https://raw.githubusercontent.com/stefanprodan/podinfo/gh-pages/cuddle_clap.gif\",\n  \"message\": \"app1\",\n  \"goos\": \"linux\",\n  \"goarch\": \"amd64\",\n  \"runtime\": \"go1.16.5\",\n  \"num_goroutine\": \"6\",\n  \"num_cpu\": \"2\"\n}\n\n$ curl -sk https://192.168.56.2:31606/app2 | jq\n{\n  \"hostname\": \"app2\",\n  \"version\": \"6.0.0\",\n  \"revision\": \"\",\n  \"color\": \"#7c4134\",\n  \"logo\": \"https://raw.githubusercontent.com/stefanprodan/podinfo/gh-pages/cuddle_clap.gif\",\n  \"message\": \"app2\",\n  \"goos\": \"linux\",\n  \"goarch\": \"amd64\",\n  \"runtime\": \"go1.16.5\",\n  \"num_goroutine\": \"6\",\n  \"num_cpu\": \"2\"\n}\n```\n\n### Ingress - certificate\n\nThe Ingress controller uses a self-signed certificate by default:\n\n```console\n$ curl -sk https://192.168.56.2:31606/app2\n...\n* Server certificate:\n*  subject: O=Acme Co; CN=Kubernetes Ingress Controller Fake Certificate\n...\n```\n\nGenerate new cert:\n\n```bash\nopenssl req -x509 -nodes -days 365 -newkey rsa:2048 \\\n  -keyout key.pem -out cert.pem \\\n  -subj \"/C=CZ/ST=Czech/L=Prague/O=IT/OU=DevOps/CN=my-secure-ingress.k8s.cluster.com\"\n```\n\nCreate k8s secret:\n\n```bash\nkubectl create secret tls tls-secret --cert=cert.pem --key=key.pem\n```\n\nUpdate ingress:\n\n```bash\nkubectl apply -f - \u003c\u003c EOF\napiVersion: networking.k8s.io/v1\nkind: Ingress\nmetadata:\n  name: my-ingress\n  annotations:\n    nginx.ingress.kubernetes.io/rewrite-target: /\nspec:\n  ingressClassName: \"nginx\"\n  tls:\n  - hosts:\n      - my-secure-ingress.k8s.cluster.com\n    secretName: tls-secret\n  rules:\n  - host: my-secure-ingress.k8s.cluster.com\n    http:\n      paths:\n      - path: /app1\n        pathType: Prefix\n        backend:\n          service:\n            name: app1\n            port:\n              number: 9898\n      - path: /app2\n        pathType: Prefix\n        backend:\n          service:\n            name: app2\n            port:\n              number: 9898\nEOF\n```\n\n```console\n$ curl -kv https://my-secure-ingress.k8s.cluster.com:31606/app2 --resolve my-secure-ingress.k8s.cluster.com:31606:192.168.56.2\n...\n* Server certificate:\n*  subject: C=CZ; ST=Czech; L=Prague; O=IT; OU=DevOps; CN=my-secure-ingress.k8s.cluster.com\n...\n```\n\n## CIS benchmarks\n\n* [CIS Kubernetes Benchmark v1.6.0 - 07-23-2020](https://github.com/cismirror/old-benchmarks-archive/blob/master/CIS_Kubernetes_Benchmark_v1.6.0.pdf)\n\n### kube-bench\n\n* [kube-bench](https://github.com/aquasecurity/kube-bench)\n\n```bash\ndocker run --pid=host -v /etc:/etc:ro -v /var:/var:ro \\\n  -v \"$(which kubectl):/usr/local/mount-from-host/bin/kubectl\" \\\n  -v ~/.kube:/.kube -e KUBECONFIG=/.kube/config \\\n  -t aquasec/kube-bench:latest master\ndocker run --pid=host -v /etc:/etc:ro -v /var:/var:ro \\\n  -v \"$(which kubectl):/usr/local/mount-from-host/bin/kubectl\" \\\n  -v ~/.kube:/.kube -e KUBECONFIG=/.kube/config \\\n  -t aquasec/kube-bench:latest node\n```\n\nYou can install it locally:\n\n```bash\nwget https://github.com/aquasecurity/kube-bench/releases/download/v0.6.5/kube-bench_0.6.5_linux_amd64.deb\napt install ./kube-bench_0.6.5_linux_amd64.deb\n```\n\n## Hashes\n\n* Download Kubernetes binaries from: [Kubernetes v1.22.2](https://github.com/kubernetes/kubernetes/releases/tag/v1.22.2)\n\n* Go to [CHANGELOG](https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG/CHANGELOG-1.22.md)\n\n* Check [Downloads for v1.22.2](https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG/CHANGELOG-1.22.md#downloads-for-v1222)\n\nGenerate SHA512 hash from the file:\n\n```bash\ncurl -L https://dl.k8s.io/v1.22.2/kubernetes-server-linux-amd64.tar.gz | sha512sum\n```\n\n## RBAC\n\n[Using RBAC Authorization](https://kubernetes.io/docs/reference/access-authn-authz/rbac/)\n\nCreate namespace `red` and `blue`:\n\n```bash\nkubectl create namespace red\nkubectl create namespace blue\n```\n\nCreate \"Roles\" and \"RoleBindings\":\n\n```bash\nkubectl create role secret-manager-role --namespace=red --verb=get --resource=secrets\nkubectl create rolebinding secret-manager-rolebinding --namespace=red --role secret-manager-role --user=jane\n\nkubectl create role secret-manager-role --namespace=blue --verb=get,list --resource=secrets\nkubectl create rolebinding secret-manager-rolebinding --namespace=blue --role secret-manager-role --user=jane\n```\n\nTest \"Roles\" and \"RoleBindings\":\n\n```bash\n$ kubectl auth can-i get secrets --namespace=red --as jane\nyes\n$ kubectl auth can-i list secrets --namespace=red --as jane\nno\n$ kubectl auth can-i list secrets --namespace=blue --as jane\nyes\n```\n\nCreate \"ClusterRoles\" and \"ClusterRoleBindings\":\n\n```bash\nkubectl create clusterrole deploy-deleter --verb=delete --resource=deployments\nkubectl create clusterrolebinding deploy-deleter --user=jane --clusterrole=deploy-deleter\n```\n\nTest \"ClusterRoles\" and \"ClusterRoleBindings\":\n\n```bash\nkubectl auth can-i delete deployments --as jane --all-namespaces\nyes\n```\n\n## Users and Certificates\n\n* [Certificate Signing Requests](https://kubernetes.io/docs/reference/access-authn-authz/certificate-signing-requests/)\n\nCreate Jane's certificate:\n\n```bash\ntouch ~/.rnd\nopenssl genrsa -out jane.key 2048\nopenssl req -new -key jane.key -out jane.csr \\\n  -subj \"/C=CZ/ST=Czech/L=Prague/O=IT/OU=DevOps/CN=jane\"\n```\n\nCreate `CertificateSigningRequest`:\n\n```bash\nkubectl apply -f - \u003c\u003c EOF\napiVersion: certificates.k8s.io/v1\nkind: CertificateSigningRequest\nmetadata:\n  name: jane\nspec:\n  groups:\n  - system:authenticated\n  request: $(base64 -w0 jane.csr)\n  signerName: kubernetes.io/kube-apiserver-client\n  usages:\n  - client auth\nEOF\n```\n\nCheck certs:\n\n```console\n$ kubectl get certificatesigningrequest\nNAME   AGE   SIGNERNAME                            REQUESTOR          REQUESTEDDURATION   CONDITION\njane   86s   kubernetes.io/kube-apiserver-client   kubernetes-admin   \u003cnone\u003e              Pending\n\n$ kubectl certificate approve jane\nNAME   AGE   SIGNERNAME                            REQUESTOR          REQUESTEDDURATION   CONDITION\njane   95s   kubernetes.io/kube-apiserver-client   kubernetes-admin   \u003cnone\u003e              Approved,Issued\n```\n\nSave signed Jane's certificate to file and create context:\n\n```bash\nkubectl get certificatesigningrequest jane \\\n  -o=jsonpath='{.status.certificate}' | base64 -d \u003e jane.crt\nkubectl config set-credentials jane \\\n  --client-key=jane.key \\\n  --client-certificate=jane.crt --embed-certs=true\nkubectl config set-context jane \\\n  --user=jane --cluster=kubernetes\n```\n\n```console\n$ kubectl config get-contexts\nCURRENT   NAME                          CLUSTER      AUTHINFO           NAMESPACE\n          jane                          kubernetes   jane\n*         kubernetes-admin@kubernetes   kubernetes   kubernetes-admin\n\n$ kubectl config use-context jane\nSwitched to context \"jane\".\n\n$ kubectl get pods\nError from server (Forbidden): pods is forbidden: User \"jane\" cannot list resource \"pods\" in API group \"\" in the namespace \"default\"\n\n$ kubectl get secrets -n blue\nNAME                  TYPE                                  DATA   AGE\ndefault-token-g7sgk   kubernetes.io/service-account-token   3      3h51m\n```\n\n## Service Accounts\n\nIt is good practice for an application to have its own ServiceAccount.\n\n```console\n$ kubectl get serviceaccount,secrets\nNAME                     SECRETS   AGE\nserviceaccount/default   1         28h\n\nNAME                         TYPE                                  DATA   AGE\nsecret/default-token-rxnbt   kubernetes.io/service-account-token   3      28h\n\n$ kubectl describe serviceaccounts default\nName:                default\nNamespace:           default\nLabels:              \u003cnone\u003e\nAnnotations:         \u003cnone\u003e\nImage pull secrets:  \u003cnone\u003e\nMountable secrets:   default-token-rxnbt\nTokens:              default-token-rxnbt\nEvents:              \u003cnone\u003e\n```\n\nCreate `ServiceAccount` and run the pod using it:\n\n```bash\nkubectl create serviceaccount nginx\n\nkubectl apply -f - \u003c\u003c EOF\napiVersion: v1\nkind: Pod\nmetadata:\n  labels:\n    run: nginx\n  name: nginx\nspec:\n  serviceAccountName: nginx\n  containers:\n  - image: nginx\n    name: nginx\nEOF\n```\n\nGet into the pod:\n\n```console\n$ kubectl exec -it nginx -- bash\n\n$ mount | grep serviceaccount\ntmpfs on /run/secrets/kubernetes.io/serviceaccount type tmpfs (ro,relatime,size=1938344k)\n\n$ find /run/secrets/kubernetes.io/serviceaccount\n/run/secrets/kubernetes.io/serviceaccount\n/run/secrets/kubernetes.io/serviceaccount/..data\n/run/secrets/kubernetes.io/serviceaccount/namespace\n/run/secrets/kubernetes.io/serviceaccount/ca.crt\n/run/secrets/kubernetes.io/serviceaccount/token\n/run/secrets/kubernetes.io/serviceaccount/..2021_10_10_12_32_28.152125108\n/run/secrets/kubernetes.io/serviceaccount/..2021_10_10_12_32_28.152125108/token\n/run/secrets/kubernetes.io/serviceaccount/..2021_10_10_12_32_28.152125108/namespace\n/run/secrets/kubernetes.io/serviceaccount/..2021_10_10_12_32_28.152125108/ca.crt\n```\n\n### Disable automount of the ServiceAccount token in the pod\n\n* [Configure Service Accounts for Pods](https://kubernetes.io/docs/tasks/configure-pod-container/configure-service-account/)\n\n```yaml\nautomountServiceAccountToken: false\n```\n\n## API Access\n\nBy default, the Kubernetes API accepts anonymous requests. Access is denied for\nthe `system:anonymous` user:\n\n```console\n$ curl -k https://localhost:6443\n{\n  \"kind\": \"Status\",\n  \"apiVersion\": \"v1\",\n  \"metadata\": {\n\n  },\n  \"status\": \"Failure\",\n  \"message\": \"forbidden: User \\\"system:anonymous\\\" cannot get path \\\"/\\\"\",\n  \"reason\": \"Forbidden\",\n  \"details\": {\n\n  },\n  \"code\": 403\n}\n```\n\nDisable anonymous API requests:\n\n```console\nsudo vi /etc/kubernetes/manifests/kube-apiserver.yaml\n...\n- --anonymous-auth=false\n...\n```\n\n```console\n$ curl -k https://localhost:6443\n{\n  \"kind\": \"Status\",\n  \"apiVersion\": \"v1\",\n  \"metadata\": {\n\n  },\n  \"status\": \"Failure\",\n  \"message\": \"Unauthorized\",\n  \"reason\": \"Unauthorized\",\n  \"code\": 401\n}\n```\n\nRevert this change, because `kube-apiserver` requires anonymous API requests for\nits own liveness probes.\n\n### Manual API request using curl\n\nExtract the following data from your kubeconfig file:\n\n```bash\nkubectl config view --raw \\\n  -o jsonpath='{.clusters[?(@.name==\"kubernetes\")].cluster.certificate-authority-data}' | base64 -d \u003e ca\nkubectl config view --raw \\\n  -o jsonpath='{.users[?(@.name==\"kubernetes-admin\")].user.client-certificate-data}' | base64 -d \u003e crt\nkubectl config view --raw \\\n  -o jsonpath='{.users[?(@.name==\"kubernetes-admin\")].user.client-key-data}' | base64 -d \u003e key\nSERVER=$(kubectl config view --raw \\\n  -o jsonpath='{.clusters[?(@.name==\"kubernetes\")].cluster.server}')\necho \"${SERVER}\"\n```\n\nAccess the Kubernetes cluster using the extracted certificates and CA:\n\n```console\n$ curl -s \"${SERVER}\" --cacert ca --cert crt --key key | jq\n{\n  \"paths\": [\n    \"/.well-known/openid-configuration\",\n...\n    \"/version\"\n  ]\n}\n```\n\n### Enable k8s API for external access\n\n* [Controlling Access to the Kubernetes API](https://kubernetes.io/docs/concepts/security/controlling-access/)\n\nVerify which IP addresses are allowed to connect to the Kubernetes API. Your\nexternal IP address (`192.168.56.2`) should be listed:\n\n```console\n$ openssl x509 -in /etc/kubernetes/pki/apiserver.crt -text\n...\nX509v3 Subject Alternative Name:\n                DNS:kubemaster, DNS:kubernetes, DNS:kubernetes.default, DNS:kubernetes.default.svc, DNS:kubernetes.default.svc.cluster.local, IP Address:10.96.0.1, IP Address:192.168.56.2\n...\n```\n\nCheck the kubernetes service:\n\n```console\n$ kubectl get service kubernetes\nNAME         TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)   AGE\nkubernetes   ClusterIP   10.96.0.1    \u003cnone\u003e        443/TCP   37h\n```\n\nChange the service to `NodePort`:\n\n```bash\nkubectl patch service kubernetes --type='json' -p '[{\"op\":\"replace\",\"path\":\"/spec/type\",\"value\":\"NodePort\"}]'\n```\n\nFind the `NodePort` port which is accessible externally:\n\n```console\n$ kubectl get service kubernetes\nNAME         TYPE       CLUSTER-IP   EXTERNAL-IP   PORT(S)         AGE\nkubernetes   NodePort   10.96.0.1    \u003cnone\u003e        443:32689/TCP   37h\n```\n\nCopy the kubeconfig file to the machine where Vagrant is running:\n\n```bash\nvagrant ssh kubemaster -c \"kubectl config view --raw\" \u003e local.conf\n```\n\nIn the copied kubeconfig file, replace the `server` parameter with the external\nIP address and NodePort:\n\n```bash\nsed -i 's@\\(.*server: https:\\).*@\\1//192.168.56.2:32689@' local.conf\n```\n\nCheck if you can see the namespaces:\n\n```bash\nkubectl --kubeconfig=local.conf get ns\n```\n\n## Kubernetes cluster upgrade\n\n* [Upgrading kubeadm clusters](https://kubernetes.io/docs/tasks/administer-cluster/kubeadm/kubeadm-upgrade/)\n\n### Upgrade master node\n\nThe cluster is currently running Kubernetes version 1.19:\n\n```console\n# vagrant ssh kubemaster\nkubectl get nodes\n```\n\n```console\nNAME         STATUS   ROLES    AGE   VERSION\nkubemaster   Ready    master   22m   v1.19.0\nkubenode01   Ready    \u003cnone\u003e   22m   v1.19.0\n```\n\nDrain the master node:\n\n```bash\nkubectl drain kubemaster --ignore-daemonsets\n```\n\nThe master node is now drained:\n\n```bash\nkubectl get nodes\n```\n\n```console\nNAME         STATUS                     ROLES    AGE   VERSION\nkubemaster   Ready,SchedulingDisabled   master   25m   v1.19.0\nkubenode01   Ready                      \u003cnone\u003e   24m   v1.19.0\n```\n\nCheck available Kubernetes versions (the next minor version is 1.20):\n\n```console\n$ apt-cache show kubeadm | grep '1.20'\nVersion: 1.20.11-00\nFilename: pool/kubeadm_1.20.11-00_amd64_1343a8b5f81f535549d498a9cf38a2307eee0fc99ea64046b043efae50e31bfe.deb\nVersion: 1.20.10-00\nFilename: pool/kubeadm_1.20.10-00_amd64_bef04cc2cb819b1298bd1c22bae9ba90c52cf581584f5f24871df8447ae93186.deb\n...\n```\n\nUpgrade k8s cluster components:\n\n```bash\nKUBE_VERSION=1.20.10\nsudo apt-get install -y --allow-change-held-packages \\\n  kubelet=\"${KUBE_VERSION}-00\" \\\n  kubeadm=\"${KUBE_VERSION}-00\" \\\n  kubectl=\"${KUBE_VERSION}-00\"\n```\n\nCheck the \"upgrade plan\":\n\n```console\n$ kubeadm upgrade plan\n...\nComponents that must be upgraded manually after you have upgraded the control plane with 'kubeadm upgrade apply':\nCOMPONENT   CURRENT        AVAILABLE\nkubelet     1 x v1.19.0    v1.20.10\n            1 x v1.20.10   v1.20.10\n\nUpgrade to the latest stable version:\n\nCOMPONENT                 CURRENT   AVAILABLE\nkube-apiserver            v1.19.0   v1.20.10\nkube-controller-manager   v1.19.0   v1.20.10\nkube-scheduler            v1.19.0   v1.20.10\nkube-proxy                v1.19.0   v1.20.10\nCoreDNS                   1.7.0     1.7.0\netcd                      3.4.9-1   3.4.13-0\n\nYou can now apply the upgrade by executing the following command:\n\n  kubeadm upgrade apply v1.20.10\n...\n```\n\nUpgrade the k8s cluster to `1.20.10`:\n\n```bash\nsudo kubeadm upgrade apply v1.20.10\n```\n\nCheck the nodes:\n\n```bash\nkubectl get nodes\n```\n\n```console\nNAME         STATUS                     ROLES                  AGE   VERSION\nkubemaster   Ready,SchedulingDisabled   control-plane,master   42m   v1.20.10\nkubenode01   Ready                      \u003cnone\u003e                 42m   v1.19.0\n```\n\nUncordon the master:\n\n```bash\nkubectl uncordon kubemaster\n```\n\n### Upgrade worker node\n\n```bash\n# vagrant ssh kubemaster\nkubectl get nodes\n```\n\n```console\nNAME         STATUS   ROLES                  AGE   VERSION\nkubemaster   Ready    control-plane,master   43m   v1.20.10\nkubenode01   Ready    \u003cnone\u003e                 43m   v1.19.0\n```\n\nDrain the worker node:\n\n```bash\nkubectl drain kubenode01 --ignore-daemonsets\n```\n\nThe worker node is now drained:\n\n```bash\nkubectl get nodes\n```\n\n```console\nNAME         STATUS                     ROLES                  AGE   VERSION\nkubemaster   Ready                      control-plane,master   48m   v1.20.10\nkubenode01   Ready,SchedulingDisabled   \u003cnone\u003e                 47m   v1.19.0\n```\n\nUpgrade k8s cluster components:\n\n```bash\n# vagrant ssh kubenode01\n\nKUBE_VERSION=1.20.10\nsudo apt-get install -y --allow-change-held-packages \\\n  kubeadm=\"${KUBE_VERSION}-00\"\n\nsudo kubeadm upgrade node\n\nsudo apt-get install -y --allow-change-held-packages \\\n  kubelet=\"${KUBE_VERSION}-00\" \\\n  kubectl=\"${KUBE_VERSION}-00\"\n```\n\nCheck the nodes:\n\n```bash\nkubectl uncordon kubenode01\n```\n\n```console\nnode/kubenode01 uncordoned\n```\n\n```bash\nkubectl get nodes\n```\n\n```console\nNAME         STATUS                     ROLES                  AGE   VERSION\nkubemaster   Ready                      control-plane,master   53m   v1.20.10\nkubenode01   Ready,SchedulingDisabled   \u003cnone\u003e                 52m   v1.20.10\n```\n\n## Kubernetes Secrets\n\n* [Encrypting Secret Data at Rest](https://kubernetes.io/docs/tasks/administer-cluster/encrypt-data/)\n\n### Create pod with secrets\n\nCreate Secrets:\n\n```bash\nkubectl create secret generic secret1 --from-literal username=admin1 --from-literal password=admin123\nkubectl create secret generic secret2 --from-literal username=admin2 --from-literal password=admin321\n```\n\nCreate a Pod that uses the Secrets:\n\n```bash\nkubectl apply -f - \u003c\u003c EOF\napiVersion: v1\nkind: Pod\nmetadata:\n  name: mypod\nspec:\n  containers:\n  - name: mypod\n    image: nginx\n    env:\n      - name: USERNAME\n        valueFrom:\n          secretKeyRef:\n            name: secret2\n            key: username\n      - name: PASSWORD\n        valueFrom:\n          secretKeyRef:\n            name: secret2\n            key: password\n    volumeMounts:\n    - name: secret1\n      mountPath: \"/secret1\"\n      readOnly: true\n  volumes:\n  - name: secret1\n    secret:\n      secretName: secret1\nEOF\n```\n\nVerify the Secrets within the containers:\n\n```console\n$ kubectl exec -it mypod -- bash -xc 'ls /secret1 ; echo $(cat /secret1/username); echo $(cat /secret1/password)'\n+ ls /secret1\npassword  username\n++ cat /secret1/username\n+ echo admin1\nadmin1\n++ cat /secret1/password\n+ echo admin123\nadmin123\n\n$ kubectl exec -it mypod -- env | grep -E '(USERNAME|PASSWORD)'\nUSERNAME=admin2\nPASSWORD=admin321\n```\n\n### Access secrets \"non-k8s\" way\n\n```console\n# vagrant ssh kubenode01\n\n$ sudo crictl ps\n...\nCONTAINER ID        IMAGE               CREATED             STATE               NAME                ATTEMPT             POD ID\n879e08f431dd5       f8f4ffc8092c9       35 seconds ago      Running             mypod               0                   57c9f78887129\n...\n```\n\nInspect the container details, where you will find the unencrypted environment\nvariables:\n\n```console\n$ sudo crictl inspect 879e08f431dd5\n...\n    \"runtimeSpec\": {\n      \"ociVersion\": \"1.0.2-dev\",\n      \"process\": {\n        \"user\": {\n          \"uid\": 0,\n          \"gid\": 0\n        },\n        \"args\": [\n          \"/docker-entrypoint.sh\",\n          \"nginx\",\n          \"-g\",\n          \"daemon off;\"\n        ],\n        \"env\": [\n...\n          \"USERNAME=admin2\",\n          \"PASSWORD=admin321\",\n...\n```\n\nNext, let's examine the values mounted via volumes:\n\n```console\n$ sudo crictl inspect 879e08f431dd5 | jq '.info.pid'\n8661\n\n$ sudo ls -l /proc/8661/root/secret1\ntotal 0\nlrwxrwxrwx 1 root root 15 Oct 11 11:02 password -\u003e ..data/password\nlrwxrwxrwx 1 root root 15 Oct 11 11:02 username -\u003e ..data/username\n```\n\n### ETCD\n\nFind etcd connection details:\n\n```console\n# cat /etc/kubernetes/manifests/kube-apiserver.yaml | grep etcd\n    - --etcd-cafile=/etc/kubernetes/pki/etcd/ca.crt\n    - --etcd-certfile=/etc/kubernetes/pki/apiserver-etcd-client.crt\n    - --etcd-keyfile=/etc/kubernetes/pki/apiserver-etcd-client.key\n    - --etcd-servers=https://127.0.0.1:2379\n```\n\n```console\n# ETCDCTL_API=3 etcdctl endpoint health \\\n  --cert=/etc/kubernetes/pki/apiserver-etcd-client.crt \\\n  --key=/etc/kubernetes/pki/apiserver-etcd-client.key \\\n  --cacert=/etc/kubernetes/pki/etcd/ca.crt\n127.0.0.1:2379 is healthy: successfully committed proposal: took = 876.099µs\n```\n\n```console\nETCDCTL_API=3 etcdctl get /registry/secrets/default/secret2 \\\n  --cert=/etc/kubernetes/pki/apiserver-etcd-client.crt \\\n  --key=/etc/kubernetes/pki/apiserver-etcd-client.key \\\n  --cacert=/etc/kubernetes/pki/etcd/ca.crt\n...\n?{\"f:data\":{\".\":{},\"f:password\":{},\"f:username\":{}},\"f:type\":{}}\npassworadmin321\nusernameadmin2Opaque\"\n...\n```\n\n### Encrypt secrets in ETCD\n\nCreate `EncryptionConfiguration` for API server:\n\n```bash\nmkdir /etc/kubernetes/etcd\ncat \u003e /etc/kubernetes/etcd/encryptionconfiguration.yaml \u003c\u003c EOF\napiVersion: apiserver.config.k8s.io/v1\nkind: EncryptionConfiguration\nresources:\n  - resources:\n    - secrets\n    providers:\n    - aescbc:\n        keys:\n        - name: key1\n          secret: c2VjcmV0IGlzIHNlY3VyZQ==\n    - identity: {}\nEOF\n```\n\nModify the API server configuration to use encryption:\n\n```console\n# vi /etc/kubernetes/manifests/kube-apiserver.yaml\n...\nspec:\n  containers:\n  - command:\n    - kube-apiserver\n    - --encryption-provider-config=/etc/kubernetes/etcd/encryptionconfiguration.yaml\n...\n    volumeMounts:\n    - mountPath: /etc/kubernetes/etcd\n      name: etcd\n      readOnly: true\n...\n  volumes:\n  - hostPath:\n      path: /etc/kubernetes/etcd\n      type: DirectoryOrCreate\n    name: etcd\n```\n\nCreate a test Secret, which should now be encrypted in etcd:\n\n```bash\nkubectl create secret generic secret3 --from-literal username=admin3 --from-literal password=admin567\n```\n\nRead the Secret from etcd; it should be encrypted:\n\n```console\nETCDCTL_API=3 etcdctl get /registry/secrets/default/secret3 \\\n  --cert=/etc/kubernetes/pki/apiserver-etcd-client.crt \\\n  --key=/etc/kubernetes/pki/apiserver-etcd-client.key \\\n  --cacert=/etc/kubernetes/pki/etcd/ca.crt | \\\n  hexdump -C\n00000000  2f 72 65 67 69 73 74 72  79 2f 73 65 63 72 65 74  |/registry/secret|\n00000010  73 2f 64 65 66 61 75 6c  74 2f 73 65 63 72 65 74  |s/default/secret|\n00000020  33 0a 6b 38 73 3a 65 6e  63 3a 61 65 73 63 62 63  |3.k8s:enc:aescbc|\n00000030  3a 76 31 3a 6b 65 79 31  3a f6 a9 70 c6 88 73 ef  |:v1:key1:..p..s.|\n00000040  94 ca 74 d9 7e 8e 98 88  e0 ad 82 0a 44 67 72 17  |..t.~.......Dgr.|\n00000050  6b 19 0d 62 f7 9b ec ed  57 40 a6 8f c4 87 d6 9c  |k..b....W@......|\n...\n```\n\n## Container Runtime\n\n* [Runtime Class](https://kubernetes.io/docs/concepts/containers/runtime-class/)\n\nCreate `RuntimeClass` for gvisor:\n\n```bash\nkubectl label node kubenode01 runtimeclass=gvisor\n\nkubectl apply -f - \u003c\u003c EOF\napiVersion: node.k8s.io/v1\nkind: RuntimeClass\nmetadata:\n  name: gvisor\nhandler: runsc\nscheduling:\n  nodeSelector:\n    runtimeclass: gvisor\nEOF\n```\n\nCreate Pod:\n\n```bash\nkubectl apply -f - \u003c\u003c EOF\napiVersion: v1\nkind: Pod\nmetadata:\n  labels:\n    run: gvisor\n  name: gvisor\nspec:\n  runtimeClassName: gvisor\n  containers:\n  - image: gvisor\n    name: nginx\nEOF\n```\n\n## Container security\n\n* [Configure a Security Context for a Pod or Container](https://kubernetes.io/docs/tasks/configure-pod-container/security-context/)\n\n### Security Context\n\nCreate busybox pod:\n\n```bash\nkubectl run busybox --image=busybox --command \\\n  --dry-run=client -o yaml -- sh -c 'sleep 1d'\n```\n\n```bash\nkubectl apply -f - \u003c\u003c EOF\napiVersion: v1\nkind: Pod\nmetadata:\n  labels:\n    run: busybox\n  name: busybox\nspec:\n  containers:\n  - command:\n    - sh\n    - -c\n    - sleep 1d\n    image: busybox\n    name: busybox\n  restartPolicy: Never\nEOF\n```\n\nThe Pod is running as the root user:\n\n```console\n$ kubectl exec -it busybox -- sh -c 'set -x ; id ; touch /test123 ; ls -l /test123'\n+ id\nuid=0(root) gid=0(root) groups=10(wheel)\n+ touch /test123\n+ ls -l /test123\n-rw-r--r--    1 root     root             0 Oct 12 05:20 /test123\n```\n\nAdd `SecurityContext` to the pod definition:\n\n```bash\nkubectl apply --force=true --grace-period=1 -f - \u003c\u003c EOF\napiVersion: v1\nkind: Pod\nmetadata:\n  labels:\n    run: busybox\n  name: busybox\nspec:\n  securityContext:\n    runAsUser: 1000\n    runAsGroup: 3000\n  containers:\n  - command:\n    - sh\n    - -c\n    - sleep 1d\n    image: busybox\n    name: busybox\n  restartPolicy: Never\nEOF\n```\n\nCheck permissions:\n\n```console\n$ kubectl exec -it busybox -- sh -c 'set -x ; id ; touch /tmp/test123 ; ls -l /tmp/test123'\n+ id\nuid=1000 gid=3000 groups=2000\n+ touch /tmp/test123\n+ ls -l /tmp/test123\n-rw-r--r--    1 1000     3000             0 Oct 12 05:19 /tmp/test123\n```\n\n### Privileged containers and PrivilegeEscalations\n\n* [Privileged](https://kubernetes.io/docs/concepts/policy/pod-security-policy/#privileged)\n\nThe container's root user is mapped to the host's root user.\n\n```bash\nkubectl apply --force=true --grace-period=1 -f - \u003c\u003c EOF\napiVersion: v1\nkind: Pod\nmetadata:\n  labels:\n    run: busybox\n  name: busybox\nspec:\n  containers:\n  - command:\n    - sh\n    - -c\n    - sleep 1d\n    image: busybox\n    name: busybox\n    securityContext:\n      privileged: true\n  restartPolicy: Never\nEOF\n```\n\nCheck permissions:\n\n```console\n$ kubectl exec -it busybox -- sh -c 'set -x ; sysctl kernel.hostname=attacker'\n+ sysctl 'kernel.hostname=attacker'\nkernel.hostname = attacker\n```\n\n### Pod Security Policy\n\n* [Pod Security Policies](https://kubernetes.io/docs/concepts/policy/pod-security-policy/)\n\nPodSecurityPolicy is deprecated\n([PodSecurityPolicy Deprecation: Past, Present, and Future](https://kubernetes.io/blog/2021/04/06/podsecuritypolicy-deprecation-past-present-and-future/))\nand you should use [Kyverno](https://github.com/kyverno/kyverno/) or [OPA/Gatekeeper](https://github.com/open-policy-agent/gatekeeper/)\ninstead.\n\n* [Create a policy and a pod](https://kubernetes.io/docs/concepts/policy/pod-security-policy/#create-a-policy-and-a-pod)\n\nCreate a new namespace and a Pod that stores the current date in `/tmp/date` on\nthe host node:\n\n```bash\nkubectl create namespace my-namespace\nkubectl apply -f - \u003c\u003c EOF\napiVersion: apps/v1\nkind: Deployment\nmetadata:\n  labels:\n    app: my-busybox\n  name: my-busybox\n  namespace: my-namespace\nspec:\n  replicas: 1\n  selector:\n    matchLabels:\n      app: my-busybox\n  template:\n    metadata:\n      labels:\n        app: my-busybox\n    spec:\n      containers:\n      - command: [\"sh\", \"-c\", \"while true ; do date | tee /tmp/date ; sleep 5 ; done\" ]\n        image: busybox\n        name: busybox\n        volumeMounts:\n        - mountPath: /tmp\n          name: test-volume\n      volumes:\n      - name: test-volume\n        hostPath:\n          path: /tmp\nEOF\n\nkubectl exec -it -n my-namespace \\\n  \"$(kubectl get pod -n my-namespace -l app=my-busybox --no-headers \\\n    -o custom-columns=':metadata.name')\" -- cat /tmp/date\n```\n\nEnable the PodSecurityPolicy admission plugin in the API server:\n\n```console\nsudo vi /etc/kubernetes/manifests/kube-apiserver.yaml\n...\n- --enable-admission-plugins=NodeRestriction,PodSecurityPolicy\n...\n```\n\nCreate a PodSecurityPolicy (PSP) that disallows `allowPrivilegeEscalation` and\n`privileged` mode, and only allows `HostPaths` for `/var/tmp`:\n\n```bash\nkubectl apply -f - \u003c\u003c EOF\napiVersion: policy/v1beta1\nkind: PodSecurityPolicy\nmetadata:\n  name: default\nspec:\n  privileged: false\n  allowPrivilegeEscalation: false\n  allowedHostPaths:\n  - pathPrefix: /var/tmp\n  allowedCapabilities:\n  - NET_ADMIN\n  - IPC_LOCK\n  seLinux:\n    rule: RunAsAny\n  supplementalGroups:\n    rule: RunAsAny\n  runAsUser:\n    rule: RunAsAny\n  fsGroup:\n    rule: RunAsAny\n  volumes:\n  - '*'\nEOF\n```\n\nCreate a `Role` and `RoleBinding` that grant permission to use PodSecurityPolicies:\n\n```bash\nkubectl create clusterrole psp-access --verb=use \\\n  --resource=podsecuritypolicies\nkubectl create clusterrolebinding psp-access \\\n  --clusterrole=psp-access \\\n  --group=system:serviceaccounts:my-namespace --namespace=my-namespace\n```\n\nAll Pods in the `my-namespace` namespace are now restricted to using\n`HostPaths` within `/var/tmp/`. If the existing Pod is restarted, you will\nobserve the following:\n\n```console\n$ kubectl delete pod -n my-namespace --all\n$ kubectl describe replicasets.apps -n my-namespace\n...\nEvents:\n  Type     Reason            Age                    From                   Message\n  ----     ------            ----                   ----                   -------\n  Normal   SuccessfulCreate  13m                    replicaset-controller  Created pod: my-busybox-7d5b7688f9-wh655\n  Warning  FailedCreate      4m5s (x17 over 4m46s)  replicaset-controller  Error creating: pods \"my-busybox-7d5b7688f9-\" is forbidden: PodSecurityPolicy: unable to admit pod: [spec.volumes[0].hostPath.pathPrefix: Invalid value: \"/tmp\": is not allowed to be used]\n```\n\nIt is necessary to modify the Deployment to use `/var/tmp/` instead of `/tmp/`:\n\n```bash\nkubectl apply -f - \u003c\u003c EOF\napiVersion: apps/v1\nkind: Deployment\nmetadata:\n  labels:\n    app: my-busybox\n  name: my-busybox\n  namespace: my-namespace\nspec:\n  replicas: 1\n  selector:\n    matchLabels:\n      app: my-busybox\n  template:\n    metadata:\n      labels:\n        app: my-busybox\n    spec:\n      containers:\n      - command: [\"sh\", \"-c\", \"while true ; do date | tee /tmp/date ; sleep 5 ; done\" ]\n        image: busybox\n        name: busybox\n        volumeMounts:\n        - mountPath: /tmp\n          name: test-volume\n      volumes:\n      - name: test-volume\n        hostPath:\n          path: /tmp\nEOF\n```\n\n## Open Policy Agent (OPA)\n\n* [OPA Gatekeeper: Policy and Governance for Kubernetes](https://kubernetes.io/blog/2019/08/06/opa-gatekeeper-policy-and-governance-for-kubernetes/)\n\nInstall Gatekeeper:\n\n```bash\nkubectl apply -f \\\n  https://raw.githubusercontent.com/open-policy-agent/gatekeeper/release-3.5/deploy/gatekeeper.yaml\n```\n\nCheck the CRDs:\n\n```console\n$ kubectl get crd | grep gatekeeper\nconfigs.config.gatekeeper.sh                         2021-10-12T10:23:31Z\nconstraintpodstatuses.status.gatekeeper.sh           2021-10-12T10:23:31Z\nconstrainttemplatepodstatuses.status.gatekeeper.sh   2021-10-12T10:23:31Z\nconstrainttemplates.templates.gatekeeper.sh          2021-10-12T10:23:31Z\nk8strustedimages.constraints.gatekeeper.sh           2021-10-12T10:41:26Z\n```\n\nBlock Pods from using images from the `k8s.gcr.io` container registry.\n\nCreate `ConstraintTemplate`:\n\n```bash\nkubectl apply -f - \u003c\u003c EOF\napiVersion: templates.gatekeeper.sh/v1beta1\nkind: ConstraintTemplate\nmetadata:\n  name: k8strustedimages\nspec:\n  crd:\n    spec:\n      names:\n        kind: K8sTrustedImages\n  targets:\n    - target: admission.k8s.gatekeeper.sh\n      rego: |\n        package k8strustedimages\n        violation[{\"msg\": msg}] {\n          image := input.review.object.spec.containers[_].image\n          startswith(image, \"k8s.gcr.io/\")\n          # not startswith(image, \"k8s.gcr.io/\")  # allow / whitelist registry\n          msg := \"Using images from k8s.gcr.io is not allowed !\"\n        }\nEOF\n```\n\nCreate `K8sTrustedImages` constraint:\n\n```bash\nkubectl apply -f - \u003c\u003c EOF\napiVersion: constraints.gatekeeper.sh/v1beta1\nkind: K8sTrustedImages\nmetadata:\n  name: pod-trusted-images\nspec:\n  match:\n    kinds:\n      - apiGroups: [\"\"]\n        kinds: [\"Pod\"]\nEOF\n```\n\nTry it:\n\n```console\n$ kubectl run pause --image=k8s.gcr.io/pause:3.1\nError from server ([pod-trusted-images] Using images from k8s.gcr.io is not allowed !): admission webhook \"validation.gatekeeper.sh\" denied the request: [pod-trusted-images] Using images from k8s.gcr.io is not allowed !\n```\n\n```console\n$ kubectl describe K8sTrustedImages pod-trusted-images\nName:         pod-trusted-images\n...\nStatus:\n  Audit Timestamp:  2021-10-12T10:56:48Z\n  By Pod:\n    Constraint UID:       39aaf68c-2921-42ec-977e-0c63f6623764\n    Enforced:             true\n    Id:                   gatekeeper-audit-6c558d7455-fv8c5\n    Observed Generation:  1\n    Operations:\n      audit\n      status\n    Constraint UID:       39aaf68c-2921-42ec-977e-0c63f6623764\n    Enforced:             true\n    Id:                   gatekeeper-controller-manager-ff8849b64-nh65b\n    Observed Generation:  1\n...\n  Total Violations:  8\n  Violations:\n    Enforcement Action:  deny\n    Kind:                Pod\n    Message:             Using images from k8s.gcr.io is not allowed !\n    Name:                coredns-558bd4d5db-k6qfn\n    Namespace:           kube-system\n    Enforcement Action:  deny\n    Kind:                Pod\n    Message:             Using images from k8s.gcr.io is not allowed !\n    Name:                coredns-558bd4d5db-q9mzm\n    Namespace:           kube-system\n    Enforcement Action:  deny\n    Kind:                Pod\n    Message:             Using images from k8s.gcr.io is not allowed !\n    Name:                etcd-kubemaster\n    Namespace:           kube-system\n...\nEvents:                  \u003cnone\u003e\n```\n\nDelete all Constraints and ConstraintTemplates:\n\n```bash\nkubectl delete K8sTrustedImages,ConstraintTemplates --all\n```\n\n## Image vulnerability scanning\n\n* [Trivy documentation](https://aquasecurity.github.io/trivy/)\n* [Installation](https://aquasecurity.github.io/trivy/v0.20.0/getting-started/installation/):\n\nInstall trivy:\n\n```bash\ncurl -sfL https://raw.githubusercontent.com/aquasecurity/trivy/main/contrib/install.sh |\n  sh -s -- -b /usr/local/bin v0.20.0\n```\n\nCheck vulnerabilities:\n\n```console\n$ trivy image nginx\n2021-10-13T07:09:27.105Z  INFO  Detected OS: debian\n2021-10-13T07:09:27.105Z  INFO  Detecting Debian vulnerabilities...\n2021-10-13T07:09:27.129Z  INFO  Number of language-specific files: 0\n\nnginx (debian 10.11)\n====================\nTotal: 172 (UNKNOWN: 0, LOW: 125, MEDIUM: 17, HIGH: 26, CRITICAL: 4)\n...\n```\n\nCheck `CRITICAL` only:\n\n```console\n$ trivy image --severity CRITICAL nginx\n2021-10-13T07:10:21.860Z  INFO  Detected OS: debian\n2021-10-13T07:10:21.860Z  INFO  Detecting Debian vulnerabilities...\n2021-10-13T07:10:21.881Z  INFO  Number of language-specific files: 0\n\nnginx (debian 10.11)\n====================\nTotal: 4 (CRITICAL: 4)\n\n+----------+------------------+----------+-------------------+---------------+---------------------------------------+\n| LIBRARY  | VULNERABILITY ID | SEVERITY | INSTALLED VERSION | FIXED VERSION |                 TITLE                 |\n+----------+------------------+----------+-------------------+---------------+---------------------------------------+\n| libc-bin | CVE-2021-33574   | CRITICAL | 2.28-10           |               | glibc: mq_notify does                 |\n|          |                  |          |                   |               | not handle separately                 |\n|          |                  |          |                   |               | allocated thread attributes           |\n|          |                  |          |                   |               | --\u003eavd.aquasec.com/nvd/cve-2021-33574 |\n+          +------------------+          +                   +---------------+---------------------------------------+\n|          | CVE-2021-35942   |          |                   |               | glibc: Arbitrary read in wordexp()    |\n|          |                  |          |                   |               | --\u003eavd.aquasec.com/nvd/cve-2021-35942 |\n+----------+------------------+          +                   +---------------+---------------------------------------+\n| libc6    | CVE-2021-33574   |          |                   |               | glibc: mq_notify does                 |\n|          |                  |          |                   |               | not handle separately                 |\n|          |                  |          |                   |               | allocated thread attributes           |\n|          |                  |          |                   |               | --\u003eavd.aquasec.com/nvd/cve-2021-33574 |\n+          +------------------+          +                   +---------------+---------------------------------------+\n|          | CVE-2021-35942   |          |                   |               | glibc: Arbitrary read in wordexp()    |\n|          |                  |          |                   |               | --\u003eavd.aquasec.com/nvd/cve-2021-35942 |\n+----------+------------------+----------+-------------------+---------------+---------------------------------------+\n```\n\nList all images in the cluster:\n\n```bash\nkubectl get pods -A --no-headers \\\n  -o=custom-columns='DATA:spec.containers[*].image'\n```\n\nList all container images that have `CRITICAL` vulnerabilities:\n\n```bash\nwhile read -r CONTAINER_IMAGE; do\n  trivy image --severity CRITICAL \"${CONTAINER_IMAGE}\"\ndone \u003c \u003c(kubectl get pods -A --no-headers -o=custom-columns='DATA:spec.containers[*].image')\n```\n\n## K8s Container and host security\n\n### Access to k8s objects form host\n\n```console\n$ strace -f ls /\nexecve(\"/bin/ls\", [\"ls\", \"/\"], 0x7ffe01cc7ee0 /* 25 vars */) = 0\nbrk(NULL)                               = 0x55b044960000\naccess(\"/etc/ld.so.nohwcap\", F_OK)      = -1 ENOENT (No such file or directory)\naccess(\"/etc/ld.so.preload\", R_OK)      = -1 ENOENT (No such file or directory)\nopenat(AT_FDCWD, \"/etc/ld.so.cache\", O_RDONLY|O_CLOEXEC) = 3\nfstat(3, {st_mode=S_IFREG|0644, st_size=23060, ...}) = 0\nmmap(NULL, 23060, PROT_READ, MAP_PRIVATE, 3, 0) = 0x7f0bdc05e000\n...\n...\n...\nwrite(1, \"boot  etc  initrd.img  lib\\t     \"..., 100boot  etc  initrd.img  lib         lost+found  mnt    proc  run   snap  sys  usr  var      vmlinuz.old\n) = 100\nclose(1)                                = 0\nclose(2)                                = 0\nexit_group(0)                           = ?\n+++ exited with 0 +++\n```\n\n```console\n$ strace -cw ls /\nbin   dev  home        initrd.img.old  lib64     media  opt  root  sbin  srv  tmp  vagrant  vmlinuz\nboot  etc  initrd.img  lib         lost+found  mnt    proc  run   snap  sys  usr  var      vmlinuz.old\n% time     seconds  usecs/call     calls    errors syscall\n------ ----------- ----------- --------- --------- ----------------\n 18.82    0.000383          13        30           mmap\n 17.89    0.000364          15        24           openat\n 15.14    0.000308          12        26           close\n  8.99    0.000183         183         1           execve\n  8.75    0.000178           7        25           fstat\n  6.93    0.000141          12        12           mprotect\n  6.29    0.000128          64         2           getdents\n  4.37    0.000089          10         9           read\n  3.64    0.000074           9         8         8 access\n  1.47    0.000030          15         2           write\n  1.43    0.000029          15         2         2 statfs\n  1.23    0.000025           8         3           brk\n  1.03    0.000021          11         2           ioctl\n  0.88    0.000018          18         1           munmap\n  0.69    0.000014           7         2           rt_sigaction\n  0.44    0.000009           9         1           stat\n  0.39    0.000008           8         1           futex\n  0.34    0.000007           7         1           rt_sigprocmask\n  0.34    0.000007           7         1           arch_prctl\n  0.34    0.000007           7         1           prlimit64\n  0.29    0.000006           6         1           set_tid_address\n  0.29    0.000006           6         1           set_robust_list\n------ ----------- ----------- --------- --------- ----------------\n100.00    0.002035                   156        10 total\n```\n\nFind the `PID` for `etcd`:\n\n```console\n$ ps -elf | grep etcd\n4 S root      6900  6529  1  80   0 - 2653294 -    Oct12 ?        00:19:39 etcd \\\n  --advertise-client-urls=https://192.168.56.2:2379 \\\n  --cert-file=/etc/kubernetes/pki/\n```\n\nAttach strace to the `etcd` process:\n\n```console\nsudo strace -f -p 6900\n```\n\nList the files opened by `etcd`:\n\n```console\n$ sudo ls -l /proc/6900/fd | grep '/'\nlrwx------ 1 root root 64 Oct 11 18:43 0 -\u003e /dev/null\nl-wx------ 1 root root 64 Oct 12 10:41 108 -\u003e /var/lib/etcd/member/wal/1.tmp\nl-wx------ 1 root root 64 Oct 11 10:52 11 -\u003e /var/lib/etcd/member/wal/0000000000000001-00000000000177e3.wal\nlrwx------ 1 root root 64 Oct 11 10:52 7 -\u003e /var/lib/etcd/member/snap/db\nlr-x------ 1 root root 64 Oct 11 10:52 9 -\u003e /var/lib/etcd/member/wal\n```\n\nCreate Secret:\n\n```bash\nkubectl create secret generic my-secret --from-literal=password=1234512345\n```\n\nFind the Secret `1234512345` in the etcd database file:\n\n```console\n$ sudo strings /proc/6900/fd/7 | grep -B10 1234512345\n#/registry/secrets/default/my-secret\nSecret\n  my-secret\ndefault\"\n*$5dd89c0e-0cd2-4d33-9be7-bb00793df3de2\nkubectl-create\nUpdate\nFieldsV1:1\n/{\"f:data\":{\".\":{},\"f:password\":{}},\"f:type\":{}}\npassword\n1234512345\n```\n\nRun a Pod with the Secret exposed as an environment variable:\n\n```bash\nkubectl apply -f - \u003c\u003c EOF\napiVersion: v1\nkind: Pod\nmetadata:\n  name: secret-test-pod\nspec:\n  containers:\n    - name: test-container\n      image: k8s.gcr.io/busybox\n      command: [ \"/bin/sh\", \"-c\", \"env ; sleep 3000\" ]\n      envFrom:\n      - secretRef:\n          name: my-secret\n  restartPolicy: Never\nEOF\n```\n\n```console\n$ kubectl logs secret-test-pod | grep password\npassword=1234512345\n```\n\n```console\n# vagrant ssh kubenode01\n\n$ ps -elf | grep sh\n...\n4 S root     30682 30597  0  80   0 -   794 -      11:26 ?        00:00:00 /bin/sh -c env ; sleep 3000\n...\n```\n\nYou can see `password=1234512345` in the `/proc` directory for the container's process:\n\n```console\n$ sudo cat /proc/30682/environ\nPATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/binHOSTNAME=secret-test-podpassword=1234512345KUBERNETES_PORT_443_TCP=tcp://10.96.0.1:443KUBERNETES_PORT_443_TCP_PROTO=tcpKUBERNETES_PORT_443_TCP_PORT=443KUBERNETES_PORT_443_TCP_ADDR=10.96.0.1KUBERNETES_SERVICE_HOST=10.96.0.1KUBERNETES_SERVICE_PORT=443KUBERNETES_SERVICE_PORT_HTTPS=443KUBERNETES_PORT=tcp://10.96.0.1:443HOME=/root\n```\n\n### Falco\n\n* [Falco documentation](https://falco.org/docs/)\n\nInstall Falco:\n\n```console\n# vagrant ssh kubenode01\n\ncurl -o install_falco -s https://falco.org/script/install\nsudo bash install_falco\nsudo systemctl daemon-reload\nsudo systemctl start falco\nsudo systemctl enable falco\n```\n\nMonitor `/var/log/syslog` on `kubenode01` while working with Falco:\n\n```bash\nsudo tail -f /var/log/syslog\n```\n\nLogin to master node and run test pod:\n\n```bash\n# vagrant ssh kubemaster\n\nkubectl run busybox --image=busybox --restart=Never --command -- sh -c \"sleep 3000\"\n```\n\nExec into the Pod:\n\n```bash\nkubectl exec -it busybox -- sh\n```\n\nFalco should report an event similar to the following:\n\n```console\nOct 13 12:33:36 kubenode01 falco[4717]: 12:33:36.936085881: Notice A shell was spawned in a container with an attached terminal (user=root user_loginuid=-1 busybox (id=2dd1f809726c) shell=sh parent=runc cmdline=sh terminal=34816 container_id=2dd1f809726c image=docker.io/library/busybox)\n```\n\nModify the `/etc/passwd` file in the `busybox` Pod:\n\n```bash\necho \"user\" \u003e\u003e /etc/passwd\n```\n\nFalco should detect this activity and log an event:\n\n```console\nOct 13 12:36:20 kubenode01 falco[4717]: 12:36:20.448778155: Error File below /etc opened for writing (user=root user_loginuid=-1 command=sh parent=\u003cNA\u003e pcmdline=\u003cNA\u003e file=/etc/passwd program=sh gparent=\u003cNA\u003e ggparent=\u003cNA\u003e gggparent=\u003cNA\u003e container_id=2dd1f809726c image=docker.io/library/busybox)\n```\n\nRun a Pod that periodically executes `apk update` to generate events for Falco\nto log:\n\n```bash\nkubectl apply -f - \u003c\u003c EOF\napiVersion: v1\nkind: Pod\nmetadata:\n  name: falco-test-pod\nspec:\n  containers:\n    - name: falco-test-container\n      image: alpine\n      command: [ \"/bin/sh\", \"-c\", \"while true ; do apk update ; sleep 10; done\" ]\n  restartPolicy: Never\nEOF\n```\n\nChange the `Launch Package Management Process in Container` rule\nin `/etc/falco/falco_rules.yaml` to log only \"container_name\" and \"image\".\n\nYou can see the the available fields here: [Supported Fields for Conditions and Outputs](https://falco.org/docs/rules/supported-fields/)\nor using command line:\n\n```bash\nfalco --list\n```\n\n```console\n$ sudo vi /etc/falco/falco_rules.local.yaml\n...\n# Container is supposed to be immutable. Package management should be done in building the image.\n- rule: Launch Package Management Process in Container\n  desc: Package management process ran inside container\n  condition: \u003e\n    spawned_process\n    and container\n    and user.name != \"_apt\"\n    and package_mgmt_procs\n    and not package_mgmt_ancestor_procs\n    and not user_known_package_manager_in_container\n#-\u003e This is the change:\n  output: \u003e\n    %container.name,%container.image.repository:%container.image.tag\n  priority: ERROR\n  tags: [process, mitre_persistence]\n```\n\nRestart Falco and check the logs:\n\n```console\n$ sudo systemctl restart falco\n$ sudo tail -f /var/log/syslog\nOct 14 08:07:58 kubenode01 falco: 08:07:58.846428059: Notice A shell was spawned in a container with an attached terminal (user=root user_loginuid=-1 test-nginx (id=73ad43b01267) shell=bash parent=runc cmdline=bash terminal=34816 container_id=73ad43b01267 image=docker.io/library/nginx)\nOct 13 12:36:20 kubenode01 falco[4717]: 12:36:20.448778155: Error File below /etc opened for writing (user=root user_loginuid=-1 command=sh parent=\u003cNA\u003e pcmdline=\u003cNA\u003e file=/etc/passwd program=sh gparent=\u003cNA\u003e ggparent=\u003cNA\u003e gggparent=\u003cNA\u003e container_id=2dd1f809726c image=docker.io/library/busybox)\n...\nOct 13 13:18:36 kubenode01 falco[24293]: 13:18:36.830684811: Error falco-test-container,docker.io/library/alpine:latest\nOct 13 13:18:36 kubenode01 falco: 13:18:36.830684811: Error falco-test-container,docker.io/library/alpine:latest\n```\n\n## Auditing\n\nEnable auditing for the Kubernetes cluster:\n\n```console\n$ sudo mkdir -pv /etc/kubernetes/audit\n$ sudo tee /etc/kubernetes/audit/audit-policy.yaml \u003c\u003c EOF\napiVersion: audit.k8s.io/v1\nkind: Policy\nrules:\n- level: Metadata\nEOF\n\n$ sudo vi /etc/kubernetes/manifests/kube-apiserver.yaml\n...\n    - --audit-policy-file=/etc/kubernetes/audit/audit-policy.yaml\n    - --audit-log-path=/etc/kubernetes/audit/logs/audit.log\n    - --audit-log-maxage=2\n    - --audit-log-maxbackup=3\n    - --audit-log-maxsize=1\n...\n    volumeMounts:\n    - mountPath: /etc/kubernetes/audit\n      name: audit\n...\n  volumes:\n  - hostPath:\n      path: /etc/kubernetes/audit\n      type: DirectoryOrCreate\n    name: audit\n...\n...\n```\n\nCreate test secret:\n\n```bash\nkubectl create secret generic test-audit-secret --from-literal password=0987654321\n```\n\nRetrieve audit logs related to `test-audit-secret`:\n\n```console\n$ sudo grep --no-filename test-audit-secret /etc/kubernetes/audit/logs/* | jq\n{\n  \"kind\": \"Event\",\n  \"apiVersion\": \"audit.k8s.io/v1\",\n  \"level\": \"Metadata\",\n  \"auditID\": \"eaf1c7d8-60c9-42e6-9c3a-a2aa0bc47985\",\n  \"stage\": \"ResponseComplete\",\n  \"requestURI\": \"/api/v1/namespaces/default/secrets?fieldManager=kubectl-create\",\n  \"verb\": \"create\",\n  \"user\": {\n    \"username\": \"kubernetes-admin\",\n    \"groups\": [\n      \"system:masters\",\n      \"system:authenticated\"\n    ]\n  },\n  \"sourceIPs\": [\n    \"192.168.56.2\"\n  ],\n  \"userAgent\": \"kubectl/v1.21.5 (linux/amd64) kubernetes/aea7bba\",\n  \"objectRef\": {\n    \"resource\": \"secrets\",\n    \"namespace\": \"default\",\n    \"name\": \"test-audit-secret\",\n    \"apiVersion\": \"v1\"\n  },\n  \"responseStatus\": {\n    \"metadata\": {},\n    \"code\": 201\n  },\n  \"requestReceivedTimestamp\": \"2021-10-13T15:27:09.287836Z\",\n  \"stageTimestamp\": \"2021-10-13T15:27:09.296157Z\",\n  \"annotations\": {\n    \"authorization.k8s.io/decision\": \"allow\",\n    \"authorization.k8s.io/reason\": \"\"\n  }\n}\n```\n\nChange the audit policy to log `RequestResponse` details only for ConfigMap\nresources, and Metadata for Secrets and ConfigMaps. NonResourceURLs like /api\nand /version will also be logged at RequestResponse level for authenticated\nusers. All other events will be set to None:\n\n```bash\n$ sudo tee /etc/kubernetes/audit/audit-policy.yaml \u003c\u003c EOF\napiVersion: audit.k8s.io/v1\nkind: Policy\nrules:\n  - level: RequestResponse\n    userGroups: [\"system:authenticated\"]\n    nonResourceURLs:\n    - \"/api*\"\n    - \"/version\"\n  - level: Metadata\n    resources:\n    - group: \"\"\n      resources: [\"secrets\", \"configmaps\"]\n  - level: None\nEOF\n```\n\n## AppArmor\n\n### AppArmor and Docker\n\nCheck the AppArmor status on the node:\n\n```console\n# vagrant ssh kubenode01\n# sudo su -\n\n$ aa-status\napparmor module is loaded.\n17 profiles are loaded.\n17 profiles are in enforce mode.\n   /sbin/dhclient\n   /usr/bin/lxc-start\n   /usr/bin/man\n   /usr/lib/NetworkManager/nm-dhcp-client.action\n   /usr/lib/NetworkManager/nm-dhcp-helper\n   /usr/lib/connman/scripts/dhclient-script\n   /usr/lib/snapd/snap-confine\n   /usr/lib/snapd/snap-confine//mount-namespace-capture-helper\n   /usr/sbin/tcpdump\n   cri-containerd.apparmor.d\n   docker-default\n   lxc-container-default\n   lxc-container-default-cgns\n   lxc-container-default-with-mounting\n   lxc-container-default-with-nesting\n   man_filter\n   man_groff\n0 profiles are in complain mode.\n7 processes have profiles defined.\n7 processes are in enforce mode.\n   cri-containerd.apparmor.d (6900)\n   cri-containerd.apparmor.d (10512)\n   cri-containerd.apparmor.d (10570)\n   cri-containerd.apparmor.d (30135)\n   cri-containerd.apparmor.d (30192)\n   cri-containerd.apparmor.d (30271)\n   cri-containerd.apparmor.d (30362)\n0 processes are in complain mode.\n0 processes are unconfined but have a profile defined.\nroot@kubemaster:/home/vagrant# aa-status\napparmor module is loaded.\n17 profiles are loaded.\n17 profiles are in enforce mode.\n   /sbin/dhclient\n   /usr/bin/lxc-start\n   /usr/bin/man\n   /usr/lib/NetworkManager/nm-dhcp-client.action\n   /usr/lib/NetworkManager/nm-dhcp-helper\n   /usr/lib/connman/scripts/dhclient-script\n   /usr/lib/snapd/snap-confine\n   /usr/lib/snapd/snap-confine//mount-namespace-capture-helper\n   /usr/sbin/tcpdump\n   cri-containerd.apparmor.d\n   docker-default\n   lxc-container-default\n   lxc-container-default-cgns\n   lxc-container-default-with-mounting\n   lxc-container-default-with-nesting\n   man_filter\n   man_groff\n0 profiles are in complain mode.\n7 processes have profiles defined.\n7 processes are in enforce mode.\n   cri-containerd.apparmor.d (6900)\n   cri-containerd.apparmor.d (10512)\n   cri-containerd.apparmor.d (10570)\n   cri-containerd.apparmor.d (30135)\n   cri-containerd.apparmor.d (30192)\n   cri-containerd.apparmor.d (30271)\n   cri-containerd.apparmor.d (30362)\n0 processes are in complain mode.\n0 processes are unconfined but have a profile defined.\n```\n\nCreate an AppArmor profile for Docker and Nginx:\n\n```bash\ncat \u003e /etc/apparmor.d/docker-nginx \u003c\u003c EOF\n#include \u003ctunables/global\u003e\n\nprofile docker-nginx flags=(attach_disconnected,mediate_deleted) {\n  #include \u003cabstractions/base\u003e\n\n  network inet tcp,\n  network inet udp,\n  network inet icmp,\n\n  deny network raw,\n\n  deny network packet,\n\n  file,\n  umount,\n\n  deny /bin/** wl,\n  deny /boot/** wl,\n  deny /dev/** wl,\n  deny /etc/** wl,\n  deny /home/** wl,\n  deny /lib/** wl,\n  deny /lib64/** wl,\n  deny /media/** wl,\n  deny /mnt/** wl,\n  deny /opt/** wl,\n  deny /proc/** wl,\n  deny /root/** wl,\n  deny /sbin/** wl,\n  deny /srv/** wl,\n  deny /tmp/** wl,\n  deny /sys/** wl,\n  deny /usr/** wl,\n\n  audit /** w,\n\n  /var/run/nginx.pid w,\n\n  /usr/sbin/nginx ix,\n\n  deny /bin/dash mrwklx,\n  deny /bin/sh mrwklx,\n  deny /usr/bin/top mrwklx,\n\n\n  capability chown,\n  capability dac_override,\n  capability setuid,\n  capability setgid,\n  capability net_bind_service,\n\n  deny @{PROC}/* w,   # deny write for all files directly in /proc (not in a subdir)\n  # deny write to files not in /proc/\u003cnumber\u003e/** or /proc/sys/**\n  deny @{PROC}/{[^1-9],[^1-9][^0-9],[^1-9s][^0-9y][^0-9s],[^1-9][^0-9][^0-9][^0-9]*}/** w,\n  deny @{PROC}/sys/[^k]** w,  # deny /proc/sys except /proc/sys/k* (effectively /proc/sys/kernel)\n  deny @{PROC}/sys/kernel/{?,??,[^s][^h][^m]**} w,  # deny everything except shm* in /proc/sys/kernel/\n  deny @{PROC}/sysrq-trigger rwklx,\n  deny @{PROC}/mem rwklx,\n  deny @{PROC}/kmem rwklx,\n  deny @{PROC}/kcore rwklx,\n\n  deny mount,\n\n  deny /sys/[^f]*/** wklx,\n  deny /sys/f[^s]*/** wklx,\n  deny /sys/fs/[^c]*/** wklx,\n  deny /sys/fs/c[^g]*/** wklx,\n  deny /sys/fs/cg[^r]*/** wklx,\n  deny /sys/firmware/** rwklx,\n  deny /sys/kernel/security/** rwklx,\n}\nEOF\n```\n\n```bash\napparmor_parser /etc/apparmor.d/docker-nginx\n```\n\n```console\n$ aa-status\napparmor module is loaded.\n18 profiles are loaded.\n18 profiles are in enforce mode.\n...\n   docker-nginx\n...\n```\n\nRun a standard Docker container with the Nginx image and attempt to run a shell\ninside:\n\n```console\n$ docker run -d nginx\n8ba791147b1b7e69d960faf1af5ff134e02572ecabd1b7a4d6087baa58ac4b2e\n$ docker exec -it 8ba791147b1b sh\n# sh\n```\n\nRepeat the process, but this time apply the AppArmor profile, which should\nprevent the shell from running:\n\n```console\n$ docker run --security-opt apparmor=docker-nginx -d nginx\nb5d752054efbcddb92e753ae9694e3cec734efd6cce207eef4d3fdead8163cc2\n$ docker exec -it b5d752054efb sh\n# sh\nsh: 1: sh: Permission denied\n```\n\n### AppArmor and Kubernetes\n\n```bash\nkubectl apply -f - \u003c\u003c EOF\napiVersion: v1\nkind: Pod\nmetadata:\n  name: nginx\n  labels:\n    run: nginx\n  annotations:\n    container.apparmor.security.beta.kubernetes.io/my-test-nginx: localhost/docker-nginx\nspec:\n  containers:\n  - image: nginx\n    name: my-test-nginx\nEOF\n```\n\nAttempt to execute a shell within the Pod (this action should be denied by the\n`docker-nginx` AppArmor profile):\n\n```console\n# kubectl exec -it nginx -- bash\nroot@nginx:/# sh\nbash: /bin/sh: Permission denied\n```\n\n## seccomp\n\n* [Restrict a Container's Syscalls with seccomp](https://kubernetes.io/docs/tutorials/clusters/seccomp/)\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fruzickap%2Fcks-notes","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fruzickap%2Fcks-notes","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fruzickap%2Fcks-notes/lists"}