Ecosyste.ms: Awesome
An open API service indexing awesome lists of open source software.
https://github.com/k0rventen/k8s-watchtower
A monitoring & observability stack for k8s. Includes apps logs & ressources usage, node-level metrics and RBAC audit parsing.
https://github.com/k0rventen/k8s-watchtower
Last synced: about 10 hours ago
JSON representation
A monitoring & observability stack for k8s. Includes apps logs & ressources usage, node-level metrics and RBAC audit parsing.
- Host: GitHub
- URL: https://github.com/k0rventen/k8s-watchtower
- Owner: k0rventen
- Created: 2023-04-28T16:34:02.000Z (over 1 year ago)
- Default Branch: main
- Last Pushed: 2024-04-08T11:40:55.000Z (10 months ago)
- Last Synced: 2024-04-08T12:47:47.173Z (10 months ago)
- Size: 1.98 MB
- Stars: 3
- Watchers: 2
- Forks: 0
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
Awesome Lists containing this project
README
# k8s-watchtower
A minimal k8s observability stack which includes:
- host level & kubernetes metrics
![host](.github/host_dash.png)
- RBAC audit log monitoring
![audit](.github/audit_dash.png)
- workloads metrics & logs aggregation
![host](.github/apps_dash.png)## Components:
- `promtail` for log application logs & k8s audit log aggregation
- `telegraf` for node-level monitoring & kubernetes level metrics
- `influx` & `loki` for storing metrics and logs
- `grafana` for the dashboarding## Deployment
You'll need at least `helm` installed. You can refer to [this section](#enabling-the-audit-log) if you want to enable the audit log beforehand.
Add the repo to helm and get the default values.yaml file
```sh
helm repo add watchtower https://k0rventen.github.io/k8s-watchtower/
helm show values watchtower/watchtower > defaults.yaml
```
Review and modify if necessary the values according to your setup (mainly for the audit Log file part)Then deploy (preferably in a separate namespace):
```sh
helm upgrade --install --namespace watchtower --create-namespace watchtower -f defaults.yaml watchtower/watchtower
```Once deployed, port-forward the `grafana` service to your machine
```sh
kubectl port-forward -n watchtower svc/watchtower-grafana 3000
```then access the dashboards at `http://localhost:3000`.
### Enabling the Audit Log
For a more thorough documentation on the audit subject, head over to the [kubernetes doc](https://kubernetes.io/docs/tasks/debug/debug-cluster/audit/).
Add the following arguments to the APIserver config (args self explanatory):
```sh
--audit-log-maxbackup=3
--audit-log-maxsize=1024
--audit-log-maxage=30
--audit-policy-file=/etc/kubernetes/kube-api-audit-policy.yaml
--audit-log-path=/var/log/k8s-apiserver-audit.log
```Create a new file at the path you specified in the `audit-policy-file` argument, which defines your audit policy:
```yaml
apiVersion: audit.k8s.io/v1
kind: Policy
omitStages:
- "RequestReceived"
- "ResponseStarted"
rules:
- level: Metadata
namespaces: ["app","app2"]
```In this example we ask the cluster to log every request at the metadata level, in namespaces `app` and `app2`. If you don't specify `namespaces`, all namespaces will be audited.
Again, you can refer to [the doc](https://kubernetes.io/docs/tasks/debug/debug-cluster/audit/) if you want to tweak your audit policy further.
### Testing on kind
If you want a quick test environment to assess the stack, you can do so using [kind](https://kind.sigs.k8s.io/):
Clone the repo `git clone https://github.com/k0rventen/k8s-watchtower.git` then `cd` into it.
The `kind/kind-config.yaml` defines a 3 nodes cluster with `audit/policy.yaml` mounted on the control plane.
You can launch this cluster using:```
kind create cluster --config kind/kind-config.yaml
```Once it's up you can change your kubeconfig to target the new cluster, then apply the helm chart as usual:
```
helm upgrade --install --namespace watchtower --create-namespace watchtower -f values.yaml ./
```You can now check on the watchtower dashboards while creating new workloads:
```
kubectl create deploy --image traefik/whoami --replicas 7 whoami
```You can also check the RBAC dash to visualize API access while creating the resources !
This will launch a pod that will generate a unauthorized request to the kube api:
```
kubectl create ns app
kubectl run -n app -it --rm --image bitnami/kubectl --restart Never rbac-test get pods
```