https://github.com/livetocode/kubernetes-storage-stress-tests
https://github.com/livetocode/kubernetes-storage-stress-tests
Last synced: about 1 year ago
JSON representation
- Host: GitHub
- URL: https://github.com/livetocode/kubernetes-storage-stress-tests
- Owner: livetocode
- Created: 2017-12-29T00:59:51.000Z (over 8 years ago)
- Default Branch: master
- Last Pushed: 2018-01-12T03:55:44.000Z (over 8 years ago)
- Last Synced: 2025-02-05T01:31:24.440Z (over 1 year ago)
- Language: Shell
- Size: 2.93 KB
- Stars: 2
- Watchers: 2
- Forks: 0
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
Awesome Lists containing this project
README
# Description
This script will stress your Kubernetes node by running all versions of nodejs in order to verify the garbage collection or the container images.
See https://kubernetes.io/docs/concepts/cluster-administration/kubelet-garbage-collection/
# Steps
It will first pick the first schedulable node, then grab the list of all the tags of the official node repository, from DockerHub.
Finally, it will execute for each tag a "node --version" for the corresponding image.
# Observe
You can then monitor your disk usage with Prometheus/Grafana and notice the file system grow, then shrink after each cleanup.
You might encounter failing pods when disk is not sufficient. They won't retry since the restart policy is set to Never.
# Warning
The pods are run within the *docker-stress-test* namespace, which is always **deleted** then created at the beginning of each run.
You will see the pods in the kubernetes dashboard, but you might not have access to their logs once the garbage collector flushed the completed containers.
# Requirements
- kubectl / docker
- jq
- docker-ls
# Start
Ensure that your KUBECONFIG var is properly set and verify the options with the script (namespace, sleep period, tooling...), then run:
```
bash run-many-different-docker-images.sh
```
# Issues
## Node crash
We experienced node reboots after a few gigs of docker images.
We had to upgrade the kernel to fix this.
## Garbage collector won't work
We noticed that the data was not garbaged collected because we used either btrfs or xfs with overlay which were not properly monitored by the cadvisor verion of the kubelet.