https://github.com/tyriis/i-see-dead-pods

Get rid of `Pod was terminated in response to imminent node shutdown.` Pods forever.
https://github.com/tyriis/i-see-dead-pods

devops gitops gke helmrelease kubernetes manifests

Last synced: 26 days ago
JSON representation

Get rid of `Pod was terminated in response to imminent node shutdown.` Pods forever.

• Host: GitHub
• URL: https://github.com/tyriis/i-see-dead-pods
• Owner: tyriis
• License: mit
• Created: 2023-03-16T20:49:33.000Z (almost 2 years ago)
• Default Branch: main
• Last Pushed: 2024-05-23T08:07:52.000Z (8 months ago)
• Last Synced: 2024-05-23T09:27:24.563Z (8 months ago)
• Topics: devops, gitops, gke, helmrelease, kubernetes, manifests
• Language: Nix
• Homepage: https://github.com/tyriis/i-see-dead-pods
• Size: 14.6 KB
• Stars: 5
• Watchers: 2
• Forks: 3
• Open Issues: 1
• Metadata Files:
  • Readme: README.md
  • License: LICENSE

Awesome Lists containing this project

README

        


[![taskfile][taskfile-shield]][taskfile-url]

[![pre-commit][pre-commit-shield]][pre-commit-url]

# I see dead Pods

Get rid of `Pod was terminated in response to imminent node shutdown.` Pods forever.

  Table of Contents

- [Story](#story)

- [Setup](#setup)

  - [kubectl](#kubectl)

  - [kustomize](#kustomize)

  - [flux helmrelease](#flux-helmrelease)

## Story

In Kubernetes, pods can remain in a broken state for a long time if graceful shutdown is enabled. This state results in an alert getting fired by kube-prometheus-stack.

```console

[FIRING:1] Pod has been in a non-ready state for more than 15 minutes.

Severity:  warning

Description: Pod default/some-container-7fb4c4fbc5-gbjwm has been in a non-ready state for longer than 15 minutes.

Details:

  • alertname: KubePodNotReady \

  • namespace: default \

  • pod: some-container-7fb4c4fbc5-gbjwm \

  • prometheus: observability/kube-prometheus-stack-prometheus \

  • severity: warning

```

Most of the `solutions` on the internet describe an uncontrolled deletion of all Pods in `Error` or `Terminated` state.

Wich I consider as a bad idea, because you will not see anymore if real `Error` Pods are in your system.

These manifests provide a kubernetes `CronJob` deleting constantly all Pods with given criterias.

## Setup

### kubectl

You can apply the manifests manually:

```console

kubectl apply -f https://raw.githubusercontent.com/tyriis/i-see-dead-pods/main/manifests/service-account.yaml

kubectl apply -f https://raw.githubusercontent.com/tyriis/i-see-dead-pods/main/manifests/rbac.yaml

kubectl apply -f https://raw.githubusercontent.com/tyriis/i-see-dead-pods/main/manifests/cronjob.yaml

```

### kustomize

or with kustomize

### flux helmrelease

or with flux, see [helmrelease](./manifests/flux-helmrelease.yaml).

[taskfile-shield]: https://img.shields.io/badge/Taskfile-enabled-brightgreen?logo=task

[taskfile-url]: https://taskfile.dev/

[pre-commit-shield]: https://img.shields.io/badge/pre--commit-enabled-brightgreen?logo=pre-commit

[pre-commit-url]: https://github.com/pre-commit/pre-commit