{"id":13644582,"url":"https://github.com/xing/kubernetes-oom-event-generator","last_synced_at":"2026-02-25T15:02:55.914Z","repository":{"id":44357916,"uuid":"165060155","full_name":"xing/kubernetes-oom-event-generator","owner":"xing","description":"Generate a Kubernetes Event when a Pod's container has been OOMKilled","archived":false,"fork":false,"pushed_at":"2023-08-30T14:06:51.000Z","size":176,"stargazers_count":168,"open_issues_count":8,"forks_count":21,"subscribers_count":23,"default_branch":"master","last_synced_at":"2025-04-03T03:06:13.383Z","etag":null,"topics":["events","k8s","kubernetes","monitoring","olympus","prometheus","prometheus-metrics"],"latest_commit_sha":null,"homepage":"","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/xing.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":"docs/CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null}},"created_at":"2019-01-10T12:56:34.000Z","updated_at":"2025-03-25T08:28:02.000Z","dependencies_parsed_at":"2024-01-14T09:32:41.588Z","dependency_job_id":null,"html_url":"https://github.com/xing/kubernetes-oom-event-generator","commit_stats":null,"previous_names":[],"tags_count":4,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/xing%2Fkubernetes-oom-event-generator","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/xing%2Fkubernetes-oom-event-generator/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/xing%2Fkubernetes-oom-event-generator/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/xing%2Fkubernetes-oom-event-generator/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/xing","download_url":"https://codeload.github.com/xing/kubernetes-oom-event-generator/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":250040518,"owners_count":21365123,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["events","k8s","kubernetes","monitoring","olympus","prometheus","prometheus-metrics"],"created_at":"2024-08-02T01:02:08.379Z","updated_at":"2026-02-25T15:02:50.870Z","avatar_url":"https://github.com/xing.png","language":"Go","funding_links":[],"categories":["Go","OPS"],"sub_categories":[],"readme":"# kubernetes-oom-event-generator\n\n[![Build Status](https://travis-ci.org/xing/kubernetes-oom-event-generator.svg?branch=master)](https://travis-ci.org/xing/kubernetes-oom-event-generator)\n\nGenerates Kubernetes Event when a container is starting and indicates that\nit was previously out-of-memory killed.\n\n## Design\n\nThe Controller listens to the Kubernetes API for new Events and changes to\nEvents. Every time a notification regarding an Event is received it checks\nwhether this Event refers to a \"ContainerStarted\" event, based on the `Reason`\nfor the Event and the `Kind` of the involved object. If this is the case\nand the Event constitutes a change (meaning it is not a not-changing update,\nwhich happens when the resync, that is executed every two minutes, is run) it checks\nthe underlying Pod resource. Should the `LastTerminationState` of the Pod refer to\nan OOM kill the controller will emit a Kubernetes Event with a level of `Warning`\nand a reason of `PreviousContainerWasOOMKilled`.\n\n## Usage\n\n    Usage:\n      kubernetes-oom-event-generator [OPTIONS]\n\n    Application Options:\n      -v, --verbose= Show verbose debug information [$VERBOSE]\n          --version  Print version information\n\n    Help Options:\n      -h, --help     Show this help message\n\nRun the pre-built image [`xingse/kubernetes-oom-event-generator`] locally (with\nlocal permission):\n\n    echo VERBOSE=2 \u003e\u003e .env\n    docker run --env-file=.env -v $HOME/.kube/config:/root/.kube/config xingse/kubernetes-oom-event-generator\n\n## Deployment\n\nExample Clusterrole:\n\n    ---\n    apiVersion: rbac.authorization.k8s.io/v1beta1\n    kind: ClusterRole\n    metadata:\n      name: xing:controller:kubernetes-oom-event-generator\n    rules:\n      - apiGroups:\n          - \"\"\n        resources:\n          - pods\n          - pods/status\n        verbs:\n          - get\n          - list\n          - watch\n      - apiGroups:\n          - \"\"\n        resources:\n          - events\n        verbs:\n          - create\n          - patch\n          - list\n          - watch\n\nRun this controller on Kubernetes with the following commands:\n\n    kubectl create serviceaccount kubernetes-oom-event-generator \\\n      --namespace=kube-system\n\n    kubectl create -f path/to/example-clusterrole.yml\n    # alternatively run: `cat | kubectl create -f -` and paste the above example, hit Ctrl+D afterwards.\n\n    kubectl create clusterrolebinding xing:controller:kubernetes-oom-event-generator \\\n      --clusterrole=xing:controller:kubernetes-oom-event-generator \\\n      --serviceaccount=kube-system:kubernetes-oom-event-generator\n\n    kubectl run kubernetes-oom-event-generator \\\n      --image=xingse/kubernetes-oom-event-generator \\\n      --env=VERBOSE=2 \\\n      --serviceaccount=kubernetes-oom-event-generator \\\n      --namespace=kube-system\n\n## Alerting on OOM killed pods\n\nThere are many different ways to send alerts when an OOM occurs. We just want to\nmention two of them here.\n\n### Forwarding OOM events to Graylog\n\nGraylog is a popular log management solution, and it includes an alerting feature.\nSee the [Graylog docs] for more details.\n\nAt XING we forward all Kubernetes cluster events to Graylog using our\n[kubernetes-event-forwarder-gelf]. This allows us to configure alerts whenever a\n`PreviousContainerWasOOMKilled` event generated by the `kubernetes-oom-event-generator`\noccurs.\n\n### Using kube-state-metrics and Prometheus alerts\n\nWhen [kube-state-metrics] is deployed in the cluster and a [Prometheus] installation\nis scraping the metrics, you can alert on OOM-killed pods using the prometheus alert manager.\n\nExample alert:\n\n    alert: ComponentOutOfMemory\n    expr: sum_over_time(kube_pod_container_status_terminated_reason{reason=\"OOMKilled\"}[5m])\n      \u003e 0\n    for: 10s\n    labels:\n      severity: warning\n    annotations:\n      description: Critical Pod {{$labels.namespace}}/{{$labels.pod}} was OOMKilled.\n\nThe downside is that `kube_pod_container_status_terminated_reason` always returns to 0 once\na container starts back up. See the introduction of\n[`kube_pod_container_status_last_terminated_reason`] for more details.\n\n# Developing\n\nYou will need a working Go installation (1.11+) and the `make` program.  You will also\nneed to clone the project to a place outside you normal go code hierarchy (usually\n`~/go`), as it uses the new [Go module system].\n\nAll build and install steps are managed in the central `Makefile`. `make test` will fetch\nexternal dependencies, compile the code and run the tests. If all goes well, hack along\nand submit a pull request. You might need to modify the `go.mod` to specify desired\nconstraints on dependencies.\n\nMake sure to run `go mod tidy` before you check in after changing dependencies in any way.\n\n[Go module system]: https://github.com/golang/go/wiki/Modules\n[`xingse/kubernetes-oom-event-generator`]: https://hub.docker.com/r/xingse/kubernetes-oom-event-generator\n[Graylog docs]: https://docs.graylog.org/\n[kubernetes-event-forwarder-gelf]: https://github.com/xing/kubernetes-event-forwarder-gelf\n[kube-state-metrics]: https://github.com/kubernetes/kube-state-metrics\n[Prometheus]: https://prometheus.io\n[`kube_pod_container_status_last_terminated_reason`]: https://github.com/kubernetes/kube-state-metrics/pull/535\n\n## Releases\n\nReleases are a two-step process, beginning with a manual step:\n\n* Create a release commit\n  * Increase the version number in [kubernetes-oom-event-generator.go/VERSION](kubernetes-oom-event-generator.go#20)\n  * Adjust the [CHANGELOG](CHANGELOG.md)\n* Run `make release`, which will create an image, retrieve the version from the\n  binary, create a git tag and push both your commit and the tag\n\nThe Travis CI run will then realize that the current tag refers to the current master commit and\nwill tag the built docker image accordingly.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fxing%2Fkubernetes-oom-event-generator","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fxing%2Fkubernetes-oom-event-generator","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fxing%2Fkubernetes-oom-event-generator/lists"}