{"id":19326622,"url":"https://github.com/thought-machine/spot-interruption-exporter","last_synced_at":"2026-05-15T08:02:21.447Z","repository":{"id":196867410,"uuid":"696716991","full_name":"thought-machine/spot-interruption-exporter","owner":"thought-machine","description":null,"archived":false,"fork":false,"pushed_at":"2024-01-16T16:47:37.000Z","size":110,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":25,"default_branch":"master","last_synced_at":"2025-01-06T07:12:11.787Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/thought-machine.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2023-09-26T09:52:33.000Z","updated_at":"2024-04-24T15:31:23.000Z","dependencies_parsed_at":"2023-11-07T15:35:06.963Z","dependency_job_id":"7f3f32df-ec4d-4afe-bf70-3a9d74b3108b","html_url":"https://github.com/thought-machine/spot-interruption-exporter","commit_stats":null,"previous_names":["thought-machine/spot-interruption-exporter"],"tags_count":9,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/thought-machine%2Fspot-interruption-exporter","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/thought-machine%2Fspot-interruption-exporter/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/thought-machine%2Fspot-interruption-exporter/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/thought-machine%2Fspot-interruption-exporter/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/thought-machine","download_url":"https://codeload.github.com/thought-machine/spot-interruption-exporter/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":240427460,"owners_count":19799498,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-10T02:14:20.671Z","updated_at":"2025-11-17T08:03:05.343Z","avatar_url":"https://github.com/thought-machine.png","language":"Go","funding_links":[],"categories":[],"sub_categories":[],"readme":"# spot-interruption-exporter\n\nPublishes a prometheus metric `interruption_events_total` that increments by 1 whenever a spot instance has been preempted.\n\nThis is a very helpful metric, as it\n\n- helps correlate workload issues with spot interruption times\n\n- can aid in seeing if certain flavours are more susceptible to interruption\n\n- can aid in seeing how much more susceptible single-zone clusters are to interruption\n\n- can be used as a signal on whether to promote spot instances to other environments\n\nThe app can be expanded to support other cloud providers, but currently is only built for GCP.\n\nA single deployment of the infrastructure and app is intended to serve all Kubernetes clusters in a given project.\n\n## How it works\n\nSpot preemption events are emitted as an audit log that contain the compute instance ID. These audit logs are forwarded to a pubsub topic via GCP Log Sink. The app then subscribes to this topic and handles the interruption event.\n\nThe audit log for instance preemption does not contain information about the Kubernetes cluster the instance may or may not have been associated with. Since the node is already deleted by the time the preemption event is received, the compute API cannot be queried for more information.\n\nTo work around this, the app keeps a mapping of compute instance ID to Kubernetes cluster. It can then use this when processing preemption events to publish the correct `kubernetes_cluster` label on the metric.\n\nA second log router + pubsub topic exist to inform the app of new instances that belong to a Kubernetes cluster. On app startup, the compute API is queried to seed the mapping.\n\n![spot-interruption-exporter-gcp](https://github.com/thought-machine/spot-interruption-exporter/assets/11613073/f2b01b81-1d13-4a2d-8303-9c842b51b3f7)\n\n## Config\n\nThe app reads in a config file from `$CONFIG_PATH` with the structure below.\n\n```yaml\nlog_level: debug\nproject_name: example-project\npubsub:\n  instance_creation_subscription_name: sie-creation-subscription\n  instance_interruption_subscription_name: sie-interruption-subscription\nprometheus:\n  port: 8090\n  path: /metrics\n```\n\n## Deploying\n\n### Infrastructure\n\nYou'll need to deploy the required infrastructure before standing up the application.\n\nThe infrastructure that the app depends for GCP on can be created via\n```bash\n$ terraform -chdir=infra/gcp init\n$ terraform -chdir=infra/gcp apply\n```\n\nand can be destroyed via\n```bash\n$ terraform -chdir=infra/gcp destroy\n```\n\n### Kubernetes manifests\n\n`kustomize/` holds relevant kubernetes config files. You will likely want to overlay the base resources. For an example of how you might do this, see `kustomize/example-overlay`.\n\n## Verifying\n\nYou can send a test interruption message via\n```bash\n$ gcloud pubsub topics publish sie-interruption-topic --project \u003cproject\u003e --message '{\n  \"protoPayload\": {\n    \"@type\": \"type.googleapis.com/google.cloud.audit.AuditLog\",\n    \"methodName\": \"compute.instances.preempted\",\n    \"resourceName\": \"projects/mock-project/zones/europe-west1-c/instances/mock-instance-spot-3706-5b909138-nr65\"\n  }\n}'\n```\n\nYou can send a test instance creation message via\n```bash\n$ gcloud pubsub topics publish sie-creation-topic --project \u003cproject\u003e --message '{\n  \"protoPayload\": {\n    \"@type\": \"type.googleapis.com/google.cloud.audit.AuditLog\",\n    \"serviceName\": \"compute.googleapis.com\",\n    \"methodName\": \"v1.compute.instances.insert\",\n    \"resourceName\": \"projects/123456789/zones/europe-west1-c/instances/fake-resource\",\n    \"request\": {\n      \"labels\": [\n        {\n          \"key\": \"goog-k8s-cluster-name\",\n          \"value\": \"fake-cluster\"\n        }\n      ]\n    }\n  }\n}'\n```\n\nAfter sending a few messages, you can view the metric count increasing\n```bash\n$ curl localhost:8080/metrics | grep interruption\n# HELP interruption_events_total The total number of interruption events for a given cluster\n# TYPE interruption_events_total counter\ninterruption_events_total{kubernetes_cluster=\"kubernetes-cluster\"} 6\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fthought-machine%2Fspot-interruption-exporter","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fthought-machine%2Fspot-interruption-exporter","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fthought-machine%2Fspot-interruption-exporter/lists"}