{"id":13990535,"url":"https://github.com/bbc/chaos-lambda","last_synced_at":"2025-05-09T00:07:41.990Z","repository":{"id":136960067,"uuid":"47708500","full_name":"bbc/chaos-lambda","owner":"bbc","description":"Randomly terminate ASG instances during business hours","archived":false,"fork":false,"pushed_at":"2024-07-10T09:34:39.000Z","size":86,"stargazers_count":163,"open_issues_count":0,"forks_count":24,"subscribers_count":33,"default_branch":"master","last_synced_at":"2025-05-09T00:07:35.847Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"other","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/bbc.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2015-12-09T17:47:29.000Z","updated_at":"2024-10-25T02:19:24.000Z","dependencies_parsed_at":null,"dependency_job_id":"b0ab4485-8746-40a7-8a4d-0792fe69e7da","html_url":"https://github.com/bbc/chaos-lambda","commit_stats":null,"previous_names":[],"tags_count":3,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/bbc%2Fchaos-lambda","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/bbc%2Fchaos-lambda/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/bbc%2Fchaos-lambda/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/bbc%2Fchaos-lambda/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/bbc","download_url":"https://codeload.github.com/bbc/chaos-lambda/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":253166520,"owners_count":21864482,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-08-09T13:02:53.544Z","updated_at":"2025-05-09T00:07:41.963Z","avatar_url":"https://github.com/bbc.png","language":"Python","readme":"# About\n\nEC2 instances are volatile and can be recycled at any time without warning.\nAmazon recommends running them under Auto Scaling Groups to ensure overall\nservice availability, but it's easy to forget that instances can suddenly fail\nuntil it happens in the early hours of the morning when everyone is on holiday.\n\nChaos Lambda increases the rate at which these failures occur during business\nhours, helping teams to build services that handle them gracefully.\n\n\n# Quick setup\n\nCreate the lambda function in the region you want it to target using the\n`cloudformation/templates/lambda_standalone.json` CloudFormation template.\nThere are two parameters you may want to change:\n* `Schedule`: change if the default run times don't suit you (once per hour\n  between 10am UTC and 4pm UTC, Monday to Friday); see\n  http://docs.aws.amazon.com/AmazonCloudWatch/latest/events/ScheduledEvents.html\n  for documentation on the syntax.\n* `DefaultProbability`: by default all Auto Scaling Groups in the region are\n  targets; set this to `0.0` and only ASGs with a `chaos-lambda-termination`\n  tag (see below) will be affected.\n\n\n# Notifications\n\n## Termination Topic\n\nBy deploying the `lambda_standalone.json` CloudFormation template, an SNS topic\nwill be created with the name `ChaosLambdaTerminationTopic`. For each instance\nthat gets terminated, a notification will be published using this structure:\n\n```json\n{\n  \"event_name\": \"chaos_lambda.terminating\",\n  \"asg_name\": \"my-autoscaling-group\",\n  \"instance_id\": \"i-00001234\"\n}\n```\n\nBy default, no subscriptions are created to this topic, so it is up to you to\nsubscribe a queue or another lambda if you wish.\n\n## Failure topic\n\nTo receive notifications if the lambda function fails for any reason, create\nanother stack using the `cloudformation/templates/alarms.json` template.  This\ntakes the lambda function name (something similar to\n`chaos-lambda-ChaosLambdaFunction-EM2XNWWNZTPW`) and the email address to\nsend the alerts to.\n\n\n# Probability of termination\n\nEvery time the lambda triggers it examines all the Auto Scaling Groups in the\nregion and potentially terminates one instance in each.  The probability of\ntermination can be changed at the ASG level with a tag, and at a global level\nwith the `DefaultProbability` stack parameter.\n\nAt the ASG level the probability can be controlled by adding a\n`chaos-lambda-termination` tag with a value between `0.0` (never terminate) and\n`1.0` (always terminate).  Typically this would be used to opt out a legacy\nsystem (`0.0`).\n\nThe `DefaultProbability` parameter sets the probability of termination for any\nASG _without_ a valid `chaos-lambda-termination` tag.  If set to `0.0` the\nsystem becomes \"opt-in\", where any ASG without this tag is ignored.  The\ndefault is `0.166` (or 1 in 6).\n\n\n# Enabling/disabling\n\nThe lambda is triggered by a CloudWatch Events rule, the name of which can be\nfound from the `ChaosLambdaFunctionOutput` output of the lambda stack.  Locate\nthis rule in the AWS console under the Rules section of the CloudWatch service,\nand you can disable or enable it via the `Actions` button.\n\n\n# Regions\n\nBy default the lambda will target ASGs running in the same region.  It's\ngenerally a good idea to avoid cross-region actions, but if necessary an\nalternative list of one or more region names can be specified in the `Regions`\nstack parameter.\n\nThe value is a comma separated list of region names with optional whitespace,\nso the following are all valid and equivalent:\n* `ap-south-1,eu-west-1,us-east-1`\n* `ap-south-1, eu-west-1, us-east-1`\n* `ap-south-1 , eu-west-1 , us-east-1`\n\n\n# Log messages\n\nChaos Lambda log lines always start with a timestamp and a word specifying the\nevent type.  The timestamp is of the form `YYYY-MM-DDThh:mm:ssZ`, eg\n`2015-12-11T14:00:37Z`, and the timezone will always be `Z`.  The different\nevent types are described below.\n\n## bad-probability\n\n`\u003ctimestamp\u003e bad-probability [\u003cvalue\u003e] in \u003casg name\u003e`\n\nExample:\n\n`2015-12-11T14:07:21Z bad-probability [not often] in test-app-ASG-7LJI5SY4VX6T`\n\nIf the value of the `chaos-lambda-termination` tag isn't a number between `0.0`\nand `1.0` inclusive then it will be logged in one of these lines.  The square\nbrackets around the value allow CloudWatch Logs to find the full value even if\nit contains spaces.\n\n## result\n\n`\u003ctimestamp\u003e result \u003cinstance id\u003e is \u003cstate\u003e`\n\nExample:\n\n`2015-12-11T14:00:40Z result i-fe705d77 is shutting-down`\n\nAfter asking EC2 to terminate each of the targeted instances the new state of\neach is logged with a `result` line.  The `\u003cstate\u003e` value is taken from the\n`code` property of the `InstanceState` AWS type described at\nhttp://docs.aws.amazon.com/AWSEC2/latest/APIReference/API_InstanceState.html\n\n## targeting\n\n`\u003ctimestamp\u003e targeting \u003cinstance id\u003e in \u003casg name\u003e`\n\nExample:\n\n`2015-12-11T14:00:38Z targeting i-168f9eaf in test-app-ASG-1LOMEKEVBXXXS`\n\nThe `targeting` lines list all of the instances that are about to be\nterminated, before the `TerminateInstances` call occurs.\n\n## triggered\n\n`\u003ctimestamp\u003e triggered \u003cregion\u003e`\n\nExample:\n\n`2015-12-11T14:00:37Z triggered eu-west-1`\n\nGenerated when the lambda is triggered, indicating the region that will be\naffected.\n","funding_links":[],"categories":["3. Fault Injection","Python","Notable Tools"],"sub_categories":["Cloud"],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fbbc%2Fchaos-lambda","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fbbc%2Fchaos-lambda","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fbbc%2Fchaos-lambda/lists"}