{"id":15659596,"url":"https://github.com/nickstenning/learningfromincidents","last_synced_at":"2026-01-08T11:41:41.019Z","repository":{"id":66096424,"uuid":"212594313","full_name":"nickstenning/learningfromincidents","owner":"nickstenning","description":"Links and resources from my talk about how to learn more from incidents!","archived":false,"fork":false,"pushed_at":"2020-09-07T13:00:53.000Z","size":2540,"stargazers_count":21,"open_issues_count":0,"forks_count":1,"subscribers_count":6,"default_branch":"master","last_synced_at":"2025-02-05T05:25:23.192Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":null,"has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/nickstenning.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2019-10-03T13:59:05.000Z","updated_at":"2023-12-05T08:05:03.000Z","dependencies_parsed_at":null,"dependency_job_id":"78a781a4-0719-4218-9497-ca862872f960","html_url":"https://github.com/nickstenning/learningfromincidents","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/nickstenning%2Flearningfromincidents","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/nickstenning%2Flearningfromincidents/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/nickstenning%2Flearningfromincidents/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/nickstenning%2Flearningfromincidents/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/nickstenning","download_url":"https://codeload.github.com/nickstenning/learningfromincidents/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":246269911,"owners_count":20750319,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-10-03T13:17:43.556Z","updated_at":"2026-01-08T11:41:40.986Z","avatar_url":"https://github.com/nickstenning.png","language":null,"funding_links":[],"categories":[],"sub_categories":[],"readme":"# Learning from Incidents\n\nThis repository contains links to resources mentioned in a conference talk on\n**how to learn more effectively from incidents**. I most recently gave this\ntalk at [Agile \u0026 Automation Afternoons 2020], and have previously shared it at\n[Lead Dev Berlin 2019] and [SREcon 2019 EMEA], and internal conferences and\nevents at Microsoft.\n\nThis talk was co-written and originally co-presented with [Jessica\nDeVita](https://twitter.com/ubergeekgirl). Most of the meaningful insights in\nthe talk are hers, and all of the typos and errors are mine!\n\n\u003e ⚡💡 If you're looking for one thing to do after the talk, we recommend\n\u003e reading the [Etsy Debriefing Guide], particularly pp. 21-23, which give\n\u003e examples of good questions to ask in post-incident reviews. 💡⚡\n\n[Agile \u0026 Automation Afternoons 2020]: https://aadays.pl/afternoons/\n[Lead Dev Berlin 2019]: https://berlin2019.theleaddeveloper.com/talks#nick-stenning\n[SREcon 2019 EMEA]: https://www.usenix.org/conference/srecon19emea/presentation/stenning\n[Etsy Debriefing Guide]: https://aka.ms/etsydebriefing\n\n## Contents\n\n  * [Slides](#slides)\n  * [References](#references)\n  * [Recommendations](#recommendations)\n  * [Further reading/viewing](#further-readingviewing)\n  * [Even more!](#even-more)\n\n## Slides\n\nHere's [a link to a PDF of slides for the most recent version of the talk][slides].\n\n[slides]: 2019-12-learning-from-incidents.pdf\n\n## References\n\nThe two links referenced in our talk were:\n\n- [How Complex Systems Fail](https://aka.ms/csfail) is a short paper which\n  focuses on how complex systems differ from the linear engineered systems we\n  may have worked on in the past.\n\n- [The Etsy Debriefing Facilitation Guide](https://aka.ms/etsydebriefing)\n  contains specific guidance on how to run better post-incident reviews (which\n  the authors call \"debriefings\".)\n\n## Recommendations\n\nWhat follows are a series of recommendations for running better post-incident\nreviews and learning more from incidents.\n\n\u003e You shouldn't attempt to adopt all of these practices at once for every\n\u003e incident! Start small with _interesting_ incidents, and not necessarily your\n\u003e _biggest_ incidents.\n\nWe recommend that you break up your post-incident learning process into the\nfollowing stages:\n\n1. (Optionally) interview participants\n2. Run a facilitated post-incident review\n3. Run a separate meeting to plan repair items\n4. (Optionally) Publish written incident reports\n\n### 1. Run a facilitated post-incident review\n\n   At most a few days after an incident, get as many people as possible who\n   were involved in incident response into a room together to talk about what\n   happened.\n\n   Have a neutral facilitator whose job it is to guide the discussion. They\n   should not have been involved in incident response themselves, if at all\n   possible.\n\n   **Focus on reconstructing the timeline of the incident and understanding\n   how actions and decisions of operators made sense to them at the time**,\n   even if we know in hindsight that they were mistakes.\n\n   Limit your post-incident reviews to 60-90 minutes. You will probably have\n   to pick and choose what to talk about.\n\n### 2. Use 1:1 interviews for complex incidents\n\n   For many incidents, making effective use of a 60-90 minute incident review\n   meeting will be challenging unless the facilitator already has some idea\n   of the incident timeline.\n\n   **Use 1:1 interviews (often no longer than 10-15m each) with people\n   involved in the incident response to ask about their experience of the\n   incident.**\n\n   Use your interview notes to look for interesting points in the timeline:\n   points where hypotheses were formed or changed, when significant actions\n   or decisions were taken, or where individual views on the situation\n   diverged one from another.\n\n### 3. Keep discussion of repair items separate\n\n   Including discussion of repair items in the main post-incident review\n   meeting will make it difficult to keep focus on understanding what\n   happened during the incident.\n\n   You will likely find that talking about repair items leads people to start\n   discussing what *didn't* happen. This is fine in a meeting about possible\n   repairs, but it doesn't help us learn from what *did* happen.\n\n   **Have a separate meeting a day or two after the post-incident review, in\n   which you discuss and agree upon repair items.** This meeting can be\n   shorter and include fewer people (typically those who have a say in\n   prioritisation).\n\n### 4. Publish written incident reports\n\n   Not everyone on your team will attend every post-incident review. Writing\n   up reports (even on one or two pages) can provide a way for the rest of\n   your team to share in what was learned.\n\n   It may make sense to prepare different documents for different audiences.\n   Your immediate team may gain more from a detailed description of how the\n   system surprised you. Your management chain may be more interested in\n   understanding how repairs will reduce the customer impact in future.\n   **Don't be afraid to prepare to different documents for different\n   audiences.**\n\n## Further reading/viewing\n\nIf you're interested in learning more about this field, here is a curated\nselection of further references:\n\n### Practical guidance\n\n- [USDA Forest Service Learning Review\n  Guide](https://aka.ms/usdalearningreviews): a guidebook for learning reviews\n  conducted by the USDA Forest Service. Much of the content in this guide is\n  applicable to our industry as well.\n\n### Accessible research\n\n- [The Field Guide to Understanding 'Human\n  Error'](https://www.oreilly.com/library/view/the-field-guide/9781317031833/)\n  ([Amazon](https://www.amazon.com/Field-Guide-Understanding-Human-Error/dp/1472439058))\n  is a short and accessible introduction to how \"human error\" is a\n  problematic concept in incident investigations.\n\n### Deeper research\n\n- [Reconstructing human contributions to accidents: the new view on error and\n  performance (Dekker,\n  2002)](https://www.sciencedirect.com/science/article/pii/S0022437502000324)\n  is a paper which investigates how human contributions to accidents are\n  often \"reconstructed\" during an accident investigation in ways that are not\n  conducive to learning or preventing future accidents.\n\n- [Behind Human\n  Error](https://www.amazon.com/Behind-Human-Error-David-Woods/dp/0754678342)\n  is a collection of papers covering the latest research into human factors\n  in incidents and accidents.\n\n### Videos\n\nSome people learn better from videos! Here is a collection of talks and\neducational videos which may be of interest:\n\n- [\"Who Destroyed Three Mile Island?\" (Nickolas Means at Lead Dev London 2018)](https://www.youtube.com/watch?v=hMk6rF4Tzsg)\n\n- [\"Life After Human Error\" (Stephen Shorrock at Velocity Europe 2014)](https://www.youtube.com/watch?v=STU3Or6ZU60)\n\n- [\"How Complex Systems Fail\" (Richard Cook at Velocity 2012)](https://www.youtube.com/watch?v=2S0k12uZR14)\n\n## Even more!\n\nHere's a [further introductory guide to resilience engineering for the\nsoftware community](https://github.com/res-eng/resilience-for-software) put\ntogether by [Lorin Hochstein](https://twitter.com/lhochstein) and [Jacob\nScott](https://twitter.com/jhscott).\n\nIf your thirst for knowledge is _still_ not quenched, you will find an\nenormous quantity of research and writing on this topic assembled by Lorin in [his `resilience-engineering` repository](https://github.com/lorin/resilience-engineering/blob/master/intro.md).","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fnickstenning%2Flearningfromincidents","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fnickstenning%2Flearningfromincidents","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fnickstenning%2Flearningfromincidents/lists"}