Pepr Postmortem Doc

[cmwylie19]

Pepr Postmortem Doc

[cmwylie19]

# Pepr Missing K8s Events Post-Incident Review

## Executive Summary

_High-Level details describing the issue, Impact (impacted Mission Heros, etc.) -
quantified if possible, and critical steps to resolution._

### What went well

_List some things that went well in this process - some positive thinking is a
good place to start_

* ...
* ...
* ...
* ...

### What went not-so-well

_List things that we wish had not happened.  This is *not* about individuals but
the systems that contributed to the issue._

## Incident Timeline

_Roughly day/week level granularity of critical moments from
discovery/identification to resolution of the issue._

*April 20, 2024*

* Issue was first identified by ...

...

*May 23, 2024*

* Issue resolved by merging `github.com/defenseunicorns/pepr/pull/825`

## Contributing Factors

_List external factors that complicated delivery of a solution._

* The debugging feedback loop was *very* slow due to the time to failure
  (~35min) and difficult to automate.

* ...
* ...
* ...
* ...

## 5 Why(s)?

_*AFTER* the pre-read of the above let's dig into the causes of the incident to
find more systemic issues we can resolve._

1. Computers are sadness...
  Computers _are_ sadness... but why?

2. Computers are built by humans!
  Computers _are_ built by humans, but why?

3. Skynet does not exist!
  Skynet _does not_ exist - but why?

4. Humans have not yet trained the AI to build Skynet!
  Why haven't humans yet built Skynet?!

5. ...

## Action Items

_To be completed last - after 5(ish) why's let's make some decisions_

1. ...
2. ...
3. ...
4. ...
5. ...