Postmortem #

Every engineering team that works long enough will experience incidents — production outages, data corruption, suddenly degrading performance, or deployments causing unexpected problems. What separates mature teams from immature ones isn’t whether they have incidents, but what they do afterward. A postmortem is the systematic mechanism for turning every failure into collective knowledge: not to find who’s at fault, but to understand how the system — processes, tooling, communication, and architecture — can be designed better so the same thing doesn’t recur. This article covers the philosophy behind effective postmortems, how to lead them, how to document them, and the anti-patterns that most often destroy their value.

What Is a Postmortem? #

A postmortem is a structured reflection process conducted after a significant incident — usually within 24-72 hours of the incident being resolved — with the goal of understanding what happened, why it happened, and how to prevent the same thing from recurring.

A postmortem is NOT:

✗ A session to find who's at fault
✗ A report to management about who "underperformed"
✗ A formality ritual done for compliance
✗ An individual performance evaluation
✗ An opportunity to embarrass the engineer who made the mistake

A postmortem IS:

✓ A collective analysis of system failure
✓ A chance to learn from conditions rarely seen in production
✓ A process for identifying real systemic improvements
✓ A way to build trust in the team that honest speaking is safe
✓ Documentation that helps prevent the same failure in the future

The Blameless Postmortem Philosophy #

The “blameless postmortem” concept was popularized by Google SRE and became the standard in modern software engineering. The idea is simple but its implications run deep: when something fails, we don’t look for a person to blame — we look for the system that needs fixing.

Why Blame Is Useless #

A blame-based postmortem scenario:

Incident: Engineer A deployed code causing 2 hours of downtime
Postmortem: "Engineer A didn't test properly before deploying"
Action item: "Engineer A must be more careful"
Result: Engineer A feels ashamed, the rest of the team is relieved it wasn't them
       Systemic problems not identified:
       → No representative staging environment
       → No automated rollback
       → Manual deployment without a checklist
       → No monitoring detecting problems earlier
Next month: Engineer B makes the same mistake

Blame diverts attention from the questions that truly matter. The right question isn’t “who did this?” but “what made our system allow this to happen?”

Blameless Postmortem Principles #

BASIC ASSUMPTIONS:
  → All engineers act in good faith based on the information
    they had at the time
  → Mistakes are symptoms of systemic problems, not causes
  → The engineers closest to the incident are the best information
    sources — they must feel safe speaking honestly

IMPLICATIONS:
  → Individual names don't appear in negative contexts in postmortem documents
  → Focus on "what happened" not "who did this"
  → Root cause analysis ends at system weaknesses, not individual weaknesses
  → Engineers who made mistakes are actively invited to contribute
    to the postmortem — they hold the most valuable information
Blameless doesn’t mean without accountability. If someone consistently makes poor decisions even when systems supporting good decisions exist, that’s a problem to address — but through performance management processes, not public postmortems. Postmortems focus on systems; individual performance is discussed in a different forum.

The Difference Between Postmortems and RCAs #

These two terms are often used interchangeably, but they actually have important differences.

ROOT CAUSE ANALYSIS (RCA):
  → An analysis technique for finding a problem's root cause
  → Technical and investigative in nature
  → Answers: "Why did this problem happen technically?"
  → Can be done by one or two people
  → Example output: "DB connection pool exhausted due to an N+1 query
    undetected in load testing"

POSTMORTEM:
  → A more comprehensive reflection and documentation process
  → Includes the RCA as one of its parts
  → Answers: "What happened, why, what was the impact, what was learned,
    and what will change?"
  → Involves all affected or involved parties
  → Output: a complete document + action items + systemic changes

The relationship between them:
  An RCA is input for a postmortem, not its replacement.
  A postmortem containing only a technical RCA without process reflection
  and concrete action items is an incomplete postmortem.

When Is a Postmortem Needed? #

Not every problem needs a formal postmortem. General guidance:

MANDATORY POSTMORTEM:
  □ Production downtime affecting users (any duration)
  □ Data loss or data corruption, however small
  □ Security incidents or potential breaches
  □ Significant performance degradation impacting user experience
  □ Deployments that had to be rolled back for causing problems
  □ SLA violations

CONSIDER A POSTMORTEM (judgment call):
  □ Near-misses caught before impacting users
  □ Short incidents with interesting root causes
  □ The same problem appearing a third time or more

NO FORMAL POSTMORTEM NEEDED:
  □ Bugs found and fixed within a sprint without production impact
  □ Minor configuration issues resolved in minutes
  □ Incidents already covered by a similar postmortem with an identical root cause

Leading a Postmortem Session #

Preparation Before the Session #

An effective postmortem starts long before everyone sits in the same room. Good preparation ensures the session itself can focus on analysis and discussion, not fact gathering.

Preparation that must happen before the session (by the facilitator):

1. Collect raw data
   → The event timeline from logs, monitoring, and alerts
   → Dashboard screenshots at incident time
   → The Slack thread or communications during the incident
   → Commands or code executed during investigation and fixing

2. Identify the right participants
   → The on-call engineer when the incident happened (mandatory)
   → Engineers who made the deployment or related changes
   → Engineers who executed the fix
   → Tech lead or manager as context provider (optional, but not as a judge)
   → Representatives from affected teams (e.g. QA, CS if user-facing)

3. Create an initial timeline draft
   → Based on collected data
   → This timeline will be verified and corrected together in the session

4. Send pre-read material
   → A short incident summary (not analysis)
   → The timeline draft
   → Links to relevant logs or monitoring
   → At least 24 hours before the session

Running the Postmortem Session #

Postmortem session structure (60-90 minutes):

OPENING (5 minutes)
  The facilitator states the ground rules:
  → "This is a blameless postmortem — our focus is the system, not individuals"
  → "All perspectives are valued — nothing said here is wrong to share"
  → "Our goal today: understand what happened and make the system better"

TIMELINE REVIEW (20-30 minutes)
  → Walk through the timeline together chronologically
  → Verify and correct with attending participants
  → Mark: when the problem started, when detected, when escalated, when resolved
  → Also mark: decisions made and their rationale at the time

ROOT CAUSE ANALYSIS (20-25 minutes)
  → Use "5 Whys" or Fishbone to dig deeper
  → Look for contributing factors, not just the trigger
  → Identify: what allowed this problem to happen?
  → Avoid stopping at the first answer — keep asking "why"

WHAT WENT WELL (10 minutes)
  → This section is often skipped but important
  → What systems or processes helped detection or mitigation?
  → What decisions were right under pressure?
  → This isn't a praise session — it's identifying things to preserve

ACTION ITEMS (15-20 minutes)
  → Based on the root cause and contributing factors
  → Every action item: clear description, owner, deadline
  → Categorize: immediate fix, short-term improvement, long-term prevention
  → Be realistic — 3 action items done beats 10 ignored

The 5 Whys Technique in Postmortems #

5 Whys is a simple but powerful technique for digging past symptoms to the real root of a problem.

Example of 5 Whys usage in a postmortem:

Incident: Production API down for 45 minutes

Why 1: Why was the API down?
  → Database connection pool exhausted

Why 2: Why was the connection pool exhausted?
  → A surge of very slow queries

Why 3: Why were the queries slow?
  → Queries weren't using indexes — full table scans

Why 4: Why didn't the queries use indexes?
  → The index that should have been used was gone after yesterday's migration

Why 5: Why did the migration remove an index undetected?
  → The migration script wasn't reviewed for performance impact
  → No automated check verifies indexes after migrations
  → The staging database lacks representative data so
    the performance problem wasn't detected during testing

ROOT CAUSE FOUND:
  Not "an engineer wrote a wrong query"
  But: "the migration review process doesn't include performance verification,
  and the staging environment isn't representative for detecting this problem"

THE RIGHT ACTION ITEMS:
  → Add an automated check: verify indexes after every migration
  → Create a staging database with more representative data volume
  → Add a migration review checklist covering performance impact

The Postmortem Document #

The postmortem document is the main output of the postmortem process — a document recording the entire analysis and all decisions, so it can be a future reference accessible to people who didn’t attend the session.

Principles of a Good Postmortem Document #

CLEAR:
  → Understandable by engineers not directly involved in the incident
  → Doesn't assume non-universal knowledge

FACTUAL:
  → Based on verified data and timelines
  → Not speculation or unsupported assumptions
  → Distinguish between "what happened" and "what might have happened"

ACTIONABLE:
  → Every action item concrete, with an owner and deadline
  → Not "be more careful" — but "add an automated test for X"

BLAMELESS:
  → Individual names don't appear in negative contexts
  → If a role must be mentioned, use the role not the name:
    "the on-call engineer" not "Engineer A"

CONCISE BUT COMPLETE:
  → No need to write a novel — just enough to be well understood
  → Detailed timeline, clear root cause, concrete action items

Postmortem Document Template #

Here’s a complete template you can use directly:

# Postmortem: [Short Incident Title]

**Incident Date:** YYYY-MM-DD
**Postmortem Date:** YYYY-MM-DD
**Severity:** Critical / High / Medium
**Incident Duration:** [start time] – [resolved time] ([total duration])
**Status:** Draft | In Review | Final

**Author:** [Facilitator Name]
**Reviewer:** [Tech Lead / Engineering Manager Name]
**Session Participants:** [list of roles, not individual names]

---

## Executive Summary

[2-3 sentences summarizing: what happened, its impact, and what
will be done. Written for readers without time to read the full document.]

Example:

"On 2026-01-27 at 14:32, the payment API experienced 47 minutes of
downtime due to a database connection pool exhaustion after a migration
deployment. Around 3,200 transactions were affected during that period.
We will add automated verification for database indexes and
improve data representation in the staging environment."

---

## Timeline

Use a consistent, readable format:

| Time (WIB) | Event |
|---|---|
| 14:32 | Sprint 42 deployment finished in production |
| 14:38 | First alert appeared: /api/payment error rate rose to 15% |
| 14:41 | On-call engineer received the PagerDuty alert |
| 14:45 | Investigation started: logs show DB connection timeouts |
| 14:52 | Identified: connection pool exhausted due to slow queries |
| 15:01 | Slow query found, traced to a missing index after migration |
| 15:08 | Rollback deployment started |
| 15:14 | Rollback finished, index restored |
| 15:19 | Error rate back to normal, service fully restored |
| 15:30 | All-clear confirmed, monitoring watched for 30 additional minutes |

**Total duration:** 47 minutes (14:32 – 15:19)
**Time to detect:** 6 minutes
**Time to identify root cause:** 26 minutes
**Time to resolve:** 47 minutes

---

## Impact

**Affected users:**
- ~3,200 failed payment transactions during the incident period
- /api/payment endpoint error rate peaked at 87% at 14:45

**Affected systems:**
- payment-service (primary)
- order-service (downstream, orders delayed but not lost)
- notification-service (confirmation emails not sent)

**Data impact:**
- No data loss
- 3,200 failed transactions need retrying (confirmed idempotent)

---

## Root Cause Analysis

### Trigger (Direct Cause)
A migration script in the sprint 42 deployment accidentally
removed the composite index `idx_payments_user_status_created` from
the `payments` table. Without this index, the payment history listing query
full-table-scans a table with 8.4 million rows.

### Contributing Factors

**1. Migration review doesn't cover performance impact**
Migration scripts are reviewed for correctness (is the schema right?)
but there's no checklist or tooling verifying that previously existing
indexes still exist after migration.

**2. Staging environment isn't representative**
The staging database only has ~5,000 rows in the payments table.
Full table scans on such small data don't show significant performance
problems, so they pass testing.

**3. Alert thresholds too loose**
The error rate alert is configured to trigger at 25%.
With this threshold, the alert only appeared 6 minutes after the problem
started — too slow for a problem escalating this fast.

**4. No automated post-deployment verification**
No automated check verifies that production schemas and indexes
match expectations after deployment.

### 5 Whys

Why was the payment API down? → DB connection pool exhausted

Why was the connection pool exhausted? → Queries ran very slowly (full table scans)

Why were queries doing full table scans? → The index those queries used didn’t exist

Why didn’t the index exist? → A migration script accidentally removed it

Why did the index-removing migration go undetected? → Migration review doesn’t check impact on existing indexes → Staging lacks enough data to detect performance problems → No automated post-deployment schema verification check


---

## What Went Well

- **Relatively fast detection:** The alert appeared within 6 minutes, the on-call engineer
  responded within 3 minutes of the alert
- **Effective stakeholder communication:** The status page was updated within 10 minutes
  of the incident being identified, the CS team was notified before many tickets came in
- **Idempotency prevented duplication:** All failed transactions can
  be retried without duplication risk because the payment flow is already idempotent
- **Smooth rollback:** The rollback process finished in 6 minutes,
  documented and rehearsed

---

## Action Items

### Immediate (within 1 week)

| # | Action | Owner | Deadline |
|---|---|---|---|
| 1 | Add a migration checklist that must include index verification | Tech Lead | 2026-02-03 |
| 2 | Lower the error rate alert threshold from 25% to 5% for the payment endpoint | On-call Engineer | 2026-02-03 |
| 3 | Create a ticket to retry the 3,200 failed transactions (verified idempotent) | Backend Engineer | 2026-01-29 |

### Short-term (within 1 month)

| # | Action | Owner | Deadline |
|---|---|---|---|
| 4 | Create an automated test: verify expected indexes exist after migrations | Backend Engineer | 2026-02-28 |
| 5 | Set up staging data seeding for at least 500K rows in critical tables | DevOps Engineer | 2026-02-28 |
| 6 | Review all existing migration scripts — any with similar potential problems? | Tech Lead | 2026-02-20 |

### Long-term (within 1 quarter)

| # | Action | Owner | Deadline |
|---|---|---|---|
| 7 | Implement a post-deployment automated smoke test suite covering critical query performance | Backend Engineer | 2026-03-31 |
| 8 | Evaluate tooling for schema drift detection in production | Tech Lead | 2026-03-31 |

---

## Lessons Learned

[This section contains insights applicable more broadly,
not just to this incident.]

1. **Migration review must cover performance impact, not just correctness**
   Every migration that changes, removes, or adds indexes
   must go through explicit performance review.

2. **An unrepresentative staging environment gives false confidence**
   Data volume in staging must be enough to represent production
   query patterns — especially for tables with millions of rows.

3. **Alert thresholds must match endpoint criticality**
   The payment endpoint needs far stricter thresholds
   than other non-critical endpoints.

---

## Appendix

**Link to the monitoring dashboard at incident time:**
[Grafana snapshot URL]

**Link to the incident thread in Slack:**
[thread URL]

**Link to the related deployment:**
[GitHub Actions run URL]

**Relevant error logs:**

[2026-01-27 14:38:12] ERROR: HikariPool timeout after 30000ms [2026-01-27 14:38:12] ERROR: Connection pool exhausted (max: 50, active: 50) [2026-01-27 14:40:55] WARN: Query execution time: 28.4s (threshold: 1s) [2026-01-27 14:40:55] WARN: Full table scan detected on payments (8.4M rows)


How to Write Each Section Effectively #

Executive Summary #

Written last, after the entire analysis is done. It must be understandable by someone uninvolved in the incident and without deep technical context. Three things are mandatory: what happened, its impact, and what will be done.

// ✗ An executive summary too technical:

"DB connection pool exhausted due to a full table scan on the payments
table caused by a post-migration missing index which led to HikariCP
timeouts and cascade failures to downstream services."

// ✓ An executive summary everyone can understand:

"On Tuesday, January 27, the payment service experienced a 47-minute
disruption that caused about 3,200 failed transactions. The cause was
a database change that accidentally removed a setting that helped
queries run fast. All failed transactions are confirmed retryable
without duplication, and we're adding automated checks to prevent
a similar occurrence."

Timeline #

The timeline must be accurate and data-based, not reconstructed from memory. Best sources: server logs, alert history, and Slack threads.

Things to record in the timeline:
  ✓ When the problem actually started (not when detected)
  ✓ When the first alert appeared
  ✓ When the engineer started responding
  ✓ Hypotheses tried and why they failed
  ✓ When the root cause was identified
  ✓ When the fix started and finished
  ✓ When the service was fully restored and confirmed

Important notes:
  → Distinguish between "server time" and "local time" if the team is distributed
  → Use a consistent timezone across the whole timeline
  → If there's a gap in the timeline, note: "investigation ongoing, no
    significant updates during this period"

Root Cause Analysis #

The most substantial part of the postmortem document. Most often written too shallowly (“human error”) or too technically without context.

A good RCA structure:

1. Trigger — what directly set off the problem?
   Not the root cause, but the analysis starting point
   Example: "The sprint 42 deployment removed a database index"

2. Root cause — what allowed the trigger to happen?
   This is the answer to the last 5 Whys
   Example: "There's no mechanism to detect missing indexes
   after migrations, neither in review nor in the deployment pipeline"

3. Contributing factors — what conditions worsened the situation?
   These are factors that, if fixed, would make the system
   more resilient even if the same trigger recurs
   Example: "Unrepresentative staging", "Alert thresholds too loose"

ANTI-PATTERNS in writing RCAs:
  ✗ "Human error" — this isn't a root cause, it's a symptom
  ✗ "The engineer wasn't careful" — this is blame, not analysis
  ✗ "The system failed" — too general, not actionable
  ✗ Stopping at the 2nd or 3rd Why before reaching the real root

Action Items #

This is the section determining whether a postmortem has real impact or just becomes an archive document.

Characteristics of effective action items:

SPECIFIC:
  ✗ "Fix the migration review process"
  ✓ "Add a migration review checklist covering: (1) verifying
     all pre-migration indexes still exist after migration,
     (2) query plan comparison for critical queries"

MEASURABLE:
  ✗ "Improve monitoring"
  ✓ "Lower the /api/payment endpoint error rate alert threshold from 25% to 5%"

HAS AN OWNER:
  ✗ "The team needs to finish this"
  ✓ "[Role] is responsible for completing this"

HAS A DEADLINE:
  ✗ "Finish it soon"
  ✓ "Done before 2026-02-03"

REALISTIC:
  3 action items actually done is better
  than 10 action items ignored because there are too many

How to Distribute and Follow Up Postmortems #

A finished postmortem document that’s never read or followed up is a failed postmortem. Two things need to happen after the final document:

Distribution #

Postmortem document distribution:

Internal engineering team:
  → Send the link to the main engineering channel with a short summary
  → Not for punishment — for learning together

Affected stakeholders:
  → A non-technical summary (executive summary only)
  → What happened, its impact on them, and what has/will be done

Delivery format:
  "Hi all, the postmortem for the [name] incident is final.
   Summary: [2-3 sentences]
   Full doc: [link]
   Main action items being worked on: [short list]"

Following Up Action Items #

An effective follow-up mechanism:

1. All action items go into the backlog as official tickets
   → Not just recorded in the postmortem document
   → With a clear sprint assignment or priority

2. Review progress at the next sprint
   → The tech lead or Scrum Master checks: "Action items from postmortem
     X — any updates?"

3. Update the postmortem document as action items complete
   → Mark every action item with a status: ✓ Done, ⟳ In Progress, ✗ Blocked
   → The document becomes a "living document" until all action items finish

4. Verify effectiveness
   → After action items finish, is the system truly more resilient?
   → If possible, do chaos engineering or drills for verification
Store all postmortem documents in one well-indexed place — for example a “Postmortems” folder in Notion or Confluence, with a YYYY-MM-DD — [Incident Name] naming format. When a new incident occurs, first check whether an old postmortem has a similar root cause. Often the same pattern reappears in a different form, and old postmortems can provide valuable insight.

Postmortem Anti-Patterns to Avoid #

Postmortems Containing Veiled Blame #

The most dangerous blame isn’t explicit — it’s veiled in “neutral” language:

// ✗ Veiled blame:

"The engineer responsible for the deployment didn't verify the migration
 script carefully enough before running it in production"
→ This is still blame — implying this was an individual failure


"The QA team failed to detect this problem during testing"
→ This blames the QA team — ignoring that the testing system itself needs fixing

// ✓ Blameless framing:

"The migration script has no automated verification
 ensuring previously existing indexes survive the migration"


"Staging testing didn't detect the performance problem because
 the staging data volume isn't representative of production"
→ Focus on system weaknesses, not individuals

Unrealistic or Unfollowed Action Items #

A postmortem producing a long action item list where nothing gets done is worse than no postmortem at all — it creates a cycle where the team feels they’ve “handled” the problem when nothing has changed.

// ✗ Action items never done:
Sprint 10 postmortem: 12 action items
Sprint 15 postmortem: 8 new action items (plus 9 from Sprint 10 unfinished)
Sprint 20 postmortem: "We already have many unfinished action items..."
→ The team loses trust in the postmortem process

// ✓ A more realistic approach:
Identify the 3 most impactful action items
Commit to a realistic timeline based on team capacity
Review progress every sprint until all are done
→ 3 completed action items beat 12 ignored ones

Postmortems Too Long After the Incident #

Postmortems done too long after an incident lose timeline accuracy, important technical details, and the momentum for change.

Recommended timing:
  Postmortem session: within 24-72 hours of the incident being resolved
  Document draft: within 48 hours of the session
  Final document: within 1 week of the incident

Why timing matters:
  → Memory of technical details fades fast
  → The urgency to fix the system is still felt
  → Involved engineers still have fresh context
  → Action items are easier to prioritize while the incident is still relevant

Postmortems Never Done Because “There’s No Time” #

This is the most common anti-pattern — and the most damaging long-term.

// ✗ Common excuses:

"Let's skip the postmortem this time, the sprint is packed"

"The incident is resolved, no need to discuss it further"

"We already know the root cause, no need to be formal"

// Long-term impact:
Sprint 5: Incident A — no postmortem
Sprint 8: Incident B with a similar root cause
Sprint 12: Incident C with the same root cause again
→ The team repeats the same mistakes because there's no systematic learning

// ✓ The right approach:
A postmortem for every significant incident is non-negotiable
A 2-3 hour postmortem investment can prevent the next incident
which might take days to resolve

The Postmortem as Part of the Documentation Ecosystem #

Postmortems are most effective when connected to broader documentation practices:

THE POSTMORTEM'S RELATIONSHIP WITH OTHER DOCUMENTS:

RFC / Design Doc:
  → A postmortem finding architecture weaknesses → an RFC for the fix
  → RFCs reviewed post-incident can provide "why this decision
    was made" context relevant to the postmortem

Release Document:
  → A deployment causing an incident → link from the release doc to the postmortem
  → Postmortem action items can appear as items in the next sprint's release doc

Reference Document:
  → Patterns or standards emerging from postmortems → documented in reference docs
  → Example: "After postmortem X, we agreed all migrations must
    list the indexes existing before and after the migration"

Knowledge Sharing Session:
  → Every significant postmortem → a post-mortem KSS within 1-2 weeks
  → KSS provides space for broader discussion of incident learnings

On-call Runbook:
  → If the incident produced a new playbook for handling similar problems,
    the runbook is updated and linked from the postmortem

Postmortem Checklist #

AFTER THE INCIDENT IS RESOLVED:
  □ Determine whether this incident needs a formal postmortem
  □ Assign a facilitator (usually a Tech Lead or senior engineer not
    directly involved in the incident)
  □ Schedule the session within 24-72 hours
  □ Identify the participants who need to attend

PREPARATION BEFORE THE SESSION:
  □ Timeline collected from logs, monitoring, and alerts
  □ Relevant dashboard screenshots and logs saved
  □ Timeline draft prepared and sent as pre-read
  □ Blameless ground rules communicated before the session

DURING THE SESSION:
  □ Facilitator states the blameless ground rules at the start
  □ Timeline reviewed and verified together
  □ 5 Whys or another RCA technique used to dig for the root cause
  □ Contributing factors — not just the trigger — identified
  □ "What went well" is not skipped
  □ Concrete action items agreed: owner, deadline, category

POSTMORTEM DOCUMENT:
  □ Executive summary understandable by non-engineers
  □ Timeline accurate and data-based, not from memory
  □ Root cause names the system, not individuals
  □ More than one contributing factor (there almost always are)
  □ Action items specific, with owners and deadlines
  □ Lessons written as insights applicable more broadly
  □ Document reviewed by an uninvolved person to ensure clarity

AFTER THE FINAL DOCUMENT:
  □ Document shared with the entire engineering team
  □ Non-technical summary sent to affected stakeholders
  □ All action items enter the backlog as official tickets
  □ Document stored somewhere easy to find (format: YYYY-MM-DD — Name)
  □ Action item progress reviewed at the next sprint
  □ Post-mortem KSS scheduled if the incident is significant enough

Summary #

  • A postmortem isn’t about who’s at fault — it’s about the system that needs fixing — blameless doesn’t mean without accountability, but that the focus is on processes and tooling, not punishing individuals.
  • An RCA is input for a postmortem, not its replacement — a postmortem containing only technical analysis without process reflection and concrete action items is incomplete.
  • 5 Whys helps find the true root cause — don’t stop at the first answer. “Human error” isn’t a root cause — it’s a symptom of a system failing to support the humans in it.
  • Timelines must be data-based, not memory-based — logs, monitoring, and alert history are the source of truth. Memory reconstruction is inaccurate and unreliable.
  • “What went well” matters as much as “what failed” — identifying successes helps preserve and strengthen effective practices, not just fix what’s broken.
  • Unfinished action items are worse than no action items — 3 completed action items beat 10 ignored ones. Prioritization and commitment matter more than list completeness.
  • Postmortem timing is critical — do it within 24-72 hours of the incident being resolved, while details are still fresh and the momentum for change exists.
  • A postmortem document is a living document — update action item statuses periodically until everything finishes. A document that stops updating loses its accountability value.
  • Store all postmortems in one indexed place — recurring patterns across different postmortems are a strong signal of deeper systemic weaknesses.
  • Effective postmortems connect to the wider documentation ecosystem — RFCs for architecture fixes, reference documents for new standards, and KSS to spread learnings across the team.

← Previous: Knowledge Sharing   Next: Gitflow →

About | Author | Content Scope | Editorial Policy | Privacy Policy | Disclaimer | Contact