OOM Killer #

One night, the Node.js process serving the production API suddenly stops. No exception, no graceful shutdown, no farewell log. The server restarts, active connections drop, and several in-flight transactions vanish. Hidden among thousands of log lines is a single line in the system log: Out of memory: Kill process 12345 (node) score 723 or sacrifice child.

The OOM Killer — the Out-of-Memory Killer — is a Linux kernel mechanism that activates when the system runs out of memory and can’t allocate more. Instead of letting the entire system freeze or crash, the kernel selects one or more processes to kill so memory can be freed. The selected process gets no chance to clean up — it’s immediately terminated with SIGKILL.

For engineers who don’t understand it, an OOM kill feels like a mysterious bug or unexplainable instability. For those who understand it, an OOM kill is a clear signal: there’s a memory management problem in that system that needs solving.

How the OOM Killer Works #

When a process requests memory and the kernel can’t fulfill it, the kernel enters OOM handling. Before killing a process, the kernel tries several steps:

flowchart TD
    A[Process requests memory allocation] --> B{Memory available?}
    B --> |Yes| C[Allocation succeeds]
    B --> |No| D[Try to free the page cache]
    D --> E{Enough?}
    E --> |Yes| C
    E --> |No| F[Try swapping to disk]
    F --> G{Swap available?}
    G --> |Yes| H[Swap and allocate]
    G --> |No or swap full| I[OOM condition]
    I --> J[Calculate oom_score for all processes]
    J --> K[Select the process with the highest score]
    K --> L[Send SIGKILL to the selected process]
    L --> M[Memory freed]
    M --> N[Try allocating again]
    N --> B

    style I fill:#ffcccc
    style L fill:#ffcccc

The OOM Killer selects processes based on the oom_score — a value between 0 and 1000 calculated by the kernel. The higher the score, the more likely that process is to be killed.

# View a process's current oom_score
cat /proc/<PID>/oom_score

# View all processes with their oom_scores
for pid in /proc/[0-9]*; do
    pid_num=$(basename $pid)
    score=$(cat $pid/oom_score 2>/dev/null)
    comm=$(cat $pid/comm 2>/dev/null)
    echo "$score $pid_num $comm"
done | sort -rn | head -20

# Example output:
# 850  12345  node
# 720  12346  python3
# 400  12347  java
# 50   1      systemd
# 0    2      kthreadd  (kernel thread — never killed)

Understanding oom_score and oom_score_adj #

Factors affecting oom_score:

  Main factor:
  → The memory size used by the process
    (including memory shared with other processes)
  → More memory used → higher score → higher kill priority

  Modifying factor:
  → oom_score_adj: a -1000 to +1000 value settable by administrators
    - Negative values: reduce the likelihood of this process being killed
    - Positive values: increase the likelihood of this process being killed
    - A value of -1000: the process will never be killed by the OOM Killer

  Simple formula:
  oom_score = base_score + oom_score_adj
  where base_score is calculated from memory usage relative to total system memory

# Viewing and changing oom_score_adj

# View a process's oom_score_adj
cat /proc/<PID>/oom_score_adj

# Set oom_score_adj for a running process
# (needs root or the CAP_SYS_RESOURCE capability)
echo -500 > /proc/<PID>/oom_score_adj
# This process is now less likely to be killed

# Set oom_score_adj to the minimum value (the process won't be killed)
echo -1000 > /proc/<PID>/oom_score_adj

# For processes that must NEVER be killed (databases, monitoring agents):
echo -1000 > /proc/$(pgrep postgres)/oom_score_adj
echo -1000 > /proc/$(pgrep mysqld)/oom_score_adj

# Set oom_score_adj via systemd service units:
# [Service]
# OOMScoreAdjust=-1000
# This is more reliable because it's applied at service start

// Go: setting oom_score_adj from within an app at startup
// Reduce the likelihood of this process being killed by the OOM Killer.
// Must be called with sufficient privileges.
func protectFromOOM() {
    pid := os.Getpid()
    oomAdjPath := fmt.Sprintf("/proc/%d/oom_score_adj", pid)

    if err := os.WriteFile(oomAdjPath, []byte("-500"), 0644); err != nil {
        // significantly reduce kill likelihood
        if errors.Is(err, os.ErrPermission) {
            log.Println("Warning: Could not set OOM protection (needs privileges)")
        } else {
            log.Printf("Warning: OOM protection failed: %v", err)
        }
        return
    }

    data, err := os.ReadFile(fmt.Sprintf("/proc/%d/oom_score", pid))
    if err == nil {
        fmt.Printf("OOM protection set. Current oom_score: %s", strings.TrimSpace(string(data)))
    }
}

// Call at application startup
func init() {
    protectFromOOM()
}

Detecting and Investigating OOM Events #

OOM events always leave traces in the kernel log. Understanding how to read them is essential for investigations.

# Search for OOM events in the kernel log
dmesg | grep -i "out of memory\|oom_kill\|kill process\|killed process"

# Or with journalctl (systemd)
journalctl -k | grep -i "out of memory\|oom"

# A typical OOM event output:
# [1234567.890] Out of memory: Kill process 12345 (node) score 850 or sacrifice child
# [1234567.891] Killed process 12345 (node) total-vm:2097152kB, anon-rss:1048576kB, file-rss:4096kB
# [1234567.892] oom_reaper: reaped process 12345 (node), now anon-rss:0kB, file-rss:0kB

# Parsing the information from OOM logs:
# - Process: node (PID 12345)
# - Score: 850 (high = genuinely using lots of memory)
# - total-vm: virtual memory = 2GB
# - anon-rss: resident set size (actual physical memory) = 1GB
# - file-rss: file-mapped memory = 4MB

# A script to monitor OOM events in real-time
#!/bin/bash

echo "Monitoring OOM events (Ctrl+C to stop)..."
dmesg -w | grep --line-buffered -i "out of memory\|oom_kill\|killed process" | while read line; do
    timestamp=$(date '+%Y-%m-%d %H:%M:%S')
    echo "[$timestamp] OOM EVENT: $line"

    # Send an alert (example: via curl to a webhook)
    # curl -s -X POST "https://hooks.slack.com/..." \
    #     -H "Content-Type: application/json" \
    #     -d "{\"text\": \"OOM Event: $line\"}"
done

# Investigating after an OOM event — find the cause

# 1. When did the OOM happen?
journalctl -k --since "1 hour ago" | grep -i oom

# 2. Which process was killed?
dmesg | grep "Killed process" | tail -20

# 3. How much memory was available at the OOM moment?
# This is stored in the kernel log during the OOM
dmesg | grep -A 30 "Out of memory" | grep "Normal\|free\|active\|inactive"
# The output contains a memory state snapshot from the OOM moment

# 4. Is there a memory leak? (check history before the OOM)
# Use monitoring tools to see the memory usage trend
# Memory leaks usually appear as continuously rising graphs

# 5. Were there other large processes?
dmesg | grep "oom_kill_process\|oom_score"

vm.overcommit Configuration: Controlling Memory Allocation Behavior #

Linux allows overcommit by default — processes can request more memory than is available, assuming not all memory will be genuinely used at once. This is a useful optimization but can cause unexpected OOMs.

# View the current overcommit setting
cat /proc/sys/vm/overcommit_memory

# Possible values:
# 0 = Heuristic overcommit (default)
#     The kernel allows overcommit but tries to limit excessive requests
#     Most commonly used
#
# 1 = Always overcommit
#     All memory allocations allowed without limits
#     malloc() never returns NULL
#     But OOM kills can happen anytime and can't be predicted
#     NOT RECOMMENDED for production
#
# 2 = Never overcommit (strict mode)
#     Total committed memory must not exceed swap + RAM * overcommit_ratio
#     Safer, but malloc() can return NULL
#     Applications must be ready to handle allocation failures

# View the overcommit_ratio (default: 50%)
cat /proc/sys/vm/overcommit_ratio
# Max committed memory = swap + (RAM * overcommit_ratio / 100)

# View how much memory has been committed
cat /proc/meminfo | grep CommitLimit
cat /proc/meminfo | grep Committed_AS

# Commit_AS > CommitLimit = an OOM condition can happen at any time

# Changing overcommit behavior (needs root)

# For databases like PostgreSQL — strict mode is safer
# sysctl -w vm.overcommit_memory=2
# sysctl -w vm.overcommit_ratio=80  # 80% RAM + swap for committed memory

# For general applications — keep the default (0)
# or add to /etc/sysctl.conf for persistence:
# vm.overcommit_memory = 0
# vm.swappiness = 10  # Reduce swap aggressiveness (0-100)

# vm.swappiness: how aggressively the kernel uses swap
# 10 = only swap when truly needed (recommended for production servers)
# 60 = default
# 0  = don't swap unless forced
echo 10 > /proc/sys/vm/swappiness

Protecting Critical Processes #

Not all processes are equally important. Databases, monitoring agents, and core system processes should not be killed by the OOM Killer.

# Protection via systemd (the best way — persistent and automatic)

# Create or edit the service file
cat > /etc/systemd/system/myapp.service << 'EOF'
[Unit]
Description=My Critical Application
After=network.target

[Service]
ExecStart=/usr/bin/myapp
Restart=always
RestartSec=5s

# OOM protection — values from -1000 to +1000
# -1000 = never killed by OOM
# -500  = very unlikely to be killed
OOMScoreAdjust=-500

# Memory limits (optional) — prevent one process from monopolizing memory
# If exceeded, the process is killed BEFORE causing a system-wide OOM
MemoryMax=2G
MemoryHigh=1.5G  # Soft limit — start throttling here

[Install]
WantedBy=multi-user.target
EOF

systemctl daemon-reload
systemctl enable myapp
systemctl start myapp

# Protection for already-running databases

# PostgreSQL
PG_PID=$(pgrep -x postgres | head -1)
echo -900 > /proc/$PG_PID/oom_score_adj
echo "PostgreSQL OOM score adj set to: $(cat /proc/$PG_PID/oom_score_adj)"

# MySQL / MariaDB
MYSQL_PID=$(pgrep -x mysqld)
echo -900 > /proc/$MYSQL_PID/oom_score_adj

# Redis
REDIS_PID=$(pgrep -x redis-server)
echo -900 > /proc/$REDIS_PID/oom_score_adj

# A script to set OOM protection for all critical services after boot
cat > /usr/local/bin/set-oom-protection.sh << 'EOF'
#!/bin/bash
# Set OOM protection for critical processes

declare -A CRITICAL_PROCESSES=(
    ["postgres"]=-900
    ["mysqld"]=-900
    ["redis-server"]=-900
    ["prometheus"]=-500
    ["node_exporter"]=-500
)

for process_name in "${!CRITICAL_PROCESSES[@]}"; do
    adj_value="${CRITICAL_PROCESSES[$process_name]}"
    pids=$(pgrep -x "$process_name" 2>/dev/null)
    for pid in $pids; do
        echo "$adj_value" > "/proc/$pid/oom_score_adj" 2>/dev/null
        echo "Set OOM adj=$adj_value for $process_name (PID: $pid)"
    done
done
EOF
chmod +x /usr/local/bin/set-oom-protection.sh

Per-Process Memory Limit Configuration #

A more proactive strategy than relying on the OOM Killer is giving every process a memory limit — so if one process leaks memory, it kills itself before causing a system-wide OOM.

# Using cgroups v2 (via systemd) to limit memory

# Example: limit a Node.js app to a maximum of 512MB
systemctl edit nodejs-app.service
# Add under [Service]:
# MemoryMax=512M
# MemoryHigh=400M  # Start throttling at 400MB, kill at 512MB

# Verify:
systemctl show nodejs-app.service | grep Memory

# Using ulimit for the current session
ulimit -v 524288  # 512MB in kilobytes (virtual memory limit)
# After this, programs in this session can't allocate more than 512MB

# Using Docker for memory isolation
docker run \
    --memory="512m" \               # Hard limit
    --memory-reservation="256m" \   # Soft limit (warns but doesn't kill)
    --oom-kill-disable=false \       # Allow OOM kills (default)
    myapp
# If the container exceeds --memory, Docker sends SIGKILL

# Don't use --oom-kill-disable=true unless very confident
# because this can cause host system OOMs

Detecting Memory Leaks #

Recurring OOM kills almost always indicate a memory leak in one of the processes.

// Go: monitoring memory usage trends for memory leak detection
type memorySample struct {
    timestamp time.Time
    rssMB     float64
}

// rssMB reads the Resident Set Size of a process from /proc
func rssMB(pid int) (float64, error) {
    data, err := os.ReadFile(fmt.Sprintf("/proc/%d/status", pid))
    if err != nil {
        return 0, err
    }
    for _, line := range strings.Split(string(data), "\n") {
        if strings.HasPrefix(line, "VmRSS:") {
            fields := strings.Fields(line)
            kb, _ := strconv.ParseFloat(fields[1], 64)
            return kb / 1024, nil // Resident Set Size in MB
        }
    }
    return 0, fmt.Errorf("VmRSS not found")
}

type memoryLeakDetector struct {
    pid                int
    windowSize         int
    growthThresholdPct float64
    samples            []memorySample
}

func newMemoryLeakDetector(pid int) *memoryLeakDetector {
    return &memoryLeakDetector{
        pid:                pid,
        windowSize:         30,  // number of samples for the trend
        growthThresholdPct: 5.0, // % growth per hour considered suspicious
    }
}

// collectSample takes a current memory usage sample
func (d *memoryLeakDetector) collectSample() *memorySample {
    rss, err := rssMB(d.pid)
    if err != nil {
        return nil // process no longer exists
    }
    s := memorySample{timestamp: time.Now(), rssMB: rss}
    d.samples = append(d.samples, s)
    if len(d.samples) > d.windowSize {
        d.samples = d.samples[len(d.samples)-d.windowSize:]
    }
    return &s
}

// analyzeTrend analyzes whether there's a suspicious growth trend
func (d *memoryLeakDetector) analyzeTrend() map[string]any {
    if len(d.samples) < 5 {
        return map[string]any{"status": "insufficient_data"}
    }

    oldest := d.samples[0]
    newest := d.samples[len(d.samples)-1]

    timeDiffHours := newest.timestamp.Sub(oldest.timestamp).Hours()
    if timeDiffHours < 0.01 {
        return map[string]any{"status": "insufficient_time"}
    }

    rssGrowthMB := newest.rssMB - oldest.rssMB
    growthPctPerHour := (rssGrowthMB / oldest.rssMB) * 100 / timeDiffHours

    isLeaking := growthPctPerHour > d.growthThresholdPct

    result := map[string]any{
        "pid":                 d.pid,
        "current_rss_mb":      math.Round(newest.rssMB*10) / 10,
        "oldest_rss_mb":       math.Round(oldest.rssMB*10) / 10,
        "growth_mb":           math.Round(rssGrowthMB*10) / 10,
        "growth_pct_per_hour": math.Round(growthPctPerHour*100) / 100,
        "time_window_hours":   math.Round(timeDiffHours*100) / 100,
        "is_likely_leaking":   isLeaking,
        "status":              "warning",
    }
    if !isLeaking {
        result["status"] = "ok"
    }

    if isLeaking {
        log.Printf("Possible memory leak detected: %+v", result)
    }

    return result
}

// Usage:
// detector := newMemoryLeakDetector(os.Getpid())
// for {
//     detector.collectSample()
//     analysis := detector.analyzeTrend()
//     if analysis["is_likely_leaking"] == true {
//         alertTeam(analysis)
//     }
//     time.Sleep(60 * time.Second)
// }

// Go: memory profiling to find leaks
// runtime/pprof is the stdlib equivalent of tracemalloc

// Start tracking memory allocations
func startMemoryTracing() {
    // pprof profiling is enabled via an HTTP endpoint or runtime hooks
}

// Get the code locations allocating the most memory
func getTopMemoryAllocations() []map[string]any {
    // Equivalent: read runtime.MemStats for heap usage snapshots
    var m runtime.MemStats
    runtime.ReadMemStats(&m)

    return []map[string]any{
        {
            "metric":     "heap_alloc_bytes",
            "value":      m.HeapAlloc,
            "note":       "bytes allocated on the heap",
        },
        {
            "metric":     "total_alloc_bytes",
            "value":      m.TotalAlloc,
            "note":       "cumulative allocation counter",
        },
    }
}

// Compare two snapshots to see growth
func compareSnapshots() {
    // pprof: download two profiles and diff them with
    // go tool pprof -base profile1.out profile2.out
}

Handling OOM Kills in Applications #

Because OOM kills use SIGKILL, which can’t be caught, there’s no way to clean up when an OOM kill happens. But several things can be done:

// Go: handling OOM kills in applications
// 1. Graceful shutdown when receiving catchable signals
// (OOM kill sometimes sends SIGTERM first to some systems — not always)
func handleSigterm() {
    log.Println("Received SIGTERM — cleaning up...")

    // Save important state
    saveCheckpoint()

    // Close database connections
    db.Close()

    // Wait for in-flight requests to finish
    // (with a timeout)
    server.Shutdown(context.Background())
}

func main() {
    sigCh := make(chan os.Signal, 1)
    signal.Notify(sigCh, syscall.SIGTERM)
    go func() {
        for range sigCh {
            handleSigterm()
        }
    }()
}

// 2. Regular checkpoints — don't keep important state only in memory
type StatefulProcessor struct {
    checkpointFile string
    state          map[string]any
}

func NewStatefulProcessor(checkpointFile string) *StatefulProcessor {
    p := &StatefulProcessor{checkpointFile: checkpointFile}
    p.state = p.loadCheckpoint()
    return p
}

func (p *StatefulProcessor) loadCheckpoint() map[string]any {
    data, err := os.ReadFile(p.checkpointFile)
    if err != nil {
        return map[string]any{"processed": 0, "last_id": nil}
    }
    var state map[string]any
    if err := gob.NewDecoder(bytes.NewReader(data)).Decode(&state); err != nil {
        return map[string]any{"processed": 0, "last_id": nil}
    }
    return state
}

func (p *StatefulProcessor) saveCheckpoint() {
    // Regularly save state to disk
    var buf bytes.Buffer
    gob.NewEncoder(&buf).Encode(p.state)
    // Atomic writes: no corrupt intermediate states
    tmp := p.checkpointFile + ".tmp"
    os.WriteFile(tmp, buf.Bytes(), 0644)
    os.Rename(tmp, p.checkpointFile)
}

func (p *StatefulProcessor) processBatch(items []Item) {
    for _, item := range items {
        p.processItem(item)
        p.state["processed"] = p.state["processed"].(int) + 1
        p.state["last_id"] = item.ID

        // Checkpoint every 1000 items
        if p.state["processed"].(int)%1000 == 0 {
            p.saveCheckpoint()
        }
    }
}

Monitoring and Alerting for OOM Events #

# Setting up alerts with systemd and monitoring scripts

# OOM event monitoring script
cat > /usr/local/bin/oom-monitor.sh << 'SCRIPT'
#!/bin/bash

LOG_FILE="/var/log/oom-events.log"
WEBHOOK_URL="${OOM_ALERT_WEBHOOK}"

# Monitor the kernel log for OOM events
journalctl -kf | grep --line-buffered "Out of memory\|Killed process\|oom_kill" | \
while read -r line; do
    timestamp=$(date '+%Y-%m-%d %H:%M:%S')
    echo "[$timestamp] $line" >> "$LOG_FILE"

    # Send an alert
    if [ -n "$WEBHOOK_URL" ]; then
        curl -s -X POST "$WEBHOOK_URL" \
            -H "Content-Type: application/json" \
            -d "{
                \"event\": \"oom_kill\",
                \"host\": \"$(hostname)\",
                \"message\": \"$line\",
                \"timestamp\": \"$timestamp\"
            }" > /dev/null
    fi

    # Log the memory state at the OOM moment
    free -h >> "$LOG_FILE"
    echo "---" >> "$LOG_FILE"
done
SCRIPT

chmod +x /usr/local/bin/oom-monitor.sh

// Go: Prometheus metrics for OOM monitoring
var (
    oomEventsTotal = prometheus.NewCounter(
        prometheus.CounterOpts{
            Name: "oom_kill_events_total",
            Help: "Total number of OOM kill events",
        },
    )

    memoryAvailableBytes = prometheus.NewGauge(
        prometheus.GaugeOpts{
            Name: "memory_available_bytes",
            Help: "Available memory in bytes",
        },
    )
)

// parseOOMEvents parses OOM events from the kernel log
func parseOOMEvents() int {
    out, err := exec.Command("journalctl", "-k", "--since", "1 minute ago", "--no-pager").Output()
    if err != nil {
        return 0
    }
    count := strings.Count(strings.ToLower(string(out)), "out of memory")
    if count > 0 {
        oomEventsTotal.Add(float64(count))
    }
    return count
}

// updateMemoryMetrics updates memory metrics for Prometheus
func updateMemoryMetrics() {
    // Read available memory from /proc/meminfo (MemAvailable)
    data, err := os.ReadFile("/proc/meminfo")
    if err != nil {
        return
    }
    for _, line := range strings.Split(string(data), "\n") {
        if strings.HasPrefix(line, "MemAvailable:") {
            fields := strings.Fields(line)
            kb, _ := strconv.ParseFloat(fields[1], 64)
            memoryAvailableBytes.Set(kb * 1024)
            return
        }
    }
}

Anti-Patterns to Avoid #

# ✗ Anti-pattern 1: ignoring OOM events
# "The server restarted, everything is back to normal"
# OOM kills without investigation mean the problem will recur
# ✓ Every OOM event must be investigated: why did it happen, which process was killed,
#   is there a memory leak, does capacity planning need revision

# ✗ Anti-pattern 2: oom_kill_disable=true in Docker
# docker run --oom-kill-disable=true myapp
# If the container leaks memory, it can make the HOST system OOM
# ✓ Use --memory limits and let containers get OOM-killed, not hosts

# ✗ Anti-pattern 3: vm.overcommit_memory=1 (always overcommit)
# malloc() never fails but OOMs can happen anytime unpredictably
# ✓ Use the default (0) or strict mode (2) for database servers

# ✗ Anti-pattern 4: no per-process/container memory limits
# One leaking process can cause the entire system to OOM
# ✓ Set MemoryMax in systemd or --memory in Docker for isolation

# ✗ Anti-pattern 5: no memory trend monitoring
# Memory leaks only detected after OOMs happen
# ✓ Monitor per-process memory usage trends, alert on suspicious growth

# ✗ Anti-pattern 6: changing all processes' oom_score_adj to -1000
# If all processes are "unkillable", OOMs cause kernel panics
# ✓ Only genuinely critical processes need full protection
#   Let the OOM Killer have other processes to choose from

OOM Killer Management Checklist #

DETECTION & MONITORING:
  □ Alerts installed for OOM events (journalctl or dmesg monitoring)
  □ Per-process memory usage trends monitored (not just totals)
  □ Alerts when available memory < 15% (before OOMs happen)
  □ Alerts when swap starts being used (early memory pressure signs)
  □ OOM events logged with details (which process, how much memory)

CRITICAL PROCESS CONFIGURATION:
  □ Databases (PostgreSQL, MySQL, Redis) have low oom_score_adj values
  □ Monitoring agents can't be OOM-killed (needed for crisis alerts)
  □ OOM protection configured via systemd (OOMScoreAdjust) — not manually
  □ Restartable applications have higher OOMScoreAdjust values

MEMORY LIMITS:
  □ Every Docker container has a --memory limit
  □ Every systemd service has a MemoryMax when relevant
  □ Memory limits set based on profiling, not arbitrary numbers

OVERCOMMIT:
  □ vm.overcommit_memory configured per needs (not blindly default)
  □ vm.swappiness set low for servers (10 is a good starting point)
  □ CommitLimit > Committed_AS monitored (indicates OOM proximity)

INVESTIGATION:
  □ Every OOM event investigated — never ignored
  □ OOM investigation runbooks exist (steps to follow)
  □ Memory leaks detected and fixed, not just services restarted

CAPACITY PLANNING:
  □ Total memory usage of all processes measured at peak load
  □ Memory usage growth rates monitored
  □ 30% headroom maintained to prevent OOMs during spikes

Summary #

The OOM Killer is an emergency mechanism, not a feature — if the OOM Killer activates, something is wrong with the system’s memory management. Every OOM event must be investigated, not ignored.
oom_score is determined by memory usage — processes using lots of memory get high scores and are more likely to be killed. oom_score_adj allows manual adjustments for critical processes.
Databases and monitoring agents must be protected — set oom_score_adj to -1000 (via systemd OOMScoreAdjust) for processes that must not be killed. Don’t set all processes to -1000 — the OOM Killer needs killable processes.
SIGKILL can’t be caught — when an OOM kill happens, no cleanup is possible. Prevention strategies (regular checkpoints, memory limits) are far better than recovery strategies.
vm.overcommit_memory=1 is a dangerous choice — malloc() never fails but OOM kills can happen anytime unpredictably. Use the default or strict mode.
Per-process memory limits isolate failures — if one process leaks memory and has a MemoryMax, it kills itself without causing a system-wide OOM.
Active swap is a memory saturation sign — swap access is thousands of times slower than RAM. Set vm.swappiness low (10) and alert when swap starts being used.
Memory leaks are detected from trends, not snapshots — monitoring memory usage every minute and analyzing its growth is far more useful than just viewing current usage.
dmesg and journalctl -k are the sources of truth for OOMs — every OOM event leaves detailed traces in the kernel log including the memory state at the moment.
Capacity planning prevents OOMs — 30% headroom of total memory, scaling before reaching saturation, and load testing to know capacity limits are the best defenses.
#

← Previous: Saturation Next: Infrastructure as Code