Skip to main content
OpsRoadmaps

Runbooks / rds-high-cpu

RDS high CPU

CPUUtilization on the RDS instance is pegged and queries are queueing — separate a bad query from a sizing ceiling before scaling.

12 canonical sections · 3 command blocks · last reviewed · updated 2026-09-21

Related focus areas:CloudOpsDataOpsSRE

Fix the underlying gap:AWS Security ImprovementCloud Cost OptimizationIncident Management MaturityObservability MaturityPlatform Engineering AdoptionProduction-Ready KubernetesResilience and DR ImprovementSRE Adoption

On this page
  1. Symptoms
  2. Severity / impact
  3. Immediate safety actions
  4. Fast checks
  5. Diagnostic commands
  6. Likely causes
  7. Decision tree
  8. Remediation
  9. Verification
  10. Escalation
  11. Prevention
  12. Related metrics / alerts
  13. Sources

Symptoms

  • CloudWatch CPUUtilization for the DB instance sits >90% for 10+ minutes; DatabaseConnections often climbs at the same time as requests queue.
  • App-side symptoms: p99 query latency jumps, connection pool exhaustion, gateway timeouts on DB-backed endpoints.

Severity / impact

  • Sev-2 while the buffer cache still serves reads and latency degrades gracefully; Sev-1 when connection pools exhaust and endpoints start failing outright, or when this instance is the primary of a production write path.

Immediate safety actions

  • Do NOT restart/failover first: failover does not fix a query that will peg the replica too, and it drops in-flight transactions.
  • If an index migration or bulk job is running from a deploy window, pause it — the fastest recovery is often "stop the new thing".
  • Protect the runbook for this incident itself: capture pg_stat_activity (or the engine equivalent) NOW; five minutes later the evidence may be rotated away.

Fast checks

  • One query or many? Sort active queries by duration and by rows: one long runner points at a query plan; many similar short queries point at a traffic pattern or a missing index on a hot path.
  • Did ReadIOPS fall off a cliff right before CPU spiked? That is the buffer cache being wiped (usually by a huge scan), not a sizing problem.
  • Maintenance window or auto-vacuum running concurrently?

Diagnostic commands

-- Postgres: what is running right now, longest first:
SELECT pid, now() - query_start AS runtime, state, wait_event_type, left(query, 120)
FROM pg_stat_activity
WHERE state <> 'idle'
ORDER BY runtime DESC LIMIT 20;
-- Statements accumulating the most time (needs pg_stat_statements):
SELECT calls, round(total_exec_time::numeric, 0) AS total_ms, left(query, 120)
FROM pg_stat_statements ORDER BY total_exec_time DESC LIMIT 10;
# Engine + instance class + storage metrics in one view (aws cli):
aws rds describe-db-instances --db-instance-identifier "$DBID" \
  --query 'DBInstances[0].[Engine,DBInstanceClass,StorageType,AllocatedStorage]'

Likely causes

1. A new or plan-flipped query doing a sequential scan on a large table (deploy, stats drift, or a data-size threshold crossed). 2. Legitimate traffic growth past the instance class ceiling — CPU correlates with connections and TPS over weeks. 3. Autovacuum/ANALYZE or a backup window coinciding with peak load. 4. Lock contention burning CPU in wait queues: many short queries, one blocker.

Decision tree

  • One dominant statement in pg_stat_statements → query-plan problem: EXPLAIN it, check for a missing index or stale stats; go to Remediation (a).
  • CPU ramps with connections/TPS over days-weeks → sizing ceiling: (b).
  • Spike started inside a maintenance/backup window → (c): reschedule the window; verify with the timeline.
  • Many queries waiting on locks → find the blocker pid in pg_stat_activity (wait_event + blocking_pids) and decide kill vs let-finish.

Remediation

  • (a) Bad plan: create the missing index CONCURRENTLY (online, no lock); ANALYZE the table if stats are stale; as a stopgap, kill the offending pid and let the app's retry + circuit breaker absorb it.
  • (b) Sizing: scale the instance class up (fast, in-place for most engines) and schedule storage/IO review afterwards — scaling up mid-incident is fine, capacity planning in the incident channel is not.
  • (c) Window collision: move the maintenance window off peak; for autovacuum, tune per-table thresholds rather than disabling it globally.
  • Locks: kill the blocker if it is safe (pg_terminate_backend(pid)), otherwise communicate the wait; fix the transaction pattern afterwards.

Verification

  • CPUUtilization back under 80% and steady for 15+ minutes; connections at pre-incident baseline.
  • p99 query latency recovered; app connection-pool waiters at zero.
  • If scaled: confirm the change is recorded as a cost impact with an owner for right-sizing later.

Escalation

  • DBRE/platform on-call for instance-level actions (scaling, failover, parameter changes); service owners for the offending query (they ship the index fix); eng leadership if write-path data loss risk emerges (e.g., replicas falling behind).

Prevention

  • CloudWatch alarm on CPU > 80% for 10 min AND on connection count trending to the max — lead time is everything.
  • pg_stat_statements (or Performance Insights) always on: incidents without query telemetry are guesswork.
  • Deploy playbook step: "check top queries after rollout" — plan flips are found in minutes, not mornings.

Sources

Know which failure modes you actually have?

A free OpsRoadmaps assessment scores your controls with explainable findings and builds a prioritized roadmap — runbook coverage is one of the things it measures.

Assess my stack

Runbooks document self-reported practice. Commands and thresholds were last reviewed 2026-09-21 — verify against your own environment before acting.