ALB 5xx spike

• Alarms on HTTPCode_ELB_5XX_Count or HTTPCode_Target_5XX_Count fire; users see 502/503/504 on ALB-fronted paths.
• Often correlates with a deploy, a target group change, or a slow backend; sometimes appears as "the site is down" with the app itself feeling fine to internal checks.
• Code semantics matter before anything else: 502 = target closed/invalid response, 503 = no healthy targets, 504 = target timeout (idle timeout exceeded).

• Sev-1 when user-facing traffic fails outright; 503 with zero healthy targets is a full outage of that path.
• Sev-2 for partial 5xx (a percentage, one target pool, one region) or 504s on a non-critical async path.

• Identify ELB-generated vs target-generated 5xx from CloudWatch BEFORE touching anything: HTTPCode_ELB_502/503/504_Count vs HTTPCode_Target_5XX_Count. The fix sets are disjoint.
• If a deploy or deregistration just happened, roll back / pause the rollout — target churn is the most common self-inflicted 5xx.
• Do not "restart the ALB" — there is no such action; changing listener/routing config mid-triage adds a variable.

• Target health for the ALB's target groups: how many healthy, and why are the rest failing (Target.Timeout? Target.FailedHealthChecks? Elb.InternalError)?
• 503 pattern: healthy count zero for a group = nothing to route to (SG/NACL/registration/deregistration).
• 504 pattern: TargetResponseTime p99 climbing toward the idle timeout = backend slowness, not the ALB.
• 502 pattern: app crashing responses (early close, invalid headers) — often an app-level error correlated with a release.

1. No healthy targets → 503: registration broke, health checks fail (wrong path/port), or SG/NACL blocks the checker (most common after infra changes). 2. Backend too slow → 504: target p99 exceeds the idle timeout (default 60s) — slow endpoint, lock contention, downstream saturation. 3. Target connection broken → 502: app crashing, keep-alive shorter than the ALB's idle timeout (ALB reuses a connection the app already closed), or invalid responses (headers/protocol). 4. Deregistration storm: deploy replaced all targets at once; in-flight requests hit closing connections → burst of 502/503. 5. True backend 5xx: target app erroring — visible in HTTPCode_Target_5XX_Count and the app's own logs; the ALB is the messenger.

• 503 with 0 healthy targets → (a): fix registration/health-check/SG; the moment one target passes, traffic resumes.
• 504 with rising TargetResponseTime → (b): chase the backend slowness (its own runbook); raise idle timeout ONLY if slow-but-valid responses are intentional.
• 502 right after deploys → (c): add connection draining, fix the app's keep-alive, slow the rollout.
• 502 persistent on one path → app crash/protocol issue: read the target's own error logs (d).
• HTTPCode_Target_5XX_Count mirrors app errors 1:1 → stop blaming the LB; open the application's incident.

• (a) Health: correct the health-check path/port/interval to match what the app actually serves (a 200 on /healthz, not a redirect); open the SG to the ALB's subnets or the VPC CIDR for the check port; re-register targets and confirm healthy ≥ replicas per AZ.
• (b) Timeouts: raise the ALB idle timeout above the backend's slowest legitimate response (listener attribute routing.http.idle_timeout) and cap the BACKEND's own timeout below it — the app must fail before the LB gives up, or users see 504 instead of a clean app error. Fix the underlying slowness as its own item.
• (c) Draining: ensure deregistration delay (default 300s) is not fighting a fast-scaled deployment; deploy with rolling updates so N-1 targets stay healthy; app keep-alive ≥ ALB idle timeout.
• (d) App-side 502s: fix the crash/invalid response; check for response headers that violate HTTP (control chars, oversized) which the ALB refuses to forward.
• Communicate which class it was — "ALB 5xx" in the postmortem without the ELB/target split sends the next responder down the wrong path.

• 5xx of the identified class at zero (or pre-incident baseline) for 30+ minutes under real traffic.
• Target group shows expected healthy counts across AZs; HealthyHostCount stable through a test deploy.
• If timeouts were changed: p99 still within SLO and no new 504 class appearing at the new boundary.

• Service owners for target-side errors and slowness; network/platform for SG/NACL/registration; AWS support (via your account's support path) only when metrics show ELB-internal errors (Elb.InternalError) — that is the one genuinely AWS-owned branch.
• Hand-off state: ELB vs target 5xx counts with timestamps, target health snapshot, idle timeout + deregistration delay values, recent deploys.

• Alarm on HTTPCode_ELB_5XX_Count AND HTTPCode_Target_5XX_Count separately; alert on HealthyHostCount < desired per group.
• ALB access logs on (they name elb_status_code vs target_status_code per request — the split, precomputed).
• Connection draining + rolling deploys as defaults; keep-alive and timeout budget documented per service (app < ALB idle < client).

• HTTPCode_ELB_502/503/504_Count, HTTPCode_Target_5XX_Count.
• HealthyHostCount / UnHealthyHostCount per target group; TargetResponseTime p99.
• RequestCount per target (uneven distribution = routing/weight problem hiding under 5xx).

• Troubleshooting Application Load Balancers
• CloudWatch metrics for your Application Load Balancer