Kafka consumer lag

• The consumer_lag alert fires for a consumer group; the lag trend is monotonically rising, not sawtoothing.
• Downline effects: pipeline tables stop updating, event-driven jobs stop firing, the "data fresh" dashboard goes stale.
• End-to-end latency SLO breaches; sometimes the producer side feels fine, which is exactly why it gets noticed late.

• Sev-2 if lag is growing but the service still meets its freshness SLO with headroom; Sev-1 once freshness for a customer-facing dataset is breached or lag exceeds the retention window (messages will be LOST on expiry — that is unrecoverable).

• Freeze deployments of the consuming service and anything that changes topic partition counts. A deploy mid-triage adds a variable you will have to subtract later.
• Check topic retention vs current lag in BYTES: if projected time-to-expiry is under one hour, page the owner now, do not wait for the next milestone.
• If the consumer writes to a database, check for lock contention or a full disk on the sink — a stalled sink masquerades as consumer lag.

• Is ONE partition lagging or ALL of them? One partition usually means one stuck consumer instance; ALL means the group is under-provisioned, paused, or the broker is slow.
• Did the group rebalance recently? Rebalance storms show as sawtooth lag with frequent consumer group rebalanced log lines.
• Is the consumer process alive and burning CPU (busy loop) or idle (blocked on the sink)?

1. Consumer too slow (most common): handler does synchronous I/O per message, or batch size is 1. 2. Rebalance storm: session timeouts too tight for GC pauses, or instances flapping. 3. Traffic step-change: a new producer shard doubled volume and the group never scaled. 4. Broker degradation: disk saturation on one broker pins the partitions it leads. 5. Sink backpressure: the consumer is fine; the database/index it writes to is the blocker.

• One partition stuck → restart the owning consumer instance first; if it re-sticks, capture a thread dump BEFORE the next restart and go to Remediation (b).
• All partitions, consumer CPU idle → check the sink (db locks, index quota, rate limits); fix the sink, lag drains itself.
• All partitions, consumer CPU pegged → throughput problem: scale instances up to the partition count, then optimize the handler (a).
• Lag sawtooths with frequent rebalances → stop-the-world GC or flapping instances: go to (c).
• Under-replicated partitions on the broker → broker disk/io issue; escalate to the platform team with the broker metrics.

• (a) Throughput: increase consumer instances to partition count; raise max.poll.records and batch the sink writes; commit offsets after the batch, not per record.
• (b) Stuck instance: restart it (one at a time, waiting for the group to stabilize between restarts); keep the thread dump — it is the evidence for the postmortem.
• (c) Rebalance storm: raise session.timeout.ms and heartbeat.interval.ms to ride GC pauses; fix the GC pause itself if over 1s; use cooperative-sticky assignment if available.
• (d) Broker degradation: move leadership off the hot broker (preferred leader election), then let storage follow up on the disk.
• If lag is projected to expire messages, and the data is reproducible upstream, the honest remediation is: let it expire, re-play from the source, and say so in the incident channel.

• Lag trend flattens and then drains: the derivative of consumer_lag is negative for 10+ minutes — a one-minute dip is not recovery.
• Freshness SLO back in budget and the end-to-end latency dashboard green.
• No rebalance events in the last 15 minutes (a group still thrashing has not recovered, it is pausing).

• Consumer service owner first (they know the handler); platform/kafka team if broker metrics are implicated; data owner if messages will expire — they decide replay vs accept-loss.
• Hand-off state: group name, lag slope, time-to-expiry estimate, what has been tried, thread dumps captured.

• Alert on lag DERIVATIVE and time-to-expiry, not raw lag — raw lag thresholds page you after the decision window is gone.
• Load-test the consumer at 2× peak before onboarding new high-volume topics.
• Runbooks linked from the alert itself: the paging rule carries this page's URL.

• consumer_lag per group and partition (derivative alert).
• kafka_network_request_handler_avg_idle_percent per broker (broker saturation).
• rebalance_count per group per hour (storm detector).
• Sink-side: database lock waits and insert latency for the consumer's target.

• Apache Kafka consumer configs
• Consumer group rebalancing behavior