Consumer lag is one of the most-watched metrics in any Kafka deployment, and one of the most misread. A lag number on its own tells you almost nothing. What matters is whether that lag is growing, stable, or shrinking, and at what rate. This guide covers how to read consumer lag accurately, what causes it to grow, and how to set thresholds that page you when something is actually wrong.

What consumer lag actually measures

Consumer lag is the difference between the latest offset produced to a partition and the latest offset committed by a consumer group. A lag of 10,000 means the consumer is 10,000 messages behind the producer. That sounds alarming. But if your consumer processes 50,000 messages per minute and the producer is writing 50,000 messages per minute, a lag of 10,000 represents about 12 seconds of backlog. Whether that is a problem depends entirely on your latency requirements.

The three lag patterns that matter

Stable lag means the consumer is keeping up with the producer but started behind. This is common after a restart. Growing lag means the consumer cannot keep up. This is the pattern that warrants investigation. Oscillating lag, where the number rises and falls on a regular cycle, often indicates a batch process upstream or a consumer that is pausing for GC. Each pattern has a different cause and a different fix. Alerting on the raw number catches all three indiscriminately.

Why raw lag thresholds cause alert fatigue

Setting a lag alert at a fixed number, say 5,000 messages, without accounting for your normal throughput variance is the most common miscalibration we see. A topic that normally carries 200 messages per minute will look very different from one that carries 200,000. We recommend setting lag alerts as a function of your average throughput: alert when lag exceeds N minutes of normal production volume, not when it exceeds N messages.

How to measure lag accurately

The kafka-consumer-groups.sh tool gives you a point-in-time snapshot. For continuous monitoring, the JMX metrics exposed by the broker and the kafka_consumergroup_lag metric in the Kafka Exporter for Prometheus are more useful. Make sure you are measuring lag per partition, not just the sum across a consumer group. A sum of zero can hide one partition that is stuck at a large lag while others are at zero.

Setting thresholds that work in production

Start by recording your normal lag range over two weeks of production traffic. Note the peak lag during your highest-throughput periods. Set your warning threshold at twice that peak. Set your critical threshold at five times that peak. Review these numbers every quarter. As your throughput grows, your thresholds need to grow with it. A threshold that was correct in January may be wrong by June.

Consumer lag is a useful signal, but only if you know what normal looks like for your specific workload. The articles on this site cover related topics including dead-letter queue design and partition tuning. If you want a second opinion on your current alert configuration, the pipeline audit is the right starting point.