Competing consumers and work queues

Chapter 7’s groupId configuration was introduced almost in passing, distinguishing publish-subscribe (each service its own consumer group) from something else it deferred. This chapter is that something else: using multiple instances of one service, sharing one consumer group, to distribute — not duplicate — a stream of work.

1. Problem the Pattern Solves

notification-service (added in Chapter 7’s exercise) sends an order-confirmation email for every OrderPlaced event. During a normal day, one instance keeps up easily. During a flash sale, order volume spikes tenfold, and a single instance’s email-sending throughput — bounded by the sending provider’s API rate limit per connection and the CPU cost of rendering each email template — can’t keep up. A growing backlog of unsent confirmation emails means customers wait twenty, then forty minutes for a receipt that should arrive in seconds.

Scaling notification-service to three replicas doesn’t help by itself if each replica is in its own Kafka consumer group (as inventory-service and payment-service correctly are, per Chapter 7’s design for independent, parallel reactions to the same event) — that configuration would mean each of the three replicas processes every email independently, sending each confirmation three times, which is a worse outcome than the original backlog.

Forces in tension:

• Throughput vs. correctness of “exactly one worker per task.” Distributing work across multiple instances scales throughput, but only if the distribution mechanism guarantees each unit of work goes to exactly one instance — not zero, and not several.
• Ordering vs. parallelism. Kafka’s ordering guarantee is per-partition; distributing work across multiple consumer instances (which Kafka does by assigning different partitions to different instances within a group) means work from different partitions can be processed out of relative order — acceptable for independent emails, unacceptable if the work items have a required processing sequence.
• Elasticity vs. partition count as a hard ceiling. Kafka can’t have more active consumers in a group than partitions on the topic — scaling beyond the partition count adds idle instances, not more throughput, a specific and easy-to-miss capacity-planning trap.
• Failure handling per unit of work. A worker crashing mid-task shouldn’t lose that task — it should become available for another worker to pick up, which depends on correctly configured offset-commit behavior (commit only after successful processing, not before).

2. Core Idea

Competing consumers means multiple instances of the same service share one Kafka consumer group, so that each message on the topic is delivered to exactly one instance within that group — the instances “compete” for messages, each processing a disjoint subset, giving horizontal scaling of throughput for a stream of independent work items.

Compare this directly to Chapter 7’s diagram: there, inventory-service and payment-service were in different consumer groups, each receiving its own full copy of every OrderPlaced event — publish-subscribe. Here, all three notification-service replicas share one consumer group — competing consumers. Same broker, same topic mechanics, a completely different delivery semantic, chosen by how consumer groups are configured, not by anything different about Kafka itself.

Participants:

• Work queue (topic) — order-confirmations, partitioned to allow parallel consumption; partition count sets the hard ceiling on how many active consumer instances can do useful work simultaneously.
• Competing consumers — multiple notification-service replicas, all in the notification-service consumer group, each assigned a disjoint subset of partitions by Kafka’s group coordinator.
• Partition assignment — Kafka’s own rebalancing mechanism, which reassigns partitions among the group’s current members whenever an instance joins or leaves (a scale-up, a scale-down, or a crash).

Commonly confused with:

• Publish-subscribe (Chapter 7 and 8). The distinguishing factor is entirely the consumer group configuration: same group, one instance per message (competing consumers, this chapter); different groups, every instance its own copy (pub-sub, Chapters 7–8). The same Kafka topic can technically be consumed both ways simultaneously by different services, as Northwind’s orders.events topic already effectively demonstrates across inventory-service, payment-service, and (via its own topic) notification-service.
• A saga’s choreography (Chapter 11). A saga’s event handlers react to specific business events with specific compensating logic; competing consumers is a generic scaling mechanism for any stream of independent, parallelizable work — a saga step could itself be implemented by a competing-consumers group if that step’s processing needs to scale, but the two concepts operate at different levels.
• Kubernetes horizontal pod autoscaling alone. Scaling notification-service’s replica count via HPA (Chapter 1’s Kubernetes discussion) only produces more throughput if those replicas are actually distributing work correctly via a shared consumer group — adding replicas without this pattern’s consumer-group configuration, as Section 1 shows, produces duplicated processing instead of parallelized processing.

3. When to Use It

Strong indicators:

• A stream of independent, order-insensitive (or order-tolerant-within-acceptable-bounds) units of work whose processing throughput needs to scale with volume — Northwind’s confirmation emails are exactly this: each is independent of every other.
• Demonstrated or anticipated volume spikes (flash sales) that a single consumer instance can’t handle within an acceptable processing-latency target.
• Work whose individual processing cost (CPU, external API calls) is high enough that parallelizing across instances meaningfully helps, as opposed to work so cheap that a single instance already processes it faster than it arrives.

Concrete use cases:

• E-commerce, as here: order-confirmation emails, receipt PDF generation, and post-purchase analytics event processing are all naturally parallelizable, independent work streams.
• Image/video processing pipelines: resizing uploaded images or transcoding video are CPU-intensive, embarrassingly parallel tasks that scale directly with more competing consumer instances.
• Batch data processing: nightly ETL jobs split into per-partition work units, processed by a pool of worker instances that scale up during the batch window and down afterward.
• Any background-job system: sending SMS notifications, generating reports, processing webhook deliveries to external systems — all are classic competing-consumer workloads once volume exceeds a single worker’s capacity.

Prerequisites:

• A topic partitioned appropriately for the target parallelism — partition count must be planned with headroom for future scaling, since increasing partition count later is possible but can disrupt existing key-based ordering guarantees (a consideration specific to Kafka’s partition-increase semantics).
• Idempotent or safely-retriable work items — a rebalance can cause a message to be reprocessed by a different instance if the original didn’t commit its offset in time, so “sent this email exactly once” needs the same idempotency discipline this series has required since Chapter 1 (Section 5 shows the specific mechanism for this workload).
• An understanding of what ordering guarantee, if any, the workload actually needs — Northwind’s confirmation emails need none; a workload that does needs careful partition-key design (Chapter 7’s per-order keying) rather than this pattern’s default parallelism.

4. When Not to Use It

• Work with a strict, cross-item ordering requirement that spans partitions. If processing order must be strictly global (rare, but real for some ledger-style workloads), competing consumers’ partition-parallel processing directly conflicts with that requirement — a single-consumer, single-partition design, accepting the throughput ceiling, may be the correct trade-off there.
• Low, steady volume that a single instance handles comfortably. Provisioning multiple notification-service replicas and worrying about partition counts for a workload that never exceeds a single instance’s capacity is unneeded operational complexity — evaluate against measured or realistically projected peak volume.
• Work that isn’t safely retriable/idempotent, with no way to make it so. If a work item’s processing has an irreversible side effect that can’t be made idempotent (rare, but worth checking explicitly), a rebalance-triggered reprocessing becomes a real correctness problem this pattern can’t fully protect against — that work item needs a different design, or a compensating mechanism (Chapter 11’s saga thinking, applied at the level of a single work item).
• Overengineering signal: massively over-partitioning a topic “for future scale” well beyond any realistic projected need, since each partition is itself a small resource cost (open file handles, replication overhead) on the broker — size partition count to a reasonable multiple of anticipated peak parallelism, not an arbitrarily large number.

5. Implementation Example

Topic provisioning, sized for Northwind’s anticipated flash-sale peak parallelism:

1
kafka-topics.sh --create --topic order-confirmations --partitions 12 --replication-factor 3 \
2
  --bootstrap-server kafka.default.svc.cluster.local:9092

Twelve partitions gives headroom for up to twelve actively-consuming notification-service replicas — chosen based on Northwind’s projected flash-sale peak of roughly ten replicas needed, with a small margin, not an arbitrarily large number (Section 4’s overengineering warning, made concrete).

The competing consumer, deployed as multiple replicas sharing one consumer group:

1
package `in`.o612.eng.northwind.notification
2

3
import `in`.o612.eng.northwind.order.api.OrderPlacedNotification
4
import org.springframework.kafka.annotation.KafkaListener
5
import org.springframework.stereotype.Component
6

7
@Component
8
class OrderConfirmationListener(
9
    private val emailSender: EmailSender,
10
    private val sentEmailLog: SentEmailLog,   // idempotency guard, see below
11
) {
12
    @KafkaListener(
13
        topics = ["order-confirmations"],
14
        groupId = "notification-service",   // shared across every replica — this is what makes them compete, not duplicate
15
        concurrency = "3",                   // 3 consumer threads per pod instance, in addition to horizontal replica scaling
16
    )
17
    fun onOrderPlaced(notification: OrderPlacedNotification) {
18
        if (sentEmailLog.alreadySent(notification.orderId)) return   // idempotency check — see Section 7
19
        val email = emailSender.sendConfirmation(notification.orderId)
20
        sentEmailLog.recordSent(notification.orderId)
21
    }
22
}

1
spring:
2
  kafka:
3
    consumer:
4
      group-id: notification-service
5
      enable-auto-commit: false   # commit only after successful processing — see Section 7
6
      max-poll-records: 10        # bound how much work one poll claims, avoiding one slow batch blocking rebalance
7
listener:
8
  ack-mode: manual_immediate       # explicit ack after successful send, not before

1
apiVersion: autoscaling/v2
2
kind: HorizontalPodAutoscaler
3
metadata:
4
  name: notification-service
5
spec:
6
  scaleTargetRef: { apiVersion: apps/v1, kind: Deployment, name: notification-service }
7
  minReplicas: 2
8
  maxReplicas: 10
9
  metrics:
10
    - type: External
11
      external:
12
        metric:
13
          name: kafka_consumergroup_lag
14
          selector: { matchLabels: { topic: order-confirmations, consumergroup: notification-service } }
15
        target: { type: AverageValue, averageValue: "50" }

Scaling on consumer lag, not CPU, is the deliberate, correct choice here — CPU utilization doesn’t directly reflect whether the backlog is growing; lag does, directly answering “is this service keeping up with incoming work,” which is precisely the question autoscaling needs answered for a queue-consuming workload.

Idempotency guard, since a rebalance mid-processing (a replica scaling down, or crashing) can cause the same message to be redelivered to a different replica after the original committed no offset:

1
package `in`.o612.eng.northwind.notification
2

3
import org.springframework.jdbc.core.JdbcTemplate
4
import org.springframework.dao.DuplicateKeyException
5
import org.springframework.stereotype.Component
6
import java.util.UUID
7

8
@Component
9
class SentEmailLog(private val jdbc: JdbcTemplate) {
10
    fun alreadySent(orderId: UUID): Boolean =
11
        jdbc.queryForObject("SELECT COUNT(*) FROM sent_emails WHERE order_id = ?", Int::class.java, orderId)!! > 0
12

13
    fun recordSent(orderId: UUID) {
14
        try {
15
            jdbc.update("INSERT INTO sent_emails (order_id, sent_at) VALUES (?, now())", orderId)
16
        } catch (e: DuplicateKeyException) {
17
            // Two competing consumers raced on the same redelivered message
18
            // — the email was (or is about to be) sent exactly once either
19
            // way; this just prevents a duplicate log entry, not a duplicate send.
20
        }
21
    }
22
}

This is the same inbox-style idempotency discipline from Chapter 12, applied to a competing-consumers workload rather than a pub-sub one — the mechanism generalizes directly.

6. Step-by-Step Flow

1. Client action. A flash sale drives a sharp spike in OrderPlaced events, each producing a corresponding order-confirmations message.
2. API request equivalent. Kafka’s consumer-group coordinator rebalances partition assignment as the HPA scales notification-service from 2 to 6 replicas in response to growing lag.
3. Service behavior. Each replica processes only the messages from its assigned partitions — no replica sees a message another replica is also handling.
4. Database interaction. Each replica checks and records into the shared sent_emails table before/after sending, guarding against the specific redelivery risk a rebalance introduces.
5. Inter-service communication. None beyond the Kafka consumption itself — notification-service doesn’t call any other Northwind service to do its job.
6. Error or failure handling. If a replica crashes mid-processing before committing its offset, that message is redelivered to whichever replica picks up the orphaned partition after rebalancing — the idempotency guard ensures this doesn’t produce a duplicate email.
7. Observability signals. Consumer lag per partition, and the rate of DuplicateKeyException catches in SentEmailLog (a direct measure of how often rebalance-triggered redelivery is actually happening) are the two most important metrics for this pattern specifically.
8. Final response/outcome. Confirmation emails go out within the target latency even during a tenfold volume spike, because throughput scaled with the number of active competing consumers, up to the partition-count ceiling.

7. Production Concerns

• Timeouts, retries, idempotency. enable-auto-commit: false plus ack-mode: manual_immediate (Section 5) ensures an offset is only committed after successful processing — the specific configuration that makes “crash mid-processing → redelivered, not lost” true; getting this backward (committing before processing) would silently lose work on a crash instead.
• Data consistency. The idempotency guard’s sent_emails table check-and-insert should itself be as atomic as the database allows (a unique constraint on order_id, as the DuplicateKeyException handling assumes) — a check-then-insert with a race window wide enough for two competing consumers to both pass the check is a real risk under high concurrency, worth verifying with a targeted concurrency test.
• Partition count as a scaling ceiling. Twelve partitions means twelve is the hard limit on useful parallelism, regardless of how many replicas the HPA spins up — monitor for “replicas without partitions assigned” (idle, wasted replicas) as a sign the partition count needs revisiting, not just the replica count.
• API versioning. Not directly relevant — this pattern is about consumption topology, not message schema, though the message schema itself follows the same versioning discipline as any other event (Chapters 7–8).
• Authentication and service-to-service trust. No new surface beyond standard Kafka ACLs (Chapter 7) — all replicas in a consumer group share the same identity/credential for consuming this topic.
• Logging, metrics, tracing, correlation IDs. Tag every log line with which replica (pod name) and which partition processed a given message — essential for diagnosing an uneven load distribution or a specific partition falling behind.
• Kubernetes deployment, autoscaling. Scale on consumer lag (Section 5), not CPU or memory — a queue-consuming workload’s “am I keeping up” question is answered directly by lag and only indirectly, unreliably, by resource utilization.
• Testing strategy. Test the idempotency guard under simulated concurrent redelivery (two threads racing to process the same orderId) explicitly — this is the one behavior in this pattern that’s genuinely hard to catch with a single-threaded test but easy to get wrong under real rebalance conditions.
• Migration strategy. If notification-service currently runs as a single instance with no consumer-group scaling concerns, introduce competing consumers only once a real volume-driven backlog (as Section 1 describes) justifies the added partition-planning and idempotency work — not preemptively for a workload that hasn’t shown the need.

8. Common Mistakes

1. Scaling replicas without a shared consumer group. Running three notification-service replicas each in its own consumer group (a copy-paste of Chapter 7’s pub-sub configuration) triples email sends instead of parallelizing them — exactly Section 1’s opening failure mode. Fix: verify every replica intended to compete for work shares the identical groupId.
2. Partition count lower than the target maximum parallelism. Provisioning a topic with 3 partitions while planning to scale to 10 replicas wastes 7 replicas that receive no partition assignment at all. Fix: provision partition count with headroom for realistic peak parallelism, as Section 5 does.
3. Committing offsets before processing completes. Using auto-commit (or manually acknowledging before the email actually sends) means a crash after the commit but before the send silently loses that confirmation email — the opposite failure mode from the intended “redelivered, not lost” guarantee. Fix: commit (acknowledge) only after processing succeeds, as Section 5’s manual_immediate configuration does.
4. No idempotency guard, assuming rebalances are rare enough to ignore. Rebalances happen routinely during normal autoscaling, not just crashes — treating redelivery as a rare edge case rather than a routine occurrence under this pattern leads to real, periodic duplicate sends. Fix: build the idempotency guard as a standard, always-on part of the consumer, not a defensive afterthought.
5. Scaling on CPU instead of consumer lag. A competing-consumers workload can have low CPU utilization per instance while still falling badly behind (e.g., each task spends most of its time waiting on a slow external email-provider API) — CPU-based autoscaling won’t react to this at all. Fix: scale on consumer lag directly, as shown in Section 5’s HPA configuration.
6. Assuming any ordering guarantee across the whole topic. Relying on messages from different partitions being processed in the order they were produced (they generally won’t be, under competing consumers) breaks any workload with cross-item sequencing needs. Fix: if ordering matters at all, design the partition key deliberately (as Chapter 7 did with orderId) so related items land on the same partition and are processed in relative order by whichever single consumer holds that partition — never assume global ordering across partitions.

9. Decision Guide

10. Hands-On Exercise

Extend it: provision a second competing-consumers topic, receipt-pdf-generation, for a new, more CPU-intensive work type (rendering a PDF receipt), and choose a partition count independently justified by that workload’s expected volume and per-item processing cost — explain why it might reasonably differ from order-confirmations’ twelve.

Simulate a failure: force a rebalance mid-processing (scale notification-service down by one replica while it’s actively consuming a backlog) and confirm, using the sent_emails table, that no order’s confirmation email is either lost or sent more than once.

Decision question, with justification required: Northwind wants confirmation emails sent in the exact order orders were placed, per customer (so a customer never receives their second order’s confirmation before their first’s, if both were placed within the same second during a flash sale). Does the current partition-by-nothing-in-particular scheme support this, and if not, what partition-key change would you make, and what does that change cost in terms of this pattern’s parallelism ceiling?

11. Key Takeaways

• Competing consumers means multiple instances of the same service share one Kafka consumer group, so each message is processed by exactly one instance — the direct opposite delivery semantic from publish-subscribe’s “every group gets its own copy,” configured purely by whether instances share a groupId.
• Partition count sets a hard ceiling on useful parallelism — provision it with headroom for realistic peak scale, and monitor for idle, unassigned replicas as a sign it needs revisiting.
• Commit offsets only after successful processing, never before — this is what makes a crash mid-processing result in safe redelivery rather than silent data loss.
• Idempotency is not optional under this pattern — rebalances (including routine autoscaling events, not just crashes) will cause redelivery, and every competing-consumer workload needs a guard against processing the same item twice.
• Scale on consumer lag, not CPU or memory — lag directly answers whether the consumer group is keeping up with incoming work, which is the actual question autoscaling needs to answer here.
• Kafka provides ordering only within a partition — competing consumers’ parallelism across partitions means no default guarantee of cross-item ordering; design the partition key deliberately if any ordering requirement exists.
• Apply this pattern to a demonstrated, volume-driven throughput problem, exactly as every other pattern in this series — not preemptively to a workload that hasn’t shown the need.