Publish-subscribe and event notification

Chapter 7 introduced OrderPlaced as a Kafka event consumed independently by inventory-service and payment-service. This chapter makes explicit a design decision that chapter glossed over: exactly how much data belongs inside that event, and why the answer differs by consumer.

1. Problem the Pattern Solves

Two months after Chapter 7 ships, the newly added loyalty-service (from that chapter’s exercise) needs to know the customer’s tier (GOLD, SILVER, STANDARD) to calculate points correctly — a field OrderPlaced doesn’t carry. The loyalty team’s fastest fix is to add a synchronous REST call back to order-service to fetch the customer’s tier whenever an OrderPlaced event arrives — quietly reintroducing exactly the temporal coupling Chapter 7 was built to remove, just one hop later than before.

Meanwhile, a different consumer, analytics-pipeline, wants the entire order payload — every line item, every price, shipping address, the works — to avoid a callback of its own. If order-service puts all of that directly into OrderPlaced to satisfy analytics-pipeline, inventory-service and payment-service now receive a payload ten times larger than they need, and every unrelated field order-service ever adds to an order becomes a change to a payload three other services deserialize.

Forces in tension:

• Coupling to the publisher’s schema vs. self-sufficiency. A large event lets consumers avoid ever calling back to the publisher, but tightly couples every consumer to the exact shape of the publisher’s internal data, which now can’t change without considering every subscriber.
• Payload size and broker load vs. round-trip elimination. Small events keep the broker’s throughput high and payloads cheap to serialize, but may force a consumer to make a follow-up call for data it needs, reintroducing a dependency the event was meant to avoid.
• Consumer autonomy vs. staleness risk. A “fat” event lets a consumer build its own local copy of exactly the data it needs (as loyalty-service’s tier lookup, done right, should) — but that local copy can drift from the source of truth if update events are ever missed or misordered.

2. Core Idea

Event notification: a thin event carrying only an identifier and the fact that something happened (OrderPlaced(orderId: UUID)), leaving it to interested consumers to call back to the publisher’s API if they need more detail. Simple, small, and minimizes schema coupling — at the cost of reintroducing a synchronous dependency for any consumer that needs data beyond the identifier.

Event-carried state transfer: a “fatter” event carrying enough data for consumers to act without calling back (OrderPlaced(orderId, items, customerId, customerTier, totalAmount)), letting each consumer maintain its own local, denormalized copy of the data it needs. Removes the callback dependency entirely — at the cost of a wider, more consequential schema that more consumers depend on directly.

Both are publish-subscribe — any number of independent subscribers, no publisher awareness of who’s listening (unchanged from Chapter 7) — the difference is purely in payload design, and Northwind, correctly, uses different answers for different consumers of the same underlying fact.

Commonly confused with:

• Event sourcing (a later chapter). Event-carried state transfer means a consumer builds a denormalized read model from events it receives; event sourcing means a service’s own authoritative state is the event log, replayable from scratch. loyalty-service’s local tier cache in this chapter is a read model built from events — not event sourcing, since order-service (the source of truth for orders) still stores current-state rows, not an event log, as established in Chapter 1.
• CQRS (a later chapter). A consumer maintaining its own read model from events is a building block CQRS often uses, but CQRS specifically means separating a single service’s own read and write models — this chapter is about data flowing between services, not within one.
• A shared database view. Building a local denormalized copy from events looks superficially similar to querying another service’s database directly, but the event is a versioned, intentional, publisher-controlled contract — a shared view is an accidental coupling to internal storage structure that Chapter 2 already ruled out.

3. When to Use It

Use thin event notification when:

• Few consumers need the extra data, and they need it rarely enough that an occasional callback is cheap relative to bloating every event for every consumer.
• The data referenced changes frequently or is large (e.g., a full customer profile) — carrying it in every event risks staleness or payload bloat for consumers who rarely need it.
• Strong consistency for that specific field matters more than avoiding a callback — a callback gets the current value; an embedded value in an old event might be stale.

Use event-carried state transfer when:

• A consumer’s core job depends on this data for every event, at high volume — loyalty-service calculating points for every single order is exactly this case; a callback per order recreates the coupling this pattern exists to avoid.
• The publisher can tolerate consumers holding slightly stale, eventually-consistent copies of the embedded data (a customer’s tier at the moment of ordering, not necessarily their tier right now).
• You want to eliminate a specific, identified synchronous dependency that has already caused (or is about to cause) an availability problem — not speculatively, for every event.

Concrete use cases:

• E-commerce: shipping notifications carrying enough address and item detail for a carrier-integration service to act without calling back to the order system for every shipment.
• SaaS billing: a “usage recorded” event carrying enough tenant and plan detail for a billing service to calculate charges without querying a tenant-management service per event, at high event volume.
• Logistics: a “shipment delayed” event notification (thin) is usually sufficient, since delay handling is infrequent enough that interested services calling back for full shipment detail is cheap.

Prerequisites:

• A clear, deliberate answer — per event type, not once for the whole system — to which shape fits, based on consumer volume, data size, and staleness tolerance, rather than a blanket policy.
• Consumers building local read models from fat events need their own reconciliation strategy for missed or out-of-order messages (Section 7).

4. When Not to Use It

• Using event-carried state transfer as a default for every event, “to avoid callbacks.” This is the actual failure Northwind almost had with analytics-pipeline’s full-payload request — every field order-service ever needs to change becomes a negotiation with every consumer that embedded it, defeating the loose coupling Chapter 7 established.
• Using thin notification for a high-volume, latency-sensitive consumer. Forcing loyalty-service to call back to order-service for every single order recreates a synchronous dependency at exactly the volume where Chapter 7’s cascading-failure risk reappears.
• Risk of both, done carelessly: a fat event whose embedded data silently becomes the de facto contract even for fields the publisher considered incidental — a consumer built against OrderPlaced.customerTier for a one-off report now depends on a field the publisher didn’t intend to promise long-term. Document embedded fields as explicitly as any other API contract, whichever shape you choose.

5. Implementation Example

analytics-pipeline — a low-frequency, exploratory consumer — uses thin notification and calls back:

1
package `in`.o612.eng.northwind.order.api
2

3
import java.util.UUID
4

5
/** Thin notification — the only guaranteed contract is that this order
6
 * exists and this fact happened. Consumers needing more call GET /orders/{id}. */
7
data class OrderPlacedNotification(val orderId: UUID, val occurredAt: java.time.Instant)

1
package `in`.o612.eng.northwind.analytics
2

3
import `in`.o612.eng.northwind.order.api.OrderPlacedNotification
4
import org.springframework.kafka.annotation.KafkaListener
5
import org.springframework.stereotype.Component
6

7
@Component
8
class OrderPlacedListener(private val orderServiceClient: OrderServiceClient, private val warehouse: AnalyticsWarehouse) {
9

10
    @KafkaListener(topics = ["orders.notifications"], groupId = "analytics-pipeline")
11
    fun onOrderPlaced(notification: OrderPlacedNotification) {
12
        // Low volume, exploratory use — a callback per event is an
13
        // acceptable, explicit trade-off here, not an accident.
14
        val fullOrder = orderServiceClient.getOrder(notification.orderId)
15
        warehouse.recordOrder(fullOrder)
16
    }
17
}

loyalty-service — high-volume, needs tier data on every event — uses event-carried state transfer and never calls back:

1
data class OrderPlacedForLoyalty(
2
    val orderId: UUID,
3
    val customerId: UUID,
4
    val customerTier: String,   // GOLD | SILVER | STANDARD, snapshotted at order time
5
    val totalAmount: java.math.BigDecimal,
6
    val occurredAt: java.time.Instant,
7
)

1
package `in`.o612.eng.northwind.order.internal
2

3
import `in`.o612.eng.northwind.order.api.OrderPlacedForLoyalty
4
import `in`.o612.eng.northwind.order.api.OrderPlacedNotification
5
import org.springframework.kafka.core.KafkaTemplate
6
import org.springframework.stereotype.Component
7

8
@Component
9
internal class OrderEventPublisher(
10
    private val kafkaTemplate: KafkaTemplate<String, Any>,
11
) {
12
    fun publishNotification(order: Order) {
13
        kafkaTemplate.send("orders.notifications", order.id.toString(),
14
            OrderPlacedNotification(order.id, order.placedAt))
15
    }
16

17
    fun publishForLoyalty(order: Order, customerTier: String) {
18
        kafkaTemplate.send("orders.loyalty-events", order.id.toString(),
19
            OrderPlacedForLoyalty(order.id, order.customerId, customerTier, order.totalAmount, order.placedAt))
20
    }
21
}

Two separate topics, deliberately — orders.notifications (thin, broad audience) and orders.loyalty-events (fat, narrow audience) — rather than one topic everyone subscribes to and filters. This keeps each topic’s schema evolution independent: analytics-pipeline’s topic can add fields without loyalty-service’s consumer ever noticing, and vice versa.

1
package `in`.o612.eng.northwind.loyalty
2

3
import `in`.o612.eng.northwind.order.api.OrderPlacedForLoyalty
4
import org.springframework.kafka.annotation.KafkaListener
5
import org.springframework.stereotype.Component
6

7
@Component
8
class OrderPlacedForLoyaltyListener(private val pointsCalculator: PointsCalculator) {
9

10
    @KafkaListener(topics = ["orders.loyalty-events"], groupId = "loyalty-service")
11
    fun onOrderPlaced(event: OrderPlacedForLoyalty) {
12
        // No callback to order-service at all — every field needed is
13
        // already in the event, snapshotted at the moment of ordering.
14
        pointsCalculator.awardPoints(event.customerId, event.customerTier, event.totalAmount)
15
    }
16
}

Reconciliation job, guarding against the staleness/drift risk that comes with any consumer maintaining its own copy of data from events:

1
package `in`.o612.eng.northwind.loyalty
2

3
import org.springframework.scheduling.annotation.Scheduled
4
import org.springframework.stereotype.Component
5

6
@Component
7
class TierDriftReconciliationJob(
8
    private val customerServiceClient: CustomerServiceClient,
9
    private val loyaltyRepository: LoyaltyRepository,
10
) {
11
    @Scheduled(cron = "0 0 3 * * *")
12
    fun reconcileTierSnapshots() {
13
        // Nightly, low-frequency check: compare a sample of recently-used
14
        // tier snapshots against the current source of truth, and alert
15
        // if drift exceeds a threshold — a safety net, not the primary path.
16
        val driftCount = loyaltyRepository.recentTierSnapshots()
17
            .count { snapshot -> customerServiceClient.currentTier(snapshot.customerId) != snapshot.tier }
18
        if (driftCount > DRIFT_ALERT_THRESHOLD) {
19
            // emit an alert metric — see Section 7
20
        }
21
    }
22

23
    companion object { const val DRIFT_ALERT_THRESHOLD = 50 }
24
}

6. Step-by-Step Flow

1. Client action. Unchanged from every previous chapter — POST /orders.
2. API request/event publication. order-service publishes two distinct events to two distinct topics after commit, each shaped for its audience.
3. Service behavior. analytics-pipeline reacts to the thin notification by calling back; loyalty-service reacts to the fat event with everything it needs already present.
4. Database interaction. Each consumer writes to its own database, per the database-per-service discipline from Chapter 3 — loyalty-service’s points ledger and analytics-pipeline’s warehouse are both independently owned.
5. Inter-service communication. analytics-pipeline’s callback is the only synchronous hop in this entire flow — a deliberate, documented exception, not an accident.
6. Error or failure handling. If order-service is down when analytics-pipeline’s callback fires, that specific event’s processing fails and retries later (dead-letter handling is the next chapters’ subject) — a cost loyalty-service never pays, since it needs nothing further from order-service to finish processing.
7. Observability signals. Track callback rate and callback failure rate for analytics-pipeline specifically — a rising callback failure rate is a direct, measurable cost of the thin-notification choice for that consumer.
8. Final response/outcome. Both consumers end up with correct data, using two different mechanisms chosen deliberately for their two different volume and coupling profiles.

7. Production Concerns

• Timeouts, retries, idempotency. The callback path (thin notification) needs its own retry and timeout handling, exactly like any synchronous call (Chapter 6) — it hasn’t stopped being a network call just because it started life as a reaction to an event.
• Data consistency and transaction boundaries. Fat events snapshot data at publish time — customerTier in OrderPlacedForLoyalty reflects the tier when the order was placed, not the tier right now. This must be a documented, intentional semantic, not an accidental staleness bug — for loyalty-point calculation, “tier at time of purchase” is usually the correct business rule, not a compromise.
• Schema evolution and backward compatibility. Fat events accumulate more fields that more consumers depend on over time — track which consumers use which fields (a schema registry with compatibility checking helps concretely here) so a field removal doesn’t silently break a consumer nobody remembered was using it.
• API versioning. The callback endpoint (GET /orders/{id}) used by thin-notification consumers is now a real, ongoing API contract, not an implementation detail — version and document it with the same care as any other public REST contract from Chapter 2 onward.
• Authentication and service-to-service trust. Callback calls need the same service-to-service authentication as any other synchronous call (Chapter 6’s concerns apply in full) — an event’s origin doesn’t grant the resulting callback any special trust.
• Logging, metrics, tracing, correlation IDs. Propagate the original correlation ID through both the event and any resulting callback, so a trace of “what happened to order X” includes the callback hop, not just the event publish.
• Kubernetes deployment. No special considerations beyond Chapter 7’s — but note that a thin-notification consumer’s callback adds load back onto the publisher (order-service), which must be capacity-planned for that traffic, not just for its own direct client traffic.
• Testing strategy. Test fat-event consumers by asserting behavior purely from the event payload (no network mocking needed, since there’s no callback) — a genuinely simpler test than a thin-notification consumer, which needs its callback mocked (WireMock, as in Chapter 2) in addition to the event itself.
• Migration strategy. If a thin-notification consumer’s callback volume grows to the point where it’s a real load or availability concern, migrate that specific consumer to a fat event (or a dedicated topic) — a decision made per consumer, based on its actual growth, not a system-wide redesign.

8. Common Mistakes

1. One giant event trying to serve every consumer. Cramming everything every current and hypothetical future consumer might need into a single OrderPlaced schema makes every field a negotiation across every subscriber. Fix: split into purpose-specific topics/events per consumer need, as this chapter does with orders.notifications and orders.loyalty-events.
2. Defaulting to thin notification for a high-volume consumer. Forcing a per-event callback for a consumer processing thousands of events per second reintroduces the exact temporal coupling event-driven architecture was adopted to remove. Fix: measure consumer call volume and choose the shape per consumer, not by a blanket policy.
3. Treating an embedded field’s staleness as a bug instead of a documented semantic. Alerting on customerTier in OrderPlacedForLoyalty “not matching the customer’s current tier” without recognizing that “tier at time of order” is the intended business meaning wastes on-call time chasing a non-issue. Fix: document the intended point-in-time semantics of every embedded field explicitly, in the event’s own schema comments.
4. No reconciliation for consumers building local read models from fat events. Assuming every event will always be delivered, in order, forever, without a periodic sanity check invites silent, undetected drift. Fix: build a low-frequency reconciliation job (as shown in Section 5) comparing a sample of the local copy against the source of truth.
5. Letting a callback-based consumer’s failure block the event pipeline. If analytics-pipeline’s consumer thread blocks retrying a failed callback without a bound, it can stall processing of subsequent messages on its partition. Fix: bound callback retries, and route persistently failing messages to a dead-letter topic (the next chapters address this explicitly) rather than blocking the consumer indefinitely.
6. Not treating a widely-embedded field as a public contract. Removing or renaming a field from a fat event because “it seemed unused” without checking which consumers actually depend on it breaks them silently, discovered only when their local read model stops updating correctly. Fix: treat every field in a widely-consumed event exactly as seriously as a REST API field (Chapter 2’s contract discipline applies here too).

9. Decision Guide

10. Hands-On Exercise

Extend it: design the event shape for a new fraud-check consumer that needs to score every order in real time against the customer’s recent order history and payment method — does this consumer’s profile point toward thin notification or event-carried state transfer? Justify using Section 3’s criteria, then implement the chosen shape.

Simulate a failure: have order-service stop responding to GET /orders/{id} entirely (simulating an outage) while analytics-pipeline’s callback-based consumer is processing a backlog of notifications. Observe what happens to that consumer’s processing, and design a dead-letter or retry-with-backoff strategy so a order-service outage doesn’t stall analytics-pipeline indefinitely.

Decision question, with justification required: loyalty-service’s OrderPlacedForLoyalty event currently embeds customerTier as a plain string. A new requirement: loyalty points should also depend on the customer’s lifetime order count, a value that changes on every order and would need to be embedded fresh each time, doubling as a potential source of race conditions if two orders are placed nearly simultaneously. Should this value be embedded in the event, fetched via callback, or computed independently inside loyalty-service from its own local event history? Weigh the trade-offs from Section 1 and Section 7.

11. Key Takeaways

• Publish-subscribe (Chapter 7) answers who gets a copy of an event; event notification versus event-carried state transfer answers how much each event carries — a separate, per-event-type decision.
• Thin notifications minimize schema coupling but may reintroduce a synchronous callback dependency — acceptable for low-volume, exploratory, or staleness-sensitive consumers.
• Fat events (event-carried state transfer) eliminate the callback but widen the schema’s blast radius — justified for high-volume consumers that would otherwise recreate the temporal coupling event-driven architecture was meant to remove.
• Different consumers of the same underlying fact can and often should receive differently-shaped events on different topics, rather than one universal event trying to serve everyone.
• Embedded data in a fat event is a point-in-time snapshot, not a live value — document that semantic explicitly so it isn’t mistaken for a staleness bug.
• Any consumer building a local read model from fat events needs a periodic reconciliation check against the source of truth — events can be missed or misordered, and silent drift is worse than a visible failure.
• Treat every field in a widely-consumed event with the same contract discipline as a public REST API field — removing one breaks consumers you may not know exist.