Part 3 of 1718% complete

2026-08-18•6 min read

Build the deterministic operations simulator

Checkpoint tag: chapter-02-operations-simulator — the simulator runs against real PostgreSQL, documented requests pass, and every failure mode the series needs is one HTTP call away.

What will be built

operations-simulator becomes real: a service inventory, per-service health, deployment metadata, incident listing/lookup, idempotent incident creation, note appending, and an admin endpoint that flips the service into latency, flaky, malformed, or outage mode. Along the way we fill domain-contracts with the identifiers and result types the rest of the platform shares.

Why it matters

An agent that calls tools needs something real on the other end. Mocking the downstream in every test would hide exactly the behaviors we most need to exercise — idempotency on retried writes, partial failures, slow responses, corrupt payloads. A deterministic simulator gives us a system of record we fully control: seeded data that never changes underneath a test, and fault modes that make failure-injection a configuration call rather than a broken build.

Prerequisites and starting tag

Starting tag: chapter-01-build-foundation. You need Docker running for the PostgreSQL integration tests.

Architecture before and after

Concepts explained

Contracts vs. entities. The records in domain-contracts describe the API — what a caller may send and receive. The tables in sim describe storage. They are deliberately separate types: renaming a column must never be a wire change, and changing the wire must never require a migration. JPA entities would blur that line, so the simulator uses JdbcClient and writes SQL it owns.

Idempotency keys. A retried POST is indistinguishable from a second request. The fix is standard: the client supplies Idempotency-Key; the server stores (key, request hash, response) and replays the stored response on key reuse — while rejecting key reuse with a different body (that is a client bug, not a retry). Chapter 10’s approval flow leans on this mechanism directly.

Fault injection as a feature, not a hack. The fault registry is a first-class component with its own endpoint. Hiding it in test-only code would mean the failure modes we test with are not the failure modes we ship.

Files added or changed

1
domain-contracts/src/main/java/in/o612/eng/opsagent/contracts/...
2
operations-simulator/src/main/resources/application.yml
3
operations-simulator/src/main/resources/db/migration/V1__init.sql
4
operations-simulator/src/main/resources/db/migration/V2__seed.sql
5
operations-simulator/src/main/java/in/o612/eng/opsagent/simulator/... (persistence, web, faults)
6
operations-simulator/src/test/java/... (Testcontainers tests)
7
infra/compose/docker-compose.yml (postgres only, for now)
8
api-requests/simulator.http

Implementation steps

Add contract types to domain-contracts.
Write V1__init.sql (schema + tables) and V2__seed.sql (deterministic data).
Implement repositories on JdbcClient.
Implement controllers and the idempotency store.
Implement the fault registry + servlet filter + admin endpoint.
Wire application.yml, docker-compose Postgres.
Write Testcontainers-backed tests for happy path, idempotency, and each fault mode.

Complete code — domain contracts

1
package in.o612.eng.opsagent.contracts.ids;
2

3
public record TenantId(String value) {
4
    public TenantId {
5
        if (value == null || !value.matches("[a-z0-9][a-z0-9-]{1,31}")) {
6
            throw new IllegalArgumentException("invalid tenant id: " + value);
7
        }
8
    }
9
}

The same shape for ServiceId, IncidentId, UserId, CorrelationId, and IdempotencyKey — each a one-field record with a compactness check in the canonical constructor. Typed identifiers are cheap; a String that might be a tenant or might be an incident ID is how cross-tenant bugs start.

1
package in.o612.eng.opsagent.contracts.incident;
2

3
public enum Severity { SEV1, SEV2, SEV3, SEV4 }

1
package in.o612.eng.opsagent.contracts.incident;
2

3
public enum IncidentStatus { OPEN, INVESTIGATING, MITIGATED, RESOLVED }

1
package in.o612.eng.opsagent.contracts.incident;
2

3
import in.o612.eng.opsagent.contracts.ids.ServiceId;
4

5
public record CreateIncidentCommand(
6
        String title,
7
        Severity severity,
8
        ServiceId serviceId,
9
        String description) {
10

11
    public CreateIncidentCommand {
12
        if (title == null || title.isBlank() || title.length() > 200) {
13
            throw new IllegalArgumentException("title required, <= 200 chars");
14
        }
15
        if (severity == null || serviceId == null) {
16
            throw new IllegalArgumentException("severity and serviceId required");
17
        }
18
        if (description != null && description.length() > 4000) {
19
            throw new IllegalArgumentException("description <= 4000 chars");
20
        }
21
    }
22
}

1
package in.o612.eng.opsagent.contracts.incident;
2

3
import java.time.Instant;
4
import java.util.List;
5

6
public record IncidentDetail(
7
        String id,
8
        String tenantId,
9
        String serviceId,
10
        String title,
11
        Severity severity,
12
        IncidentStatus status,
13
        Instant createdAt,
14
        List<String> notes) {}

ServiceStatusResult (serviceId, health enum HEALTHY/DEGRADED/DOWN, lastDeployment, checkedAt) and IncidentSummary (the detail minus notes) follow the same record pattern. ApiError carries code, message, correlationId — and deliberately never a stack trace.

Complete code — simulator

application.yml

1
server:
2
  port: 8082
3

4
spring:
5
  application:
6
    name: operations-simulator
7
  datasource:
8
    url: ${SIM_DB_URL:jdbc:postgresql://localhost:5432/simdb}
9
    username: ${SIM_DB_USER:sim}
10
    password: ${SIM_DB_PASSWORD:sim-dev-password}
11
  flyway:
12
    schemas: sim
13
    default-schema: sim
14
  jdbc:
15
    template:
16
      query-timeout: 5s
17

18
simulator:
19
  seed-fixed-clock: "2026-09-01T00:00:00Z"

V1__init.sql

1
CREATE SCHEMA IF NOT EXISTS sim;
2

3
CREATE TABLE sim.services (
4
    service_id   TEXT PRIMARY KEY,
5
    tenant_id    TEXT NOT NULL,
6
    name         TEXT NOT NULL,
7
    health       TEXT NOT NULL CHECK (health IN ('HEALTHY','DEGRADED','DOWN')),
8
    updated_at   TIMESTAMPTZ NOT NULL
9
);
10

11
CREATE TABLE sim.deployments (
12
    id           BIGINT GENERATED ALWAYS AS IDENTITY PRIMARY KEY,
13
    service_id   TEXT NOT NULL REFERENCES sim.services(service_id),
14
    version      TEXT NOT NULL,
15
    deployed_at  TIMESTAMPTZ NOT NULL
16
);
17

18
CREATE TABLE sim.incidents (
19
    id           TEXT PRIMARY KEY,
20
    tenant_id    TEXT NOT NULL,
21
    service_id   TEXT NOT NULL REFERENCES sim.services(service_id),
22
    title        TEXT NOT NULL,
23
    severity     TEXT NOT NULL,
24
    status       TEXT NOT NULL,
25
    description  TEXT,
26
    created_at   TIMESTAMPTZ NOT NULL,
27
    UNIQUE (tenant_id, id)
28
);
29

30
CREATE TABLE sim.incident_notes (
31
    id           BIGINT GENERATED ALWAYS AS IDENTITY PRIMARY KEY,
32
    incident_id  TEXT NOT NULL REFERENCES sim.incidents(id),
33
    author       TEXT NOT NULL,
34
    body         TEXT NOT NULL,
35
    created_at   TIMESTAMPTZ NOT NULL
36
);
37

38
CREATE TABLE sim.idempotency_keys (
39
    idem_key     TEXT NOT NULL,
40
    tenant_id    TEXT NOT NULL,
41
    request_hash TEXT NOT NULL,
42
    incident_id  TEXT NOT NULL,
43
    created_at   TIMESTAMPTZ NOT NULL,
44
    PRIMARY KEY (idem_key, tenant_id)
45
);

V2__seed.sql seeds two tenants sharing a catalog shape but owning different rows — acme owns payment-gateway and ledger-core; globex owns checkout-web — plus a handful of incidents with fixed created_at values pinned to simulator.seed-fixed-clock. Deterministic seeds are what let Chapter 13’s evaluation cases name expected documents.

FaultRegistry and FaultFilter:

1
package in.o612.eng.opsagent.simulator.faults;
2

3
import java.util.concurrent.atomic.AtomicReference;
4

5
public class FaultRegistry {
6

7
    public sealed interface FaultMode {
8
        record None() implements FaultMode {}
9
        record Latency(long millis) implements FaultMode {}
10
        record Flaky(double errorRate) implements FaultMode {}
11
        record Malformed() implements FaultMode {}
12
        record Outage() implements FaultMode {}
13
    }
14

15
    private final AtomicReference<FaultMode> current = new AtomicReference<>(new FaultMode.None());
16

17
    public FaultMode current() { return current.get(); }
18
    public void set(FaultMode mode) { current.set(mode); }
19
}

1
package in.o612.eng.opsagent.simulator.faults;
2

3
import jakarta.servlet.FilterChain;
4
import jakarta.servlet.ServletException;
5
import jakarta.servlet.http.HttpServletRequest;
6
import jakarta.servlet.http.HttpServletResponse;
7
import org.springframework.core.annotation.Order;
8
import org.springframework.stereotype.Component;
9
import org.springframework.web.filter.OncePerRequestFilter;
10

11
import java.io.IOException;
12
import java.util.concurrent.ThreadLocalRandom;
13

14
@Component
15
@Order(1)
16
public class FaultFilter extends OncePerRequestFilter {
17

18
    private final FaultRegistry registry;
19

20
    public FaultFilter(FaultRegistry registry) { this.registry = registry; }
21

22
    @Override
23
    protected boolean shouldNotFilter(HttpServletRequest request) {
24
        return !request.getRequestURI().startsWith("/sim/v1/");
25
    }
26

27
    @Override
28
    protected void doFilterInternal(HttpServletRequest req, HttpServletResponse res, FilterChain chain)
29
            throws ServletException, IOException {
30
        switch (registry.current()) {
31
            case FaultRegistry.FaultMode.None n -> chain.doFilter(req, res);
32
            case FaultRegistry.FaultMode.Latency l -> {
33
                try { Thread.sleep(l.millis()); } catch (InterruptedException e) { Thread.currentThread().interrupt(); }
34
                chain.doFilter(req, res);
35
            }
36
            case FaultRegistry.FaultMode.Flaky f -> {
37
                if (ThreadLocalRandom.current().nextDouble() < f.errorRate()) {
38
                    res.sendError(503, "injected transient failure");
39
                } else {
40
                    chain.doFilter(req, res);
41
                }
42
            }
43
            case FaultRegistry.FaultMode.Malformed m -> {
44
                res.setStatus(200);
45
                res.setContentType("application/json");
46
                res.getWriter().write("NOT JSON {{{");
47
            }
48
            case FaultRegistry.FaultMode.Outage o -> res.sendError(503, "injected outage");
49
        }
50
    }
51
}

1
package in.o612.eng.opsagent.simulator.faults;
2

3
import org.springframework.web.bind.annotation.*;
4

5
@RestController
6
@RequestMapping("/sim/admin/faults")
7
public class FaultAdminController {
8

9
    private final FaultRegistry registry;
10

11
    public FaultAdminController(FaultRegistry registry) { this.registry = registry; }
12

13
    @PostMapping("/{mode}")
14
    public String set(@PathVariable String mode,
15
                      @RequestParam(defaultValue = "2000") long latencyMs,
16
                      @RequestParam(defaultValue = "0.5") double errorRate) {
17
        registry.set(switch (mode) {
18
            case "none", "reset" -> new FaultRegistry.FaultMode.None();
19
            case "latency" -> new FaultRegistry.FaultMode.Latency(latencyMs);
20
            case "flaky" -> new FaultRegistry.FaultMode.Flaky(errorRate);
21
            case "malformed" -> new FaultRegistry.FaultMode.Malformed();
22
            case "outage" -> new FaultRegistry.FaultMode.Outage();
23
            default -> throw new IllegalArgumentException("unknown fault mode: " + mode);
24
        });
25
        return mode;
26
    }
27
}

Incident creation with idempotency — the heart of the module:

1
package in.o612.eng.opsagent.simulator.web;
2

3
import in.o612.eng.opsagent.contracts.incident.*;
4
import in.o612.eng.opsagent.simulator.persistence.IncidentRepository;
5
import org.springframework.http.HttpStatus;
6
import org.springframework.http.ResponseEntity;
7
import org.springframework.web.bind.annotation.*;
8
import org.springframework.web.server.ResponseStatusException;
9

10
import java.util.List;
11

12
@RestController
13
@RequestMapping("/sim/v1/incidents")
14
public class IncidentController {
15

16
    private final IncidentRepository incidents;
17

18
    public IncidentController(IncidentRepository incidents) { this.incidents = incidents; }
19

20
    @GetMapping
21
    public List<IncidentSummary> list(@RequestHeader("X-Tenant-Id") String tenant,
22
                                      @RequestParam(required = false) String serviceId,
23
                                      @RequestParam(required = false) Severity severity,
24
                                      @RequestParam(defaultValue = "20") int limit) {
25
        return incidents.listRecent(tenant, serviceId, severity, Math.min(limit, 100));
26
    }
27

28
    @GetMapping("/{id}")
29
    public IncidentDetail get(@RequestHeader("X-Tenant-Id") String tenant,
30
                              @PathVariable String id) {
31
        return incidents.find(tenant, id)
32
                .orElseThrow(() -> new ResponseStatusException(HttpStatus.NOT_FOUND, "no such incident"));
33
    }
34

35
    @PostMapping
36
    public ResponseEntity<IncidentDetail> create(@RequestHeader("X-Tenant-Id") String tenant,
37
                                                 @RequestHeader("Idempotency-Key") String idemKey,
38
                                                 @RequestBody CreateIncidentCommand cmd) {
39
        var outcome = incidents.createIdempotent(tenant, idemKey, cmd);
40
        return switch (outcome) {
41
            case IncidentRepository.CreateOutcome.Created c -> ResponseEntity.status(201).body(c.incident());
42
            case IncidentRepository.CreateOutcome.Replayed r -> ResponseEntity.ok(r.incident());
43
            case IncidentRepository.CreateOutcome.KeyConflict k ->
44
                    throw new ResponseStatusException(HttpStatus.CONFLICT,
45
                            "idempotency key reused with different payload");
46
        };
47
    }
48

49
    @PostMapping("/{id}/notes")
50
    public ResponseEntity<Void> addNote(@RequestHeader("X-Tenant-Id") String tenant,
51
                                        @PathVariable String id,
52
                                        @RequestBody NoteRequest note) {
53
        incidents.find(tenant, id)
54
                .orElseThrow(() -> new ResponseStatusException(HttpStatus.NOT_FOUND, "no such incident"));
55
        incidents.appendNote(id, note.author(), note.body());
56
        return ResponseEntity.accepted().build();
57
    }
58

59
    public record NoteRequest(String author, String body) {}
60
}

IncidentRepository.createIdempotent (abridged shape — the full file in the repo hashes the request body with SHA-256, inserts the key row first, and lets the primary-key constraint arbitrate concurrent retries):

1
// Core pattern:
2
// 1. hash normalized command -> requestHash
3
// 2. SELECT request_hash, incident_id FROM sim.idempotency_keys WHERE idem_key=? AND tenant_id=?
4
// 3a. found, hash matches -> Replayed(fetch incident)
5
// 3b. found, hash differs -> KeyConflict
6
// 3c. absent -> INSERT key row + INSERT incident in one tx -> Created
7
// On PK race: re-read row, repeat step 3.

The X-Tenant-Id header is intentionally crude — this is the simulator’s contract for its trusted caller (the MCP server), not end-user auth. Tenant enforcement for real callers happens in Chapters 9–10 at the MCP and agent boundaries.

infra/compose/docker-compose.yml — Postgres only; the stack grows with the chapters:

1
services:
2
  postgres:
3
    image: pgvector/pgvector:0.8.6-pg17
4
    environment:
5
      POSTGRES_USER: postgres
6
      POSTGRES_PASSWORD: postgres-dev
7
    ports: ["5432:5432"]
8
    volumes:
9
      - ../postgres/init.sql:/docker-entrypoint-initdb.d/init.sql:ro
10
    healthcheck:
11
      test: ["CMD-SHELL", "pg_isready -U postgres"]
12
      interval: 5s
13
      timeout: 3s
14
      retries: 10

infra/postgres/init.sql creates simdb, opsdb, and the sim/agent login roles with least-privilege grants — committed in full in the repo.

Commands to build and run

1
docker compose -f infra/compose/docker-compose.yml up -d postgres
2
./gradlew :operations-simulator:bootRun

API requests and expected responses

1
### health of a seeded service
2
GET http://localhost:8082/sim/v1/services/payment-gateway/health
3
X-Tenant-Id: acme
4

5
### idempotent create — run twice, identical response, one row
6
POST http://localhost:8082/sim/v1/incidents
7
X-Tenant-Id: acme
8
Idempotency-Key: demo-0001
9
Content-Type: application/json
10

11
{"title":"payment-gateway elevated error rate","severity":"SEV2","serviceId":"payment-gateway","description":"5xx rate above threshold since 02:14Z"}
12

13
### flip into outage mode, then reset
14
POST http://localhost:8082/sim/admin/faults/outage
15
POST http://localhost:8082/sim/admin/faults/reset

Expected: the first POST returns 201 with a generated inc-<hex> ID; the identical replay returns 200 with the same incident; the same key with a changed title returns 409; with outage active every /sim/v1/** call returns 503 until reset.

Automated tests

Testcontainers-backed (@ServiceConnection for Boot, or an explicit PostgreSQLContainer("pgvector/pgvector:0.8.6-pg17")):

IncidentRepositoryTest: create/replay/conflict triple; notes appended in order; list filters by tenant, service, severity.
IncidentControllerIT: full HTTP round-trip; second POST with same key returns 200 + identical body and does not create a row (assert count(*) unchanged).
FaultFilterIT: latency adds ≥ the configured delay; flaky at errorRate=1.0 always 503s; malformed returns 200 with a body that fails JSON parsing; outage 503s even health-adjacent endpoints but never /sim/admin/**.

Failure-injection lab

This chapter is the failure lab — every mode is exercised:

flaky at 0.5: script 20 identical GETs, count 503s (expect roughly half; the point is variance exists).
malformed while your HTTP client parses JSON: observe the DecodingException — this exact error shape returns in Chapter 8’s client tests.
latency at 3000ms vs. the simulator’s own 5s JDBC timeout: slow ≠ failed; the distinction matters for Chapter 11’s retry classification.

Security considerations

The simulator trusts X-Tenant-Id — safe only because it is an internal test double. It is still scoped to a private network in Compose and K8s, and its write endpoints still require idempotency, because the downstream contract must match what a real system would demand. The admin fault endpoint must never be reachable from the agent-facing boundary; NetworkPolicy in Chapter 15 enforces that.

Observability checks

Actuator health + a simulator.faults.current gauge (mode as a tag) land now; structured access logs carry tenant_id and correlation_id placeholders so Chapter 12’s trace work has somewhere to attach.

Troubleshooting

Flyway checksum mismatch after editing a migration: migrations are immutable once applied — add a new V3__… instead; flyway repair exists but hides real history drift.
relation sim.services does not exist in tests: the container DB ran init before Flyway — ensure spring.flyway.schemas=sim and the schema is created by the migration, not the init script.

Checkpoint verification checklist

./gradlew :operations-simulator:check green, including Testcontainers tests.
Idempotent replay verified by row count, not just identical responses.
All four fault modes observed over HTTP.
No JPA anywhere; contracts module still dependency-clean (arch tests green).

Commit message and Git tag

1
feat(simulator): deterministic ops backend with idempotent writes and fault modes

git tag chapter-02-operations-simulator

What comes next

Chapter 3 stands up agent-api with a real model — Ollama locally, a hosted profile behind config — plus the deterministic stub model that keeps default CI offline.

Project State Ledger — `chapter-02-operations-simulator`

Contracts added: ids (TenantId, ServiceId, IncidentId, UserId, CorrelationId, IdempotencyKey), incident (Severity, IncidentStatus, CreateIncidentCommand, IncidentDetail, IncidentSummary), service (HealthStatus, ServiceStatusResult), ApiError
Endpoints live: /sim/v1/services[/…], /sim/v1/incidents[/…], /sim/v1/incidents/{id}/notes, /sim/admin/faults/{mode}
Migrations: V1__init.sql, V2__seed.sql in sim schema
Idempotency: Idempotency-Key + tenant + SHA-256 request hash; replay→200, conflict→409
Fault modes: none|latency|flaky|malformed|outage|reset, /sim/v1/** only
Tenant model: simulator trusts X-Tenant-Id (internal contract); real enforcement lands Ch 9–10
Next: chapter-03-basic-agent-api

Spring BootJavaPostgresTesting