Part 4 of 1724% complete

2026-08-19•5 min read

Create the agent API: ChatClient, provider profiles, and a deterministic stub model

Checkpoint tag: chapter-03-basic-agent-api — agent-api answers messages through a real model locally, streams over SSE, and the entire default test suite runs offline against a stub.

What will be built

agent-api gains its request surface: create a conversation, post a message, get an answer, and stream events over SSE. Underneath sits the seam the whole series protects — a ConversationService that talks to a ChatModel port, with three interchangeable bindings: Ollama (local default), a hosted OpenAI-compatible profile, and a deterministic StubChatModel for tests and CI.

Why it matters

The first architectural decision in an AI application is where the model lives in the dependency graph. If ChatClient types leak into your application service, every later concern — retrieval, tools, approval, evaluation — gets welded to one provider’s SDK. We put the model behind a port on day one, which is why Chapter 6 can swap in structured output and Chapter 8 can add tool calls without touching HTTP code.

The second decision is how you stream. We use Spring MVC plus SseEmitter, not WebFlux: SSE is a server-push channel with trivial backpressure requirements (the client reads or it doesn’t), and a reactive stack would buy us nothing here while taxing every subsequent chapter.

Prerequisites and starting tag

Starting tag: chapter-02-operations-simulator. For the real-model path: Ollama running (ollama pull qwen3:4b) or the Compose service from the updated stack file below. Everything testable runs without either.

Concepts explained

ChatClient vs. ChatModel. ChatModel is the low-level SPI (call(Prompt) → ChatResponse, stream(Prompt) → Flux<ChatResponse>). ChatClient is the fluent facade — prompt assembly, advisors, tool callbacks. Our application code depends on a ModelGateway port that wraps ChatClient; the stub implements the port directly so no Spring AI machinery sits between a test and a canned answer.

Profiles as provider selection. agent.model.provider selects the bean wiring, not just config values: ollama, openai, or stub. Credentials never live in the repo; the hosted profile reads OPENAI_API_KEY from the environment and the app fails fast if it is absent, rather than silently falling back to a local model with different behavior.

SSE over MVC. SseEmitter gives each connected client a queue; the conversation’s event bus fans events out to its emitters. When the emitter times out or the client disconnects, we remove it — the same path Chapter 10 reuses for approval-request events.

Files added or changed

1
domain-contracts: + conversation types (ConversationId, ChatRequest, ChatReply, AgentEvent)
2
agent-api/src/main/resources/application.yml (+ application-openai.yml)
3
agent-api/src/main/resources/prompts/system.st
4
agent-api/src/main/java/in/o612/eng/opsagent/agent/
5
    api/ConversationController.java, api/ApiExceptionHandler.java
6
    conversation/ConversationService.java, conversation/ConversationStore.java, conversation/ConversationEventBus.java
7
    model/ModelGateway.java, model/ChatClientGateway.java, model/StubModelGateway.java, model/ModelConfig.java
8
    config/AgentProperties.java
9
agent-api/src/main/resources/db/migration/V1__agent_init.sql
10
agent-api/src/test/java/... (stub-driven tests)
11
infra/compose/docker-compose.yml (+ ollama)
12
api-requests/agent.http

Complete code — contracts

1
package in.o612.eng.opsagent.contracts.conversation;
2

3
import java.time.Instant;
4

5
public sealed interface AgentEvent {
6

7
    String conversationId();
8
    Instant at();
9

10
    record Token(String conversationId, String text, Instant at) implements AgentEvent {}
11
    record ToolProposed(String conversationId, String tool, String argsJson, Instant at) implements AgentEvent {}
12
    record ApprovalRequired(String conversationId, String approvalId, String tool,
13
                            String argsJson, Instant expiresAt, Instant at) implements AgentEvent {}
14
    record Completed(String conversationId, String answer, Instant at) implements AgentEvent {}
15
    record Failed(String conversationId, String code, String message, Instant at) implements AgentEvent {}
16
}

Plus ChatRequest(String message), ChatReply(String answer, List<Citation> citations, List<String> evidence), and a Citation record (documentId, headingPath, chunkId) that Chapter 5 fills in — until then both lists are empty.

Complete code — agent-api

application.yml

1
server:
2
  port: 8080
3

4
spring:
5
  application:
6
    name: agent-api
7
  datasource:
8
    url: ${OPS_DB_URL:jdbc:postgresql://localhost:5432/opsdb}
9
    username: ${OPS_DB_USER:agent}
10
    password: ${OPS_DB_PASSWORD:agent-dev-password}
11
  flyway:
12
    schemas: agent
13
    default-schema: agent
14
  ai:
15
    ollama:
16
      base-url: ${OLLAMA_BASE_URL:http://localhost:11434}
17
      chat:
18
        options:
19
          model: qwen3:4b
20
          temperature: 0.1
21

22
agent:
23
  model:
24
    provider: ${MODEL_PROVIDER:ollama}   # ollama | openai | stub
25
    timeout: 60s
26
    max-prompt-chars: 16000
27
  system-prompt: classpath:prompts/system.st

application-openai.yml — the hosted profile. It only activates alongside agent.model.provider=openai; the missing-key failure is loud:

1
spring:
2
  ai:
3
    openai:
4
      api-key: ${OPENAI_API_KEY:?Set OPENAI_API_KEY for the hosted profile}
5
      chat:
6
        options:
7
          model: ${OPENAI_MODEL:gpt-4o-mini}
8
          temperature: 0.1

Dependencies added to agent-api/build.gradle.kts: spring-ai-starter-model-ollama, spring-ai-starter-model-openai, spring-boot-starter-jdbc, spring-boot-starter-flyway, runtimeOnly postgresql.

ModelGateway — the port:

1
package in.o612.eng.opsagent.agent.model;
2

3
import java.util.List;
4
import java.util.function.Consumer;
5

6
public interface ModelGateway {
7

8
    /** Blocking call; returns the model's complete text answer. */
9
    String complete(String systemPrompt, List<Turn> history, String userMessage);
10

11
    /** Streaming call; invokes onToken per chunk, onDone when finished. */
12
    void stream(String systemPrompt, List<Turn> history, String userMessage,
13
                Consumer<String> onToken, Runnable onDone, Consumer<Throwable> onError);
14

15
    record Turn(String role, String content) {}
16
}

ChatClientGateway — the Spring AI binding:

1
package in.o612.eng.opsagent.agent.model;
2

3
import org.springframework.ai.chat.client.ChatClient;
4
import org.springframework.ai.chat.messages.AssistantMessage;
5
import org.springframework.ai.chat.messages.Message;
6
import org.springframework.ai.chat.messages.UserMessage;
7

8
import java.util.ArrayList;
9
import java.util.List;
10
import java.util.function.Consumer;
11

12
public class ChatClientGateway implements ModelGateway {
13

14
    private final ChatClient chatClient;
15

16
    public ChatClientGateway(ChatClient chatClient) { this.chatClient = chatClient; }
17

18
    @Override
19
    public String complete(String systemPrompt, List<Turn> history, String userMessage) {
20
        return chatClient.prompt()
21
                .system(systemPrompt)
22
                .messages(toMessages(history))
23
                .user(userMessage)
24
                .call()
25
                .content();
26
    }
27

28
    @Override
29
    public void stream(String systemPrompt, List<Turn> history, String userMessage,
30
                       Consumer<String> onToken, Runnable onDone, Consumer<Throwable> onError) {
31
        chatClient.prompt()
32
                .system(systemPrompt)
33
                .messages(toMessages(history))
34
                .user(userMessage)
35
                .stream()
36
                .content()
37
                .doOnNext(onToken)
38
                .doOnError(onError)
39
                .doOnComplete(onDone)
40
                .subscribe();
41
    }
42

43
    private List<Message> toMessages(List<Turn> history) {
44
        var messages = new ArrayList<Message>();
45
        for (Turn t : history) {
46
            messages.add(t.role().equals("assistant")
47
                    ? new AssistantMessage(t.content())
48
                    : new UserMessage(t.content()));
49
        }
50
        return messages;
51
    }
52
}

StubModelGateway — deterministic, offline, and the reason default CI needs no GPU, no container, and no API key:

1
package in.o612.eng.opsagent.agent.model;
2

3
import java.util.List;
4
import java.util.function.Consumer;
5

6
public class StubModelGateway implements ModelGateway {
7

8
    @Override
9
    public String complete(String systemPrompt, List<Turn> history, String userMessage) {
10
        return "stub-answer: " + userMessage;
11
    }
12

13
    @Override
14
    public void stream(String systemPrompt, List<Turn> history, String userMessage,
15
                       Consumer<String> onToken, Runnable onDone, Consumer<Throwable> onError) {
16
        String answer = complete(systemPrompt, history, userMessage);
17
        for (String word : answer.split(" ")) {
18
            onToken.accept(word + " ");
19
        }
20
        onDone.run();
21
    }
22
}

ModelConfig — explicit bean selection, no classpath roulette:

1
package in.o612.eng.opsagent.agent.model;
2

3
import org.springframework.ai.chat.client.ChatClient;
4
import org.springframework.ai.chat.model.ChatModel;
5
import org.springframework.boot.autoconfigure.condition.ConditionalOnProperty;
6
import org.springframework.context.annotation.Bean;
7
import org.springframework.context.annotation.Configuration;
8

9
@Configuration
10
public class ModelConfig {
11

12
    @Bean
13
    @ConditionalOnProperty(name = "agent.model.provider", havingValue = "stub")
14
    ModelGateway stubGateway() {
15
        return new StubModelGateway();
16
    }
17

18
    @Bean
19
    @ConditionalOnProperty(name = "agent.model.provider", havingValue = "ollama", matchIfMissing = true)
20
    ChatClient ollamaChatClient(ChatModel ollamaChatModel) {
21
        return ChatClient.builder(ollamaChatModel).build();
22
    }
23

24
    @Bean
25
    @ConditionalOnProperty(name = "agent.model.provider", havingValue = "openai")
26
    ChatClient openAiChatClient(ChatModel openAiChatModel) {
27
        return ChatClient.builder(openAiChatModel).build();
28
    }
29

30
    @Bean
31
    @ConditionalOnProperty(name = "agent.model.provider", havingValue = "ollama", matchIfMissing = true)
32
    ModelGateway chatClientGateway(ChatClient ollamaChatClient) { return new ChatClientGateway(ollamaChatClient); }
33
}

ConversationService — provider-neutral orchestration. History is capped (last 20 turns), prompt size is enforced before the model is ever invoked, and every message is persisted with its tenant and correlation IDs:

1
package in.o612.eng.opsagent.agent.conversation;
2

3
import in.o612.eng.opsagent.agent.config.AgentProperties;
4
import in.o612.eng.opsagent.agent.model.ModelGateway;
5
import in.o612.eng.opsagent.contracts.conversation.AgentEvent;
6
import org.springframework.stereotype.Service;
7

8
import java.time.Instant;
9
import java.util.List;
10

11
@Service
12
public class ConversationService {
13

14
    private final ConversationStore store;
15
    private final ConversationEventBus events;
16
    private final ModelGateway model;
17
    private final AgentProperties props;
18
    private final String systemPrompt;
19

20
    public ConversationService(ConversationStore store, ConversationEventBus events,
21
                               ModelGateway model, AgentProperties props,
22
                               org.springframework.core.io.ResourceLoader loader) {
23
        this.store = store;
24
        this.events = events;
25
        this.model = model;
26
        this.props = props;
27
        this.systemPrompt = loadPrompt(loader, props.systemPrompt());
28
    }
29

30
    public String createConversation(String tenantId, String userId) {
31
        return store.create(tenantId, userId);
32
    }
33

34
    public void handleMessage(String conversationId, String tenantId, String message) {
35
        if (message.length() > props.maxPromptChars()) {
36
            events.publish(conversationId,
37
                new AgentEvent.Failed(conversationId, "PROMPT_TOO_LARGE",
38
                        "message exceeds " + props.maxPromptChars() + " chars", Instant.now()));
39
            return;
40
        }
41
        store.append(conversationId, "user", message);
42
        List<ModelGateway.Turn> history = store.recentTurns(conversationId, 20);
43
        StringBuilder answer = new StringBuilder();
44
        model.stream(systemPrompt, history, message,
45
            token -> {
46
                answer.append(token);
47
                events.publish(conversationId,
48
                    new AgentEvent.Token(conversationId, token, Instant.now()));
49
            },
50
            () -> {
51
                store.append(conversationId, "assistant", answer.toString());
52
                events.publish(conversationId,
53
                    new AgentEvent.Completed(conversationId, answer.toString(), Instant.now()));
54
            },
55
            err -> events.publish(conversationId,
56
                    new AgentEvent.Failed(conversationId, "MODEL_ERROR",
57
                            "model call failed", Instant.now())));
58
    }
59

60
    private static String loadPrompt(org.springframework.core.io.ResourceLoader l, String path) {
61
        try (var in = l.getResource(path).getInputStream()) {
62
            return new String(in.readAllBytes(), java.nio.charset.StandardCharsets.UTF_8);
63
        } catch (java.io.IOException e) {
64
            throw new IllegalStateException("system prompt not readable: " + path, e);
65
        }
66
    }
67
}

ConversationEventBus — a ConcurrentHashMap<String, CopyOnWriteArrayList<SseEmitter>>; subscribe(conversationId) registers an emitter with onTimeout/onCompletion removal, publish iterates and prunes dead emitters. Single-node by design — the multi-node variant (a real broker) is a stated limitation, not hidden complexity.

ConversationStore — JdbcClient over the agent schema below. recentTurns returns the last N messages oldest-first.

ConversationController:

1
package in.o612.eng.opsagent.agent.api;
2

3
import in.o612.eng.opsagent.agent.conversation.ConversationEventBus;
4
import in.o612.eng.opsagent.agent.conversation.ConversationService;
5
import in.o612.eng.opsagent.contracts.conversation.ChatReply;
6
import in.o612.eng.opsagent.contracts.conversation.ChatRequest;
7
import org.springframework.http.MediaType;
8
import org.springframework.web.bind.annotation.*;
9
import org.springframework.web.servlet.mvc.method.annotation.SseEmitter;
10

11
import java.util.List;
12
import java.util.Map;
13

14
@RestController
15
@RequestMapping("/api/v1")
16
public class ConversationController {
17

18
    private final ConversationService conversations;
19
    private final ConversationEventBus events;
20

21
    public ConversationController(ConversationService c, ConversationEventBus e) {
22
        this.conversations = c;
23
        this.events = e;
24
    }
25

26
    @PostMapping("/conversations")
27
    public Map<String, String> create(@RequestHeader("X-Tenant-Id") String tenant,
28
                                      @RequestHeader("X-User-Id") String user) {
29
        return Map.of("conversationId", conversations.createConversation(tenant, user));
30
    }
31

32
    @PostMapping("/conversations/{id}/messages")
33
    public ChatReply message(@PathVariable String id,
34
                             @RequestHeader("X-Tenant-Id") String tenant,
35
                             @RequestBody ChatRequest request) {
36
        // synchronous convenience path: same pipeline, non-streamed
37
        conversations.handleMessage(id, tenant, request.message());
38
        return new ChatReply("streamed via SSE; see /events", List.of(), List.of());
39
    }
40

41
    @GetMapping(path = "/conversations/{id}/events", produces = MediaType.TEXT_EVENT_STREAM_VALUE)
42
    public SseEmitter events(@PathVariable String id) {
43
        return events.subscribe(id);
44
    }
45
}

V1__agent_init.sql

1
CREATE SCHEMA IF NOT EXISTS agent;
2

3
CREATE TABLE agent.conversations (
4
    id            TEXT PRIMARY KEY,
5
    tenant_id     TEXT NOT NULL,
6
    user_id       TEXT NOT NULL,
7
    created_at    TIMESTAMPTZ NOT NULL DEFAULT now()
8
);
9

10
CREATE TABLE agent.messages (
11
    id               BIGINT GENERATED ALWAYS AS IDENTITY PRIMARY KEY,
12
    conversation_id  TEXT NOT NULL REFERENCES agent.conversations(id),
13
    role             TEXT NOT NULL CHECK (role IN ('user','assistant','system','tool')),
14
    content          TEXT NOT NULL,
15
    created_at       TIMESTAMPTZ NOT NULL DEFAULT now()
16
);
17

18
CREATE INDEX messages_conversation_idx ON agent.messages (conversation_id, id);

prompts/system.st — the baseline system prompt, versioned as a file, not a string literal in a service:

1
You are an operations assistant for an internal platform team.
2
Answer only from the provided runbook excerpts and tool results.
3
If the evidence does not support an answer, say so explicitly.
4
Never invent incidents, statuses, commands, or runbook content.
5
Cite the document id and heading for every factual claim.

Finally, docker-compose.yml gains an ollama service (ollama/ollama image, port 11434, a small entrypoint script that pulls qwen3:4b and nomic-embed-text on first start).

Commands to build and run

1
docker compose -f infra/compose/docker-compose.yml up -d postgres ollama
2
MODEL_PROVIDER=stub ./gradlew :agent-api:bootRun    # fully offline
3
./gradlew :agent-api:bootRun                        # real model via Ollama
4
MODEL_PROVIDER=openai OPENAI_API_KEY=sk-... ./gradlew :agent-api:bootRun  # hosted

API requests and expected responses

1
POST http://localhost:8080/api/v1/conversations
2
X-Tenant-Id: acme
3
X-User-Id: priya
4

5
# -> {"conversationId": "conv-..."}
6

7
GET http://localhost:8080/api/v1/conversations/conv-.../events
8
Accept: text/event-stream
9

10
POST http://localhost:8080/api/v1/conversations/conv-.../messages
11
X-Tenant-Id: acme
12
Content-Type: application/json
13

14
{"message": "how do I restart a degraded service"}

With MODEL_PROVIDER=stub, the SSE channel emits data: {"text":"stub-answer: "}-style Token events followed by Completed. With Ollama, you get real tokens; latency is the model’s, not the framework’s — keep that distinction, Chapter 14 depends on it.

Automated tests

ConversationServiceTest (unit, stub gateway): oversized message produces a Failed event and no model call; tokens arrive before Completed; assistant turn persisted.
ConversationControllerIT (MODEL_PROVIDER=stub): create → message → SSE receives Token then Completed.
ModelConfigTest: provider=stub yields StubModelGateway; provider=openai without OPENAI_API_KEY fails context startup with a clear missing-env error.
No test touches the network.

Failure-injection lab

Kill Ollama mid-stream: SSE emits Failed(code=MODEL_ERROR); the partially persisted assistant turn is not written (only completed answers persist — verify in agent.messages).
Send a 20 KB message: Failed(code=PROMPT_TOO_LARGE); the model was never invoked — check the stub’s call counter.
Set MODEL_PROVIDER=openai with no key: startup fails fast. Silent provider fallback would be a production incident masquerading as convenience.

Security considerations

Headers X-Tenant-Id/X-User-Id are placeholders — trusted-header auth is a lie and Chapter 9 replaces them with JWT validation; they exist now only so tenant scoping is structural, not bolted on. Message size caps already bound the cheapest DoS vector. The stub provider means CI artifacts never contain prompts or API keys.

Observability checks

agent.conversations.created and agent.messages.handled counters; agent.model.latency timer tagged by provider (stub|ollama|openai) — the label that will later separate our latency from the model’s.

Troubleshooting

ChatModel bean ambiguity (both Ollama and OpenAI on the classpath): the @ConditionalOnProperty wiring above avoids it; if you see it anyway, check for a stray auto-configured ChatClient.Builder injection.
SSE connects but no events: events publish to the conversation’s emitter list — confirm the GET /events subscription happened before POST /messages.
Ollama 404 on chat: model not pulled; docker exec ollama ollama pull qwen3:4b.

Checkpoint verification checklist

MODEL_PROVIDER=stub ./gradlew :agent-api:bootRun works with zero external services beyond Postgres.
SSE stream delivers Token events then Completed.
Default ./gradlew check is fully offline.
Conversations persist per tenant; oversized prompts are rejected pre-model.

Commit message and Git tag

1
feat(agent-api): conversation API with provider-neutral model gateway and SSE

git tag chapter-03-basic-agent-api

What comes next

Chapter 4 feeds the agent something worth saying: the runbook ingestion pipeline that turns versioned Markdown into pgvector chunks.

Project State Ledger — `chapter-03-basic-agent-api`

New endpoints: POST /api/v1/conversations, POST /conversations/{id}/messages, GET /conversations/{id}/events (SSE)
New tables: agent.conversations, agent.messages (V1__agent_init.sql)
Ports: ModelGateway (complete, stream); impls ChatClientGateway, StubModelGateway
Provider switch: agent.model.provider = ollama|openai|stub; env OPENAI_API_KEY, OPENAI_MODEL, OLLAMA_BASE_URL, MODEL_PROVIDER
Prompts: classpath:prompts/system.st, versioned
Limits: agent.model.max-prompt-chars=16000, history capped at 20 turns
Known limitation: event bus is single-node; auth headers are placeholders until Ch 9
Next: chapter-04-knowledge-ingestion

Spring BootJavaAI